You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
depot_tools/recipes/recipe_modules/gsutil/examples/custom_boto.py

70 lines
1.9 KiB
Python

Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
# Copyright 2013 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
from recipe_engine import post_process
from recipe_engine.recipe_api import Property
PYTHON_VERSION_COMPATIBILITY = 'PY3'
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
DEPS = [
'gsutil',
'recipe_engine/platform',
'recipe_engine/properties',
]
PROPERTIES = {
'boto_configs': Property(default={}, kind=dict),
}
def RunSteps(api, boto_configs):
with api.gsutil.configure_gsutil(**boto_configs):
api.gsutil(['cp', 'gs://some/gs/path', '/some/local/path'])
def GenTests(api):
yield api.test(
'no_args',
api.platform('linux', 64),
api.post_process(post_process.DropExpectation),
)
yield api.test(
'no_env',
api.properties(boto_configs={'some_config': 'some_val'}),
api.post_process(post_process.DropExpectation),
)
yield api.test(
'mac',
api.platform('mac', 64),
api.post_process(post_process.DropExpectation),
)
yield api.test(
'with_boto_config',
api.properties(boto_configs={'some_config': 'some_val'}),
api.properties.environ(
BOTO_CONFIG='/some/boto/config'
),
api.post_check(lambda check, steps: \
check(steps['gsutil cp'].env['BOTO_CONFIG'] is None)),
api.post_check(lambda check, steps: \
check('/some/boto/config' in steps['gsutil cp'].env['BOTO_PATH'])),
api.post_process(post_process.DropExpectation),
)
yield api.test(
'with_boto_path',
api.properties(boto_configs={'some_config': 'some_val'}),
api.properties.environ(
BOTO_PATH='/some/boto/path'
),
api.post_check(lambda check, steps: \
check(steps['gsutil cp'].env['BOTO_CONFIG'] is None)),
api.post_check(lambda check, steps: \
check('/some/boto/path' in steps['gsutil cp'].env['BOTO_PATH'])),
api.post_process(post_process.DropExpectation),
)