You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
depot_tools/recipes/README.recipes.md

1225 lines
68 KiB
Markdown

<!--- AUTOGENERATED BY `./recipes.py test train` -->
# Repo documentation for [depot\_tools](https://chromium.googlesource.com/chromium/tools/depot_tools.git)
## Table of Contents
**[Recipe Modules](#Recipe-Modules)**
* [bot_update](#recipe_modules-bot_update) (Python3 ✅) &mdash; Recipe module to ensure a checkout is consistent on a bot.
* [depot_tools](#recipe_modules-depot_tools) (Python3 ✅) &mdash; The `depot_tools` module provides safe functions to access paths within the depot_tools repo.
* [gclient](#recipe_modules-gclient) (Python3 ✅)
* [gerrit](#recipe_modules-gerrit) (Python3 ✅)
* [git](#recipe_modules-git) (Python3 ✅)
* [git_cl](#recipe_modules-git_cl) (Python3 ✅)
* [gitiles](#recipe_modules-gitiles) (Python3 ✅)
* [gsutil](#recipe_modules-gsutil) (Python3 ✅)
* [osx_sdk](#recipe_modules-osx_sdk) (Python3 ✅) &mdash; The `osx_sdk` module provides safe functions to access a semi-hermetic XCode installation.
* [presubmit](#recipe_modules-presubmit) (Python3 ✅)
* [tryserver](#recipe_modules-tryserver) (Python3 ✅)
* [windows_sdk](#recipe_modules-windows_sdk) (Python3 ✅) &mdash; The `windows_sdk` module provides safe functions to access a hermetic Microsoft Visual Studio installation which is derived from Chromium's MSVC toolchain.
**[Recipes](#Recipes)**
* [bot_update:examples/full](#recipes-bot_update_examples_full) (Python3 ✅)
* [bot_update:tests/do_not_retry_patch_failures_in_cq](#recipes-bot_update_tests_do_not_retry_patch_failures_in_cq) (Python3 ✅)
* [bot_update:tests/download_topics](#recipes-bot_update_tests_download_topics) (Python3 ✅)
* [bot_update:tests/ensure_checkout](#recipes-bot_update_tests_ensure_checkout) (Python3 ✅)
* [depot_tools:examples/full](#recipes-depot_tools_examples_full) (Python3 ✅)
* [gclient:examples/full](#recipes-gclient_examples_full) (Python3 ✅)
* [gclient:tests/diff_deps](#recipes-gclient_tests_diff_deps) (Python3 ✅)
* [gclient:tests/patch_project](#recipes-gclient_tests_patch_project) (Python3 ✅)
* [gclient:tests/sync_failure](#recipes-gclient_tests_sync_failure) (Python3 ✅)
* [gerrit:examples/full](#recipes-gerrit_examples_full) (Python3 ✅)
* [git:examples/full](#recipes-git_examples_full) (Python3 ✅)
* [git:tests/number](#recipes-git_tests_number) (Python3 ✅)
* [git_cl:examples/full](#recipes-git_cl_examples_full) (Python3 ✅)
* [gitiles:examples/full](#recipes-gitiles_examples_full) (Python3 ✅)
* [gitiles:tests/parse_repo_url](#recipes-gitiles_tests_parse_repo_url) (Python3 ✅)
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
* [gsutil:examples/custom_boto](#recipes-gsutil_examples_custom_boto) (Python3 ✅)
* [gsutil:examples/full](#recipes-gsutil_examples_full) (Python3 ✅)
* [osx_sdk:examples/full](#recipes-osx_sdk_examples_full) (Python3 ✅)
* [presubmit:examples/full](#recipes-presubmit_examples_full) (Python3 ✅)
* [presubmit:tests/execute](#recipes-presubmit_tests_execute) (Python3 ✅)
* [presubmit:tests/prepare](#recipes-presubmit_tests_prepare) (Python3 ✅)
* [tryserver:examples/full](#recipes-tryserver_examples_full) (Python3 ✅)
* [tryserver:tests/gerrit_change_fetch_ref_timeout](#recipes-tryserver_tests_gerrit_change_fetch_ref_timeout) (Python3 ✅)
* [tryserver:tests/gerrit_change_owner](#recipes-tryserver_tests_gerrit_change_owner) (Python3 ✅)
* [tryserver:tests/gerrit_change_target_ref](#recipes-tryserver_tests_gerrit_change_target_ref) (Python3 ✅)
* [tryserver:tests/get_files_affected_by_patch](#recipes-tryserver_tests_get_files_affected_by_patch) (Python3 ✅)
* [tryserver:tests/require_is_tryserver](#recipes-tryserver_tests_require_is_tryserver) (Python3 ✅)
* [windows_sdk:examples/full](#recipes-windows_sdk_examples_full) (Python3 ✅)
## Recipe Modules
### *recipe_modules* / [bot\_update](/recipes/recipe_modules/bot_update)
[DEPS](/recipes/recipe_modules/bot_update/__init__.py#3): [depot\_tools](#recipe_modules-depot_tools), [gclient](#recipe_modules-gclient), [gerrit](#recipe_modules-gerrit), [gitiles](#recipe_modules-gitiles), [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/commit\_position][recipe_engine/recipe_modules/commit_position], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/cq][recipe_engine/recipe_modules/cq], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/milo][recipe_engine/recipe_modules/milo], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/runtime][recipe_engine/recipe_modules/runtime], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
Recipe module to ensure a checkout is consistent on a bot.
#### **class [BotUpdateApi](/recipes/recipe_modules/bot_update/api.py#12)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
&mdash; **def [\_\_call\_\_](/recipes/recipe_modules/bot_update/api.py#22)(self, name, cmd, \*\*kwargs):**
Wrapper for easy calling of bot_update.
&mdash; **def [deapply\_patch](/recipes/recipe_modules/bot_update/api.py#528)(self, bot_update_step):**
Deapplies a patch, taking care of DEPS and solution revisions properly.
&mdash; **def [ensure\_checkout](/recipes/recipe_modules/bot_update/api.py#81)(self, gclient_config=None, suffix=None, patch=True, update_presentation=True, patch_root=None, with_branch_heads=False, with_tags=False, no_fetch_tags=False, refs=None, clobber=False, root_solution_revision=None, gerrit_no_reset=False, gerrit_no_rebase_patch_ref=False, assert_one_gerrit_change=True, patch_refs=None, ignore_input_commit=False, add_blamelists=False, set_output_commit=False, step_test_data=None, enforce_fetch=False, download_topics=False, \*\*kwargs):**
Args:
* gclient_config: The gclient configuration to use when running bot_update.
If omitted, the current gclient configuration is used.
* no_fetch_tags: When true, the root git repo being checked out will not
Allow the bot_update api to not fetch tags from the git server. There are 3 layers modified, starting from the bottom up: 1. git_cache.py populate Now takes a --no-fetch-tags option. If specified, the cache will not fetch updated tags from the server by passing --no-tags to git fetch. This prevents the server from sending all the tag refs. In chromium this prevents significant time bottlenecks dealing with 10k+ tags. 2. bot_update.py bot_update has to deal with multiple git repos, it has the root repo that is checked out through git-cache, as well as repos synched via DEPS and gclient. The script now takes a --no_fetch_tags option. If specified, the git-cache populate call will include --no-fetch-tags. Otherwise, it won't. So this controls (for chromium) if fetches to the src.git server are passed --no-tags. 3. bot_update/api.py This is the entry point for recipes and also has to deal with multiple git repos. The behaviour at this point is not changed from the default. A |no_fetch_tags| parameter is added to ensure_checkout() that defaults to False. This CL is a refactor with no intended behavior change. The next step will be to change the chromium build repo to pass True for ensure_checkout(no_fetch_tags) on chromium trybots. This represents solution #2 in https://docs.google.com/document/d/1hDIunJjjfpmr50y3YnZ4o3aq1NZF4gJa1TS0p7AIL64/edit# Bug: 1019824 Change-Id: I935430603299daa9e301a95a5184c0ce486fd912 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1894352 Reviewed-by: Aaron Gable <agable@chromium.org> Reviewed-by: Erik Chen <erikchen@chromium.org> Commit-Queue: danakj <danakj@chromium.org> Auto-Submit: danakj <danakj@chromium.org>
5 years ago
fetch any tags referenced from the references being fetched. When a repo
has many references, it can become a performance bottleneck, so avoid
tags if the checkout will not need them present.
* ignore_input_commit: if True, ignore api.buildbucket.gitiles_commit.
Exists for historical reasons. Please do not use.
* add_blamelists: if True, add blamelist pins for all of the repos that had
revisions specified in the gclient config.
* set_output_commit: if True, mark the checked out commit as the
primary output commit of this build, i.e. call
api.buildbucket.set_output_gitiles_commit.
In case of multiple repos, the repo is the one specified in
api.buildbucket.gitiles_commit or the first configured solution.
When sorting builds by commit position, this commit will be used.
Requires falsy ignore_input_commit.
* step_test_data: a null function that returns test bot_update.py output.
Use test_api.output_json to generate test data.
* enforce_fetch: Enforce a new fetch to refresh the git cache, even if the
Reland "Reland "Add optional enforce_fetch kwarg to fetch and refresh git cache"" This is a reland of 3774c589cb395e9949f5e7d92f837e4a424617d8 Added missing options.enforce_fetch arg Confirmed via a fake test roll with this CL's patch commit that the bot_update step passes with this change: https://chromium-review.googlesource.com/c/chromium/tools/build/+/2443649 Original change's description: > Reland "Add optional enforce_fetch kwarg to fetch and refresh git cache" > > This is a reland of f38f54f2f1a3121643d86abab9e6ebe597238d9b > > Original change's description: > > Add optional enforce_fetch kwarg to fetch and refresh git cache > > > > Allows builders to do a force refresh of the git cache to ensure a new > > fetch happens. android-binary-size needs this to get an accurate diff > > of the passed in src revision and the applied patch: > > https://bugs.chromium.org/p/chromium/issues/detail?id=1129235#c19 > > > > Bug: 1129235 > > Change-Id: I993521cdde2a491fc9d42d75ff28dedcd2cea64e > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2438802 > > Commit-Queue: Stephanie Kim <kimstephanie@google.com> > > Reviewed-by: Josip Sokcevic <sokcevic@google.com> > > Bug: 1129235 > Change-Id: I034a91a8b57821d08b10042f7fc94d32350540ba > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2443235 > Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org> > Commit-Queue: Stephanie Kim <kimstephanie@google.com> Bug: 1129235 Change-Id: Ibbd4535f8529243d2dc1d4d32881d8a61efe374a Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2443574 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Stephanie Kim <kimstephanie@google.com>
4 years ago
solution revision passed in already exists in the current git cache.
* assert_one_gerrit_change: if True, assert that there is at most one
change in self.m.buildbucket.build.input.gerrit_changes, because
bot_update module ONLY supports one change. Users may specify a change
via tryserver.set_change() and explicitly set this flag False.
* download_topics: If True, gclient downloads and patches locally from all
open Gerrit CLs that have the same topic as the tested patch ref.
&mdash; **def [get\_project\_revision\_properties](/recipes/recipe_modules/bot_update/api.py#505)(self, project_name, gclient_config=None):**
Returns all property names used for storing the checked-out revision of
a given project.
Args:
* project_name (str): The name of a checked-out project as deps path, e.g.
src or src/v8.
* gclient_config: The gclient configuration to use. If omitted, the current
gclient configuration is used.
Returns (list of str): All properties that'll hold the checked-out revision
of the given project. An empty list if no such properties exist.
&emsp; **@property**<br>&mdash; **def [last\_returned\_properties](/recipes/recipe_modules/bot_update/api.py#49)(self):**
&mdash; **def [resolve\_fixed\_revision](/recipes/recipe_modules/bot_update/api.py#456)(self, bot_update_json, name):**
Sets a fixed revision for a single dependency using project revision
properties.
### *recipe_modules* / [depot\_tools](/recipes/recipe_modules/depot_tools)
Revert "Retry switching to use CIPD ninja v1.8.2" This reverts commit 3133f98f3f1e3b46b10a284ff8280d79d7450698. Reason for revert: angle recipe fails to find third_party/depot_tool/ninja.exe https://luci-milo.appspot.com/ui/p/angle/builders/try/win-trace/2061/overview https://source.chromium.org/chromium/chromium/src/+/main:third_party/angle/src/tests/capture_replay_tests.py;l=134;drc=1d67951e0851d855c016c04cd5d11e10a0e12454 Original change's description: > Retry switching to use CIPD ninja v1.8.2 > > This reverts commit 0a5bae7ebcceab7abdfa2d61b168c93d0fb01697 + some modifications on recipe module. > > The previous CL https://crrev.com/c/3697958 ensures depot_tools/.cipd_bin in recipe environments. This CL copies ninja.exe from .cipd_bin to depot_tools root as cipd_bin_setup.bat. > For developers, it's expected to run update_depot_tools which triggers cipd_bin_setup. > For CI/CQ, depot_tools._cipd_bin_setup() should be called before using ninja/autoninja recipe module APIs. > > Builds with depot_tools recipe bundle are tested using led. > > # Windows > > led get-build 8811491780369249585 | led edit-recipe-bundle -O depot_tools="$HOME/depot_tools" | led launch > https://ci.chromium.org/swarming/task/5b7512ee9d91fc10? > > # Linux > > led get-build 8811504372914233697 | led edit-recipe-bundle -O depot_tools="$HOME/depot_tools" | led launch > https://ci.chromium.org/swarming/task/5b7515832d0f7c10 > > Original change's description: > > Revert "Switch to use CIPD ninja v1.8.2" > > > > This reverts commit c90a98210619a2ae7d04bbf75541e41ef9f855da. > > > > Reason for revert: > > https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket/8811949917908303825/+/u/compile/stdout > > > > Original change's description: > > > Switch to use CIPD ninja v1.8.2 > > > > > > https://crrev.com/c/3674981 installed CIPD ninja packages. > > > This CL switches from the old ninja binaries to them. > > > > > > Note that this CL doesn't change ninja version. the old/CIPD ninja binaries are v1.8.2. > > > > > > The differences from the old binaries are > > > - the CIPD packages don't use chromium's clang. > > > - the CIPD package for Linux doesn't use chromium's sysroot. > > > - the min macOS version is 10.13, which was 10.6. > > > Please also see https://crsrc.org/i/3pp/ninja/install_bootstrap.sh > > > > > > > > > On Windows, ninja.exe will be installed under .cipd_bin. > > > But there are many places that assume that ninja.exe exists on depot_tools root. So this CL also copies the ninja.exe from .cipd_bin to root in cipd_bin_setup.bat. > > > > > > Bug: 931218 > > > Change-Id: Ib67eee5e9b6ad9b2937b789626970d9c85867dbd > > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3674982 > > > Commit-Queue: Junji Watanabe <jwata@google.com> > > > Reviewed-by: Philipp Wollermann <philwo@google.com> > > > Reviewed-by: Takuto Ikuta <tikuta@chromium.org> > > > Reviewed-by: Nico Weber <thakis@chromium.org> > > > Reviewed-by: Josip Sokcevic <sokcevic@google.com> > > > > Bug: 931218 > > Change-Id: I08d61dad119a0d23ae9ec8b1ed787dd3915de697 > > No-Presubmit: true > > No-Tree-Checks: true > > No-Try: true > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3686854 > > Auto-Submit: Takuto Ikuta <tikuta@chromium.org> > > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com> > > Commit-Queue: Takuto Ikuta <tikuta@chromium.org> > > Owners-Override: Takuto Ikuta <tikuta@chromium.org> > > Bug: 931218 > Recipe-Nontrivial-Roll: build > Recipe-Nontrivial-Roll: build_limited > Recipe-Nontrivial-Roll: chrome_release > Recipe-Nontrivial-Roll: infra > Change-Id: Ic7a9395c0e858823375f1a6ec79034dbc54230bb > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3696963 > Reviewed-by: Fumitoshi Ukai <ukai@google.com> > Reviewed-by: Josip Sokcevic <sokcevic@google.com> > Commit-Queue: Junji Watanabe <jwata@google.com> > Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Bug: 931218 Recipe-Nontrivial-Roll: build Recipe-Nontrivial-Roll: build_limited Recipe-Nontrivial-Roll: chrome_release Recipe-Nontrivial-Roll: infra Change-Id: I6d44be002940331eb4b7cf88ca7aa1e717ba1514 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3708294 Auto-Submit: Junji Watanabe <jwata@google.com> Commit-Queue: Junji Watanabe <jwata@google.com> Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Owners-Override: Takuto Ikuta <tikuta@chromium.org> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com> Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
3 years ago
[DEPS](/recipes/recipe_modules/depot_tools/__init__.py#7): [recipe\_engine/cipd][recipe_engine/recipe_modules/cipd], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/runtime][recipe_engine/recipe_modules/runtime]
PYTHON_VERSION_COMPATIBILITY: PY2+3
The `depot_tools` module provides safe functions to access paths within
the depot_tools repo.
#### **class [DepotToolsApi](/recipes/recipe_modules/depot_tools/api.py#12)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
&emsp; **@property**<br>&mdash; **def [autoninja\_path](/recipes/recipe_modules/depot_tools/api.py#49)(self):**
&emsp; **@property**<br>&mdash; **def [cros\_path](/recipes/recipe_modules/depot_tools/api.py#30)(self):**
&emsp; **@property**<br>&mdash; **def [download\_from\_google\_storage\_path](/recipes/recipe_modules/depot_tools/api.py#17)(self):**
&emsp; **@property**<br>&mdash; **def [gn\_py\_path](/recipes/recipe_modules/depot_tools/api.py#34)(self):**
&emsp; **@property**<br>&mdash; **def [gsutil\_py\_path](/recipes/recipe_modules/depot_tools/api.py#40)(self):**
&emsp; **@property**<br>&mdash; **def [ninja\_path](/recipes/recipe_modules/depot_tools/api.py#44)(self):**
&emsp; **@contextlib.contextmanager**<br>&mdash; **def [on\_path](/recipes/recipe_modules/depot_tools/api.py#58)(self):**
Use this context manager to put depot_tools on $PATH.
Example:
```python
with api.depot_tools.on_path():
# run some steps
```
&emsp; **@property**<br>&mdash; **def [presubmit\_support\_py\_path](/recipes/recipe_modules/depot_tools/api.py#54)(self):**
&emsp; **@property**<br>&mdash; **def [root](/recipes/recipe_modules/depot_tools/api.py#25)(self):**
Returns (Path): The "depot_tools" root directory.
&emsp; **@property**<br>&mdash; **def [upload\_to\_google\_storage\_path](/recipes/recipe_modules/depot_tools/api.py#21)(self):**
### *recipe_modules* / [gclient](/recipes/recipe_modules/gclient)
[DEPS](/recipes/recipe_modules/gclient/__init__.py#3): [git](#recipe_modules-git), [gitiles](#recipe_modules-gitiles), [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
#### **class [GclientApi](/recipes/recipe_modules/gclient/api.py#77)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
&emsp; **@property**<br>&mdash; **def [DepsDiffException](/recipes/recipe_modules/gclient/api.py#424)(self):**
&mdash; **def [\_\_call\_\_](/recipes/recipe_modules/gclient/api.py#87)(self, name, cmd, infra_step=True, \*\*kwargs):**
Wrapper for easy calling of gclient steps.
&mdash; **def [break\_locks](/recipes/recipe_modules/gclient/api.py#291)(self):**
Remove all index.lock files. If a previous run of git crashed, bot was
reset, etc... we might end up with leftover index.lock files.
&mdash; **def [checkout](/recipes/recipe_modules/gclient/api.py#251)(self, gclient_config=None, revert=RevertOnTryserver, inject_parent_got_revision=True, extra_sync_flags=None, \*\*kwargs):**
Return a step generator function for gclient checkouts.
&emsp; **@staticmethod**<br>&mdash; **def [config\_to\_pythonish](/recipes/recipe_modules/gclient/api.py#139)(cfg):**
&mdash; **def [diff\_deps](/recipes/recipe_modules/gclient/api.py#367)(self, cwd):**
&mdash; **def [get\_config\_defaults](/recipes/recipe_modules/gclient/api.py#133)(self):**
&mdash; **def [get\_gerrit\_patch\_root](/recipes/recipe_modules/gclient/api.py#298)(self, gclient_config=None):**
Returns local path to the repo where gerrit patch will be applied.
If there is no patch, returns None.
If patch is specified, but such repo is not found among configured solutions
or repo_path_map, returns name of the first solution. This is done solely
for backward compatibility with existing tests.
Please do not rely on this logic in new code.
Instead, properly map a repository to a local path using repo_path_map.
TODO(nodir): remove this. Update all recipe tests to specify a git_repo
matching the recipe.
&mdash; **def [get\_repo\_path](/recipes/recipe_modules/gclient/api.py#325)(self, repo_url, gclient_config=None):**
Returns local path to the repo checkout given its url.
Consults cfg.repo_path_map and fallbacks to urls in configured solutions.
Returns None if not found.
&emsp; **@staticmethod**<br>&mdash; **def [got\_revision\_reverse\_mapping](/recipes/recipe_modules/gclient/api.py#144)(cfg):**
Returns the merged got_revision_reverse_mapping.
Returns (dict): A mapping from property name -> project name. It merges the
values of the deprecated got_revision_mapping and the new
got_revision_reverse_mapping.
&mdash; **def [inject\_parent\_got\_revision](/recipes/recipe_modules/gclient/api.py#226)(self, gclient_config=None, override=False):**
Match gclient config to build revisions obtained from build_properties.
Args:
gclient_config (gclient config object) - The config to manipulate. A value
of None manipulates the module's built-in config (self.c).
override (bool) - If True, will forcibly set revision and custom_vars
even if the config already contains values for them.
&mdash; **def [resolve\_revision](/recipes/recipe_modules/gclient/api.py#161)(self, revision):**
&mdash; **def [runhooks](/recipes/recipe_modules/gclient/api.py#285)(self, args=None, name='runhooks', \*\*kwargs):**
&mdash; **def [set\_patch\_repo\_revision](/recipes/recipe_modules/gclient/api.py#355)(self, gclient_config=None):**
Updates config revision corresponding to patched project.
Useful for bot_update only, as this is the only consumer of gclient's config
revision map. This doesn't overwrite the revision if it was already set.
&emsp; **@spec_alias.deleter**<br>&mdash; **def [spec\_alias](/recipes/recipe_modules/gclient/api.py#129)(self):**
&mdash; **def [sync](/recipes/recipe_modules/gclient/api.py#166)(self, cfg, extra_sync_flags=None, \*\*kwargs):**
&emsp; **@use_mirror.setter**<br>&mdash; **def [use\_mirror](/recipes/recipe_modules/gclient/api.py#116)(self, val):**
### *recipe_modules* / [gerrit](/recipes/recipe_modules/gerrit)
[DEPS](/recipes/recipe_modules/gerrit/__init__.py#3): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [recipe\_engine/time][recipe_engine/recipe_modules/time]
PYTHON_VERSION_COMPATIBILITY: PY2+3
#### **class [GerritApi](/recipes/recipe_modules/gerrit/api.py#7)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
Module for interact with Gerrit endpoints
&mdash; **def [\_\_call\_\_](/recipes/recipe_modules/gerrit/api.py#14)(self, name, cmd, infra_step=True, \*\*kwargs):**
Wrapper for easy calling of gerrit_utils steps.
&mdash; **def [abandon\_change](/recipes/recipe_modules/gerrit/api.py#256)(self, host, change, message=None, name=None, step_test_data=None):**
&mdash; **def [call\_raw\_api](/recipes/recipe_modules/gerrit/api.py#31)(self, host, path, method=None, body=None, accept_statuses=None, name=None, \*\*kwargs):**
Call an arbitrary Gerrit API that returns a JSON response.
Returns:
The JSON response data.
&mdash; **def [create\_gerrit\_branch](/recipes/recipe_modules/gerrit/api.py#60)(self, host, project, branch, commit, \*\*kwargs):**
Creates a new branch from given project and commit
Returns:
The ref of the branch created
&mdash; **def [create\_gerrit\_tag](/recipes/recipe_modules/gerrit/api.py#79)(self, host, project, tag, commit, \*\*kwargs):**
Creates a new tag at the given commit.
Returns:
The ref of the tag created.
&mdash; **def [get\_change\_description](/recipes/recipe_modules/gerrit/api.py#118)(self, host, change, patchset, timeout=None, step_test_data=None):**
Gets the description for a given CL and patchset.
Args:
host: URL of Gerrit host to query.
change: The change number.
patchset: The patchset number.
Returns:
The description corresponding to given CL and patchset.
&mdash; **def [get\_changes](/recipes/recipe_modules/gerrit/api.py#178)(self, host, query_params, start=None, limit=None, o_params=None, step_test_data=None, \*\*kwargs):**
Queries changes for the given host.
Args:
* host: URL of Gerrit host to query.
* query_params: Query parameters as list of (key, value) tuples to form a
query as documented here:
https://gerrit-review.googlesource.com/Documentation/user-search.html#search-operators
* start: How many changes to skip (starting with the most recent).
* limit: Maximum number of results to return.
* o_params: A list of additional output specifiers, as documented here:
https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#list-changes
* step_test_data: Optional mock test data for the underlying gerrit client.
Returns:
A list of change dicts as documented here:
https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#list-changes
&mdash; **def [get\_gerrit\_branch](/recipes/recipe_modules/gerrit/api.py#100)(self, host, project, branch, \*\*kwargs):**
Gets a branch from given project and commit
Returns:
The revision of the branch
&mdash; **def [get\_related\_changes](/recipes/recipe_modules/gerrit/api.py#220)(self, host, change, revision='current', step_test_data=None):**
Queries related changes for a given host, change, and revision.
Args:
* host: URL of Gerrit host to query.
* change: The change-id of the change to get related changes for as
documented here:
https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#change-id
* revision: The revision-id of the revision to get related changes for as
documented here:
https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#revision-id
This defaults to current, which names the most recent patch set.
* step_test_data: Optional mock test data for the underlying gerrit client.
Returns:
A related changes dictionary as documented here:
https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#related-changes-info
&mdash; **def [get\_revision\_info](/recipes/recipe_modules/gerrit/api.py#137)(self, host, change, patchset, timeout=None, step_test_data=None):**
Returns the info for a given patchset of a given change.
Args:
host: Gerrit host to query.
change: The change number.
patchset: The patchset number.
Returns:
A dict for the target revision as documented here:
https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#list-changes
&mdash; **def [move\_changes](/recipes/recipe_modules/gerrit/api.py#294)(self, host, project, from_branch, to_branch, step_test_data=None):**
&mdash; **def [set\_change\_label](/recipes/recipe_modules/gerrit/api.py#276)(self, host, change, label_name, label_value, name=None, step_test_data=None):**
&mdash; **def [update\_files](/recipes/recipe_modules/gerrit/api.py#318)(self, host, project, branch, new_contents_by_file_path, commit_msg, params=frozenset(['status=NEW']), cc_list=frozenset([]), submit=False, submit_later=False, step_test_data_create_change=None, step_test_data_submit_change=None):**
Update a set of files by creating and submitting a Gerrit CL.
Args:
* host: URL of Gerrit host to name.
* project: Gerrit project name, e.g. chromium/src.
* branch: The branch to land the change, e.g. main
* new_contents_by_file_path: Dict of the new contents with file path as
the key.
* commit_msg: Description to add to the CL.
* params: A list of additional ChangeInput specifiers, with format
'key=value'.
* cc_list: A list of addresses to notify.
* submit: Should land this CL instantly.
* submit_later: If this change has related CLs, we may want to commit
them in a chain. So only set Bot-Commit+1, making it ready for
submit together. Ignored if submit is True.
* step_test_data_create_change: Optional mock test data for the step
create gerrit change.
* step_test_data_submit_change: Optional mock test data for the step
submit gerrit change.
Returns:
A ChangeInfo dictionary as documented here:
https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#create-change
Or if the change is submitted, here:
https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#submit-change
### *recipe_modules* / [git](/recipes/recipe_modules/git)
[DEPS](/recipes/recipe_modules/git/__init__.py#3): [depot\_tools](#recipe_modules-depot_tools), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/runtime][recipe_engine/recipe_modules/runtime], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
#### **class [GitApi](/recipes/recipe_modules/git/api.py#19)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
&mdash; **def [\_\_call\_\_](/recipes/recipe_modules/git/api.py#22)(self, \*args, \*\*kwargs):**
Returns a git command step.
&mdash; **def [bundle\_create](/recipes/recipe_modules/git/api.py#382)(self, bundle_path, rev_list_args=None, \*\*kwargs):**
Runs 'git bundle create' on a Git repository.
Args:
* bundle_path (Path): The path of the output bundle.
* refs (list): The list of refs to include in the bundle. If None, all
refs in the Git checkout will be bundled.
* kwargs: Forwarded to '__call__'.
&mdash; **def [cat\_file\_at\_commit](/recipes/recipe_modules/git/api.py#42)(self, file_path, commit_hash, remote_name=None, \*\*kwargs):**
Outputs the contents of a file at a given revision.
&mdash; **def [checkout](/recipes/recipe_modules/git/api.py#116)(self, url, ref=None, dir_path=None, recursive=False, submodules=True, submodule_update_force=False, keep_paths=None, step_suffix=None, curl_trace_file=None, raise_on_failure=True, set_got_revision=False, remote_name=None, display_fetch_size=None, file_name=None, submodule_update_recursive=True, use_git_cache=False, progress=True, tags=False):**
Performs a full git checkout and returns sha1 of checked out revision.
Args:
* url (str): url of remote repo to use as upstream
* ref (str): ref to fetch and check out
* dir_path (Path): optional directory to clone into
* recursive (bool): whether to recursively fetch submodules or not
* submodules (bool): whether to sync and update submodules or not
* submodule_update_force (bool): whether to update submodules with --force
* keep_paths (iterable of strings): paths to ignore during git-clean;
paths are gitignore-style patterns relative to checkout_path.
* step_suffix (str): suffix to add to a each step name
* curl_trace_file (Path): if not None, dump GIT_CURL_VERBOSE=1 trace to that
file. Useful for debugging git issue reproducible only on bots. It has
a side effect of all stderr output of 'git fetch' going to that file.
* raise_on_failure (bool): if False, ignore errors during fetch or checkout.
* set_got_revision (bool): if True, resolves HEAD and sets got_revision
property.
* remote_name (str): name of the git remote to use
* display_fetch_size (bool): if True, run `git count-objects` before and
after fetch and display delta. Adds two more steps. Defaults to False.
* file_name (str): optional path to a single file to checkout.
* submodule_update_recursive (bool): if True, updates submodules
recursively.
* use_git_cache (bool): if True, git cache will be used for this checkout.
WARNING, this is EXPERIMENTAL!!! This wasn't tested with:
* submodules
* since origin url is modified
to a local path, may cause problem with scripts that do
"git fetch origin" or "git push origin".
* arbitrary refs such refs/whatever/not-fetched-by-default-to-cache
progress (bool): whether to show progress for fetch or not
* tags (bool): Also fetch tags.
Returns: If the checkout was successful, this returns the commit hash of
the checked-out-repo. Otherwise this returns None.
&mdash; **def [config\_get](/recipes/recipe_modules/git/api.py#351)(self, prop_name, \*\*kwargs):**
Returns git config output.
Args:
* prop_name: (str) The name of the config property to query.
* kwargs: Forwarded to '__call__'.
Returns: (str) The Git config output, or None if no output was generated.
&mdash; **def [count\_objects](/recipes/recipe_modules/git/api.py#50)(self, previous_result=None, raise_on_failure=False, \*\*kwargs):**
Returns `git count-objects` result as a dict.
Args:
* previous_result (dict): the result of previous count_objects call.
If passed, delta is reported in the log and step text.
* raise_on_failure (bool): if True, an exception will be raised if the
operation fails. Defaults to False.
Returns:
A dict of count-object values, or None if count-object run failed.
&mdash; **def [fetch\_tags](/recipes/recipe_modules/git/api.py#36)(self, remote_name=None, \*\*kwargs):**
Fetches all tags from the remote.
&mdash; **def [get\_remote\_url](/recipes/recipe_modules/git/api.py#370)(self, remote_name=None, \*\*kwargs):**
Returns the remote Git repository URL, or None.
Args:
* remote_name: (str) The name of the remote to query, defaults to 'origin'.
* kwargs: Forwarded to '__call__'.
Returns: (str) The URL of the remote Git repository, or None.
&mdash; **def [get\_timestamp](/recipes/recipe_modules/git/api.py#322)(self, commit='HEAD', test_data=None, \*\*kwargs):**
Find and return the timestamp of the given commit.
&mdash; **def [new\_branch](/recipes/recipe_modules/git/api.py#395)(self, branch, name=None, upstream=None, upstream_current=False, \*\*kwargs):**
Runs git new-branch on a Git repository, to be used before git cl
upload.
Args:
* branch (str): new branch name, which must not yet exist.
* name (str): step name.
* upstream (str): to origin/main.
* upstream_current (bool): whether to use '--upstream_current'.
* kwargs: Forwarded to '__call__'.
&mdash; **def [number](/recipes/recipe_modules/git/api.py#426)(self, commitrefs=None, test_values=None):**
Computes the generation number of some commits.
Args:
* commitrefs (list[str]): A list of commit references. If none are
provided, the generation number for HEAD will be retrieved.
* test_values (list[str]): A list of numbers to use as the return
value during tests. It is an error if the length of the list
does not match the number of commitrefs (1 if commitrefs is not
provided).
Returns:
A list of strings containing the generation numbers of the commits.
If non-empty commitrefs was provided, the order of the returned
numbers will correspond to the order of the provided commitrefs.
&mdash; **def [rebase](/recipes/recipe_modules/git/api.py#331)(self, name_prefix, branch, dir_path, remote_name=None, \*\*kwargs):**
Runs rebase HEAD onto branch
Args:
* name_prefix (str): a prefix used for the step names
* branch (str): a branch name or a hash to rebase onto
* dir_path (Path): directory to clone into
* remote_name (str): the remote name to rebase from if not origin
### *recipe_modules* / [git\_cl](/recipes/recipe_modules/git_cl)
[DEPS](/recipes/recipe_modules/git_cl/__init__.py#3): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
#### **class [GitClApi](/recipes/recipe_modules/git_cl/api.py#9)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
&mdash; **def [get\_description](/recipes/recipe_modules/git_cl/api.py#23)(self, patch_url=None, \*\*kwargs):**
*** note
**DEPRECATED**. Consider using gerrit.get_change_description instead.
***
&mdash; **def [issue](/recipes/recipe_modules/git_cl/api.py#48)(self, \*\*kwargs):**
&mdash; **def [set\_description](/recipes/recipe_modules/git_cl/api.py#31)(self, description, patch_url=None, \*\*kwargs):**
&mdash; **def [upload](/recipes/recipe_modules/git_cl/api.py#41)(self, message, upload_args=None, \*\*kwargs):**
### *recipe_modules* / [gitiles](/recipes/recipe_modules/gitiles)
[DEPS](/recipes/recipe_modules/gitiles/__init__.py#7): [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [recipe\_engine/url][recipe_engine/recipe_modules/url]
PYTHON_VERSION_COMPATIBILITY: PY2+3
#### **class [Gitiles](/recipes/recipe_modules/gitiles/api.py#18)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
Module for polling a git repository using the Gitiles web interface.
&mdash; **def [canonicalize\_repo\_url](/recipes/recipe_modules/gitiles/api.py#231)(self, repo_url):**
Returns a canonical form of repo_url. If not recognized, returns as is.
&mdash; **def [commit\_log](/recipes/recipe_modules/gitiles/api.py#121)(self, url, commit, step_name=None, attempts=None):**
Returns: (dict) the Gitiles commit log structure for a given commit.
Args:
* url (str): The base repository URL.
* commit (str): The commit hash.
* step_name (str): If not None, override the step name.
* attempts (int): Number of times to try the request before failing.
&mdash; **def [download\_archive](/recipes/recipe_modules/gitiles/api.py#171)(self, repository_url, destination, revision='refs/heads/main'):**
Downloads an archive of the repo and extracts it to `destination`.
If the gitiles server attempts to provide a tarball with paths which escape
`destination`, this function will extract all valid files and then
raise StepFailure with an attribute `StepFailure.gitiles_skipped_files`
containing the names of the files that were skipped.
Args:
* repository_url (str): Full URL to the repository
* destination (Path): Local path to extract the archive to. Must not exist
prior to this call.
* revision (str): The ref or revision in the repo to download. Defaults to
'refs/heads/main'.
&mdash; **def [download\_file](/recipes/recipe_modules/gitiles/api.py#137)(self, repository_url, file_path, branch='main', step_name=None, attempts=None, \*\*kwargs):**
Downloads raw file content from a Gitiles repository.
Args:
* repository_url (str): Full URL to the repository.
* branch (str): Branch of the repository.
* file_path (str): Relative path to the file from the repository root.
* step_name (str): Custom name for this step (optional).
* attempts (int): Number of times to try the request before failing.
Returns:
Raw file content.
&mdash; **def [log](/recipes/recipe_modules/gitiles/api.py#75)(self, url, ref, limit=0, cursor=None, step_name=None, attempts=None, \*\*kwargs):**
Returns the most recent commits under the given ref with properties.
Args:
* url (str): URL of the remote repository.
* ref (str): Name of the desired ref (see Gitiles.refs).
* limit (int): Number of commits to limit the fetching to.
Gitiles does not return all commits in one call; instead paging is
used. 0 implies to return whatever first gerrit responds with.
Otherwise, paging will be used to fetch at least this many
commits, but all fetched commits will be returned.
* cursor (str or None): The paging cursor used to fetch the next page.
* step_name (str): Custom name for this step (optional).
Returns:
A tuple of (commits, cursor).
Commits are a list of commits (as Gitiles dict structure) in reverse
chronological order. The number of commits may be higher than limit
argument.
Cursor can be used for subsequent calls to log for paging. If None,
signals that there are no more commits to fetch.
&mdash; **def [parse\_repo\_url](/recipes/recipe_modules/gitiles/api.py#220)(self, repo_url):**
Returns (host, project) pair.
Returns (None, None) if repo_url is not recognized.
&mdash; **def [refs](/recipes/recipe_modules/gitiles/api.py#63)(self, url, step_name='refs', attempts=None):**
Returns a list of refs in the remote repository.
&mdash; **def [unparse\_repo\_url](/recipes/recipe_modules/gitiles/api.py#227)(self, host, project):**
Generates a Gitiles repo URL. See also parse_repo_url.
### *recipe_modules* / [gsutil](/recipes/recipe_modules/gsutil)
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
[DEPS](/recipes/recipe_modules/gsutil/__init__.py#5): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
#### **class [GSUtilApi](/recipes/recipe_modules/gsutil/api.py#10)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
&mdash; **def [\_\_call\_\_](/recipes/recipe_modules/gsutil/api.py#21)(self, cmd, name=None, use_retry_wrapper=True, version=None, parallel_upload=False, multithreaded=False, infra_step=True, \*\*kwargs):**
A step to run arbitrary gsutil commands.
On LUCI this should automatically use the ambient task account credentials.
On Buildbot, this assumes that gsutil authentication environment variables
(AWS_CREDENTIAL_FILE and BOTO_CONFIG) are already set, though if you want to
set them to something else you can always do so using the env={} kwarg.
Note also that gsutil does its own wildcard processing, so wildcards are
valid in file-like portions of the cmd. See 'gsutil help wildcards'.
Args:
* cmd (List[str|Path]) - Arguments to pass to gsutil. Include gsutil-level
options first (see 'gsutil help options').
* name (str) - Name of the step to use. Defaults to the first non-flag
token in the cmd.
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
&mdash; **def [cat](/recipes/recipe_modules/gsutil/api.py#115)(self, url, args=None, \*\*kwargs):**
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
&emsp; **@contextlib.contextmanager**<br>&mdash; **def [configure\_gsutil](/recipes/recipe_modules/gsutil/api.py#169)(self, \*\*kwargs):**
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
Temporarily configures the behavior of gsutil.
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
For the duration of its context, this method will temporarily append a
custom Boto file to the BOTO_PATH env var without overwriting bbagent's
BOTO_CONFIG. See https://cloud.google.com/storage/docs/boto-gsutil for
possible configurations.
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
Args:
kwargs: Every keyword arg is treated as config line in the temp Boto file.
&mdash; **def [copy](/recipes/recipe_modules/gsutil/api.py#129)(self, source_bucket, source, dest_bucket, dest, args=None, link_name='gsutil.copy', metadata=None, unauthenticated_url=False, \*\*kwargs):**
&mdash; **def [download](/recipes/recipe_modules/gsutil/api.py#101)(self, bucket, source, dest, args=None, \*\*kwargs):**
&mdash; **def [download\_url](/recipes/recipe_modules/gsutil/api.py#108)(self, url, dest, args=None, \*\*kwargs):**
&emsp; **@property**<br>&mdash; **def [gsutil\_py\_path](/recipes/recipe_modules/gsutil/api.py#17)(self):**
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
&mdash; **def [list](/recipes/recipe_modules/gsutil/api.py#148)(self, url, args=None, \*\*kwargs):**
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
&mdash; **def [remove\_url](/recipes/recipe_modules/gsutil/api.py#162)(self, url, args=None, \*\*kwargs):**
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
&mdash; **def [signurl](/recipes/recipe_modules/gsutil/api.py#155)(self, private_key_file, bucket, dest, args=None, \*\*kwargs):**
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
&mdash; **def [stat](/recipes/recipe_modules/gsutil/api.py#122)(self, url, args=None, \*\*kwargs):**
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
&mdash; **def [upload](/recipes/recipe_modules/gsutil/api.py#84)(self, source, bucket, dest, args=None, link_name='gsutil.upload', metadata=None, unauthenticated_url=False, \*\*kwargs):**
### *recipe_modules* / [osx\_sdk](/recipes/recipe_modules/osx_sdk)
[DEPS](/recipes/recipe_modules/osx_sdk/__init__.py#7): [recipe\_engine/cipd][recipe_engine/recipe_modules/cipd], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/step][recipe_engine/recipe_modules/step], [recipe\_engine/version][recipe_engine/recipe_modules/version]
PYTHON_VERSION_COMPATIBILITY: PY2+3
The `osx_sdk` module provides safe functions to access a semi-hermetic
XCode installation.
Available only to Google-run bots.
#### **class [OSXSDKApi](/recipes/recipe_modules/osx_sdk/api.py#37)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
API for using OS X SDK distributed via CIPD.
&emsp; **@contextmanager**<br>&mdash; **def [\_\_call\_\_](/recipes/recipe_modules/osx_sdk/api.py#63)(self, kind):**
Sets up the XCode SDK environment.
Is a no-op on non-mac platforms.
This will deploy the helper tool and the XCode.app bundle at
`[START_DIR]/cache/osx_sdk`.
To avoid machines rebuilding these on every run, set up a named cache in
your cr-buildbucket.cfg file like:
caches: {
# Cache for mac_toolchain tool and XCode.app
name: "osx_sdk"
path: "osx_sdk"
}
If you have builders which e.g. use a non-current SDK, you can give them
a uniqely named cache:
caches: {
# Cache for N-1 version mac_toolchain tool and XCode.app
name: "osx_sdk_old"
path: "osx_sdk"
}
Similarly, if you have mac and iOS builders you may want to distinguish the
cache name by adding '_ios' to it. However, if you're sharing the same bots
for both mac and iOS, consider having a single cache and just always
fetching the iOS version. This will lead to lower overall disk utilization
and should help to reduce cache thrashing.
Usage:
with api.osx_sdk('mac'):
# sdk with mac build bits
with api.osx_sdk('ios'):
# sdk with mac+iOS build bits
Args:
kind ('mac'|'ios'): How the SDK should be configured. iOS includes the
base XCode distribution, as well as the iOS simulators (which can be
quite large).
Raises:
StepFailure or InfraFailure.
&mdash; **def [initialize](/recipes/recipe_modules/osx_sdk/api.py#48)(self):**
### *recipe_modules* / [presubmit](/recipes/recipe_modules/presubmit)
[DEPS](/recipes/recipe_modules/presubmit/__init__.py#13): [bot\_update](#recipe_modules-bot_update), [depot\_tools](#recipe_modules-depot_tools), [gclient](#recipe_modules-gclient), [git](#recipe_modules-git), [tryserver](#recipe_modules-tryserver), [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/cq][recipe_engine/recipe_modules/cq], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/resultdb][recipe_engine/recipe_modules/resultdb], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
#### **class [PresubmitApi](/recipes/recipe_modules/presubmit/api.py#16)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
&mdash; **def [\_\_call\_\_](/recipes/recipe_modules/presubmit/api.py#28)(self, \*args, \*\*kwargs):**
Returns a presubmit step.
&mdash; **def [execute](/recipes/recipe_modules/presubmit/api.py#120)(self, bot_update_step, skip_owners=False, run_all=False):**
Runs presubmit and sets summary markdown if applicable.
Args:
* bot_update_step: the StepResult from a previously executed bot_update step.
* skip_owners: a boolean indicating whether Owners checks should be skipped.
Returns:
a RawResult object, suitable for being returned from RunSteps.
&mdash; **def [prepare](/recipes/recipe_modules/presubmit/api.py#68)(self, root_solution_revision=None):**
Sets up a presubmit run.
This includes:
- setting up the checkout w/ bot_update
- locally committing the applied patch
- running hooks, if requested
This expects the gclient configuration to already have been set.
Args:
root_solution_revision: revision of the root solution
Returns:
the StepResult from the bot_update step.
&emsp; **@property**<br>&mdash; **def [presubmit\_support\_path](/recipes/recipe_modules/presubmit/api.py#24)(self):**
### *recipe_modules* / [tryserver](/recipes/recipe_modules/tryserver)
[DEPS](/recipes/recipe_modules/tryserver/__init__.py#7): [gerrit](#recipe_modules-gerrit), [git](#recipe_modules-git), [git\_cl](#recipe_modules-git_cl), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/led][recipe_engine/recipe_modules/led], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
#### **class [TryserverApi](/recipes/recipe_modules/tryserver/api.py#27)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
&emsp; **@property**<br>&mdash; **def [constants](/recipes/recipe_modules/tryserver/api.py#51)(self):**
&emsp; **@property**<br>&mdash; **def [gerrit\_change](/recipes/recipe_modules/tryserver/api.py#56)(self):**
Returns current gerrit change, if there is exactly one.
Returns a self.m.buildbucket.common_pb2.GerritChange or None.
&emsp; **@property**<br>&mdash; **def [gerrit\_change\_fetch\_ref](/recipes/recipe_modules/tryserver/api.py#157)(self):**
Returns gerrit patch ref, e.g. "refs/heads/45/12345/6, or None.
Populated iff gerrit_change is populated.
&emsp; **@property**<br>&mdash; **def [gerrit\_change\_number](/recipes/recipe_modules/tryserver/api.py#175)(self):**
Returns gerrit change patchset, e.g. 12345 for a patch ref of
"refs/heads/45/12345/6".
Populated iff gerrit_change is populated. Returns None if not populated.
&emsp; **@property**<br>&mdash; **def [gerrit\_change\_owner](/recipes/recipe_modules/tryserver/api.py#88)(self):**
Returns owner of the current Gerrit CL.
Populated iff gerrit_change is populated.
Is a dictionary with keys like "name".
&emsp; **@property**<br>&mdash; **def [gerrit\_change\_repo\_host](/recipes/recipe_modules/tryserver/api.py#72)(self):**
Returns the host of the gitiles repo of the current Gerrit CL.
Populated iff gerrit_change is populated.
&emsp; **@property**<br>&mdash; **def [gerrit\_change\_repo\_project](/recipes/recipe_modules/tryserver/api.py#80)(self):**
Returns the project of the gitiles repo of the current Gerrit CL.
Populated iff gerrit_change is populated.
&emsp; **@property**<br>&mdash; **def [gerrit\_change\_repo\_url](/recipes/recipe_modules/tryserver/api.py#64)(self):**
Returns canonical URL of the gitiles repo of the current Gerrit CL.
Populated iff gerrit_change is populated.
&emsp; **@property**<br>&mdash; **def [gerrit\_change\_review\_url](/recipes/recipe_modules/tryserver/api.py#97)(self):**
Returns the review URL for the active patchset.
&emsp; **@property**<br>&mdash; **def [gerrit\_change\_target\_ref](/recipes/recipe_modules/tryserver/api.py#166)(self):**
Returns gerrit change destination ref, e.g. "refs/heads/main".
Populated iff gerrit_change is populated.
&emsp; **@property**<br>&mdash; **def [gerrit\_patchset\_number](/recipes/recipe_modules/tryserver/api.py#187)(self):**
Returns gerrit change patchset, e.g. 6 for a patch ref of
"refs/heads/45/12345/6".
Populated iff gerrit_change is populated Returns None if not populated..
&mdash; **def [get\_files\_affected\_by\_patch](/recipes/recipe_modules/tryserver/api.py#233)(self, patch_root, report_files_via_property=None, \*\*kwargs):**
Returns list of paths to files affected by the patch.
Args:
* patch_root: path relative to api.path['root'], usually obtained from
api.gclient.get_gerrit_patch_root().
* report_files_via_property: name of the output property to report the
list of the files. If None (default), do not report.
Returned paths will be relative to to api.path['root'].
&mdash; **def [get\_footer](/recipes/recipe_modules/tryserver/api.py#375)(self, tag, patch_text=None):**
Gets a specific tag from a CL description
&mdash; **def [get\_footers](/recipes/recipe_modules/tryserver/api.py#337)(self, patch_text=None):**
Retrieves footers from the patch description.
footers are machine readable tags embedded in commit messages. See
git-footers documentation for more information.
&mdash; **def [initialize](/recipes/recipe_modules/tryserver/api.py#42)(self):**
&emsp; **@property**<br>&mdash; **def [is\_gerrit\_issue](/recipes/recipe_modules/tryserver/api.py#204)(self):**
Returns true iff the properties exist to match a Gerrit issue.
&emsp; **@property**<br>&mdash; **def [is\_patch\_in\_git](/recipes/recipe_modules/tryserver/api.py#214)(self):**
&emsp; **@property**<br>&mdash; **def [is\_tryserver](/recipes/recipe_modules/tryserver/api.py#199)(self):**
Returns true iff we have a change to check out.
&mdash; **def [normalize\_footer\_name](/recipes/recipe_modules/tryserver/api.py#383)(self, footer):**
&mdash; **def [require\_is\_tryserver](/recipes/recipe_modules/tryserver/api.py#220)(self):**
&mdash; **def [set\_change](/recipes/recipe_modules/tryserver/api.py#386)(self, change):**
Set the gerrit change for this module.
Args:
* change: a self.m.buildbucket.common_pb2.GerritChange.
&mdash; **def [set\_compile\_failure\_tryjob\_result](/recipes/recipe_modules/tryserver/api.py#298)(self):**
Mark the tryjob result as a compile failure.
&mdash; **def [set\_invalid\_test\_results\_tryjob\_result](/recipes/recipe_modules/tryserver/api.py#310)(self):**
Mark the tryjob result as having invalid test results.
This means we run some tests, but the results were not valid
(e.g. no list of specific test cases that failed, or too many
tests failing, etc).
&mdash; **def [set\_patch\_failure\_tryjob\_result](/recipes/recipe_modules/tryserver/api.py#294)(self):**
Mark the tryjob result as failure to apply the patch.
&mdash; **def [set\_subproject\_tag](/recipes/recipe_modules/tryserver/api.py#272)(self, subproject_tag):**
Adds a subproject tag to the build.
This can be used to distinguish between builds that execute different steps
depending on what was patched, e.g. blink vs. pure chromium patches.
&mdash; **def [set\_test\_expired\_tryjob\_result](/recipes/recipe_modules/tryserver/api.py#327)(self):**
Mark the tryjob result as a test expiration.
This means a test task expired and was never scheduled, most likely due to
lack of capacity.
&mdash; **def [set\_test\_failure\_tryjob\_result](/recipes/recipe_modules/tryserver/api.py#302)(self):**
Mark the tryjob result as a test failure.
This means we started running actual tests (not prerequisite steps
like checkout or compile), and some of these tests have failed.
&mdash; **def [set\_test\_timeout\_tryjob\_result](/recipes/recipe_modules/tryserver/api.py#319)(self):**
Mark the tryjob result as a test timeout.
This means tests were scheduled but didn't finish executing within the
timeout.
&emsp; **@property**<br>&mdash; **def [valid\_footers](/recipes/recipe_modules/tryserver/api.py#47)(self):**
### *recipe_modules* / [windows\_sdk](/recipes/recipe_modules/windows_sdk)
[DEPS](/recipes/recipe_modules/windows_sdk/__init__.py#7): [recipe\_engine/cipd][recipe_engine/recipe_modules/cipd], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
The `windows_sdk` module provides safe functions to access a hermetic
Microsoft Visual Studio installation which is derived from Chromium's MSVC
toolchain.
See (internal):
* go/chromium-msvc-toolchain
* go/windows-sdk-cipd-update
Available only on Google-run bots.
#### **class [WindowsSDKApi](/recipes/recipe_modules/windows_sdk/api.py#22)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
API for using Windows SDK distributed via CIPD.
&emsp; **@contextmanager**<br>&mdash; **def [\_\_call\_\_](/recipes/recipe_modules/windows_sdk/api.py#32)(self, path=None, version=None, enabled=True, target_arch='x64'):**
Sets up the SDK environment when enabled.
Args:
* path (path): Path to a directory where to install the SDK
(default is '[CACHE]/windows_sdk')
* version (str): CIPD version of the SDK
(default is set via $infra/windows_sdk.version property)
* enabled (bool): Whether the SDK should be used or not.
* target_arch (str): 'x86' or 'x64'.
Yields:
If enabled, yields SDKPaths object with paths to well-known roots within
the deployed bundle:
* win_sdk - a Path to the root of the extracted Windows SDK.
* dia_sdk - a Path to the root of the extracted Debug Interface Access
SDK.
Raises:
StepFailure or InfraFailure.
## Recipes
### *recipes* / [bot\_update:examples/full](/recipes/recipe_modules/bot_update/examples/full.py)
[DEPS](/recipes/recipe_modules/bot_update/examples/full.py#9): [bot\_update](#recipe_modules-bot_update), [gclient](#recipe_modules-gclient), [gerrit](#recipe_modules-gerrit), [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/runtime][recipe_engine/recipe_modules/runtime]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/bot_update/examples/full.py#27)(api):**
### *recipes* / [bot\_update:tests/do\_not\_retry\_patch\_failures\_in\_cq](/recipes/recipe_modules/bot_update/tests/do_not_retry_patch_failures_in_cq.py)
[DEPS](/recipes/recipe_modules/bot_update/tests/do_not_retry_patch_failures_in_cq.py#9): [bot\_update](#recipe_modules-bot_update), [gclient](#recipe_modules-gclient), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/cq][recipe_engine/recipe_modules/cq], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/bot_update/tests/do_not_retry_patch_failures_in_cq.py#19)(api):**
### *recipes* / [bot\_update:tests/download\_topics](/recipes/recipe_modules/bot_update/tests/download_topics.py)
[DEPS](/recipes/recipe_modules/bot_update/tests/download_topics.py#9): [bot\_update](#recipe_modules-bot_update), [gclient](#recipe_modules-gclient), [recipe\_engine/json][recipe_engine/recipe_modules/json]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/bot_update/tests/download_topics.py#16)(api):**
### *recipes* / [bot\_update:tests/ensure\_checkout](/recipes/recipe_modules/bot_update/tests/ensure_checkout.py)
[DEPS](/recipes/recipe_modules/bot_update/tests/ensure_checkout.py#9): [bot\_update](#recipe_modules-bot_update), [gclient](#recipe_modules-gclient), [recipe\_engine/json][recipe_engine/recipe_modules/json]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/bot_update/tests/ensure_checkout.py#16)(api):**
### *recipes* / [depot\_tools:examples/full](/recipes/recipe_modules/depot_tools/examples/full.py)
[DEPS](/recipes/recipe_modules/depot_tools/examples/full.py#7): [depot\_tools](#recipe_modules-depot_tools), [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/runtime][recipe_engine/recipe_modules/runtime], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/depot_tools/examples/full.py#16)(api):**
### *recipes* / [gclient:examples/full](/recipes/recipe_modules/gclient/examples/full.py)
[DEPS](/recipes/recipe_modules/gclient/examples/full.py#7): [gclient](#recipe_modules-gclient), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/gclient/examples/full.py#66)(api):**
### *recipes* / [gclient:tests/diff\_deps](/recipes/recipe_modules/gclient/tests/diff_deps.py)
[DEPS](/recipes/recipe_modules/gclient/tests/diff_deps.py#9): [gclient](#recipe_modules-gclient), [recipe\_engine/assertions][recipe_engine/recipe_modules/assertions], [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/gclient/tests/diff_deps.py#20)(api):**
### *recipes* / [gclient:tests/patch\_project](/recipes/recipe_modules/gclient/tests/patch_project.py)
[DEPS](/recipes/recipe_modules/gclient/tests/patch_project.py#11): [gclient](#recipe_modules-gclient), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/gclient/tests/patch_project.py#20)(api):**
### *recipes* / [gclient:tests/sync\_failure](/recipes/recipe_modules/gclient/tests/sync_failure.py)
[DEPS](/recipes/recipe_modules/gclient/tests/sync_failure.py#9): [gclient](#recipe_modules-gclient), [recipe\_engine/path][recipe_engine/recipe_modules/path]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/gclient/tests/sync_failure.py#15)(api):**
### *recipes* / [gerrit:examples/full](/recipes/recipe_modules/gerrit/examples/full.py)
[DEPS](/recipes/recipe_modules/gerrit/examples/full.py#7): [gerrit](#recipe_modules-gerrit), [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/gerrit/examples/full.py#13)(api):**
### *recipes* / [git:examples/full](/recipes/recipe_modules/git/examples/full.py)
[DEPS](/recipes/recipe_modules/git/examples/full.py#7): [git](#recipe_modules-git), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/git/examples/full.py#20)(api):**
### *recipes* / [git:tests/number](/recipes/recipe_modules/git/tests/number.py)
[DEPS](/recipes/recipe_modules/git/tests/number.py#9): [git](#recipe_modules-git), [recipe\_engine/assertions][recipe_engine/recipe_modules/assertions], [recipe\_engine/properties][recipe_engine/recipe_modules/properties]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/git/tests/number.py#16)(api):**
### *recipes* / [git\_cl:examples/full](/recipes/recipe_modules/git_cl/examples/full.py)
[DEPS](/recipes/recipe_modules/git_cl/examples/full.py#11): [git\_cl](#recipe_modules-git_cl), [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/git_cl/examples/full.py#19)(api):**
### *recipes* / [gitiles:examples/full](/recipes/recipe_modules/gitiles/examples/full.py)
[DEPS](/recipes/recipe_modules/gitiles/examples/full.py#7): [gitiles](#recipe_modules-gitiles), [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/gitiles/examples/full.py#16)(api):**
### *recipes* / [gitiles:tests/parse\_repo\_url](/recipes/recipe_modules/gitiles/tests/parse_repo_url.py)
[DEPS](/recipes/recipe_modules/gitiles/tests/parse_repo_url.py#7): [gitiles](#recipe_modules-gitiles), [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/gitiles/tests/parse_repo_url.py#14)(api):**
Add a method to gsutil recipe_module to disable multiprocessing on mac In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
### *recipes* / [gsutil:examples/custom\_boto](/recipes/recipe_modules/gsutil/examples/custom_boto.py)
[DEPS](/recipes/recipe_modules/gsutil/examples/custom_boto.py#10): [gsutil](#recipe_modules-gsutil), [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/gsutil/examples/custom_boto.py#21)(api, boto_configs):**
### *recipes* / [gsutil:examples/full](/recipes/recipe_modules/gsutil/examples/full.py)
[DEPS](/recipes/recipe_modules/gsutil/examples/full.py#7): [gsutil](#recipe_modules-gsutil), [recipe\_engine/path][recipe_engine/recipe_modules/path]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/gsutil/examples/full.py#13)(api):**
Move things around in a loop!
### *recipes* / [osx\_sdk:examples/full](/recipes/recipe_modules/osx_sdk/examples/full.py)
[DEPS](/recipes/recipe_modules/osx_sdk/examples/full.py#7): [osx\_sdk](#recipe_modules-osx_sdk), [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/osx_sdk/examples/full.py#15)(api):**
### *recipes* / [presubmit:examples/full](/recipes/recipe_modules/presubmit/examples/full.py)
[DEPS](/recipes/recipe_modules/presubmit/examples/full.py#7): [presubmit](#recipe_modules-presubmit), [recipe\_engine/json][recipe_engine/recipe_modules/json]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/presubmit/examples/full.py#13)(api):**
### *recipes* / [presubmit:tests/execute](/recipes/recipe_modules/presubmit/tests/execute.py)
[DEPS](/recipes/recipe_modules/presubmit/tests/execute.py#12): [gclient](#recipe_modules-gclient), [presubmit](#recipe_modules-presubmit), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/cq][recipe_engine/recipe_modules/cq], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/runtime][recipe_engine/recipe_modules/runtime]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/presubmit/tests/execute.py#25)(api):**
### *recipes* / [presubmit:tests/prepare](/recipes/recipe_modules/presubmit/tests/prepare.py)
[DEPS](/recipes/recipe_modules/presubmit/tests/prepare.py#11): [gclient](#recipe_modules-gclient), [presubmit](#recipe_modules-presubmit), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/runtime][recipe_engine/recipe_modules/runtime]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/presubmit/tests/prepare.py#28)(api, patch_project, patch_repository_url):**
### *recipes* / [tryserver:examples/full](/recipes/recipe_modules/tryserver/examples/full.py)
[DEPS](/recipes/recipe_modules/tryserver/examples/full.py#7): [gerrit](#recipe_modules-gerrit), [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/tryserver/examples/full.py#22)(api):**
### *recipes* / [tryserver:tests/gerrit\_change\_fetch\_ref\_timeout](/recipes/recipe_modules/tryserver/tests/gerrit_change_fetch_ref_timeout.py)
[DEPS](/recipes/recipe_modules/tryserver/tests/gerrit_change_fetch_ref_timeout.py#9): [gerrit](#recipe_modules-gerrit), [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/properties][recipe_engine/recipe_modules/properties]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/tryserver/tests/gerrit_change_fetch_ref_timeout.py#17)(api):**
### *recipes* / [tryserver:tests/gerrit\_change\_owner](/recipes/recipe_modules/tryserver/tests/gerrit_change_owner.py)
[DEPS](/recipes/recipe_modules/tryserver/tests/gerrit_change_owner.py#9): [gerrit](#recipe_modules-gerrit), [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/tryserver/tests/gerrit_change_owner.py#16)(api):**
### *recipes* / [tryserver:tests/gerrit\_change\_target\_ref](/recipes/recipe_modules/tryserver/tests/gerrit_change_target_ref.py)
[DEPS](/recipes/recipe_modules/tryserver/tests/gerrit_change_target_ref.py#9): [gerrit](#recipe_modules-gerrit), [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/tryserver/tests/gerrit_change_target_ref.py#18)(api):**
### *recipes* / [tryserver:tests/get\_files\_affected\_by\_patch](/recipes/recipe_modules/tryserver/tests/get_files_affected_by_patch.py)
[DEPS](/recipes/recipe_modules/tryserver/tests/get_files_affected_by_patch.py#9): [tryserver](#recipe_modules-tryserver), [recipe\_engine/assertions][recipe_engine/recipe_modules/assertions], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/tryserver/tests/get_files_affected_by_patch.py#18)(api):**
### *recipes* / [tryserver:tests/require\_is\_tryserver](/recipes/recipe_modules/tryserver/tests/require_is_tryserver.py)
[DEPS](/recipes/recipe_modules/tryserver/tests/require_is_tryserver.py#12): [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/properties][recipe_engine/recipe_modules/properties]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/tryserver/tests/require_is_tryserver.py#19)(api):**
### *recipes* / [windows\_sdk:examples/full](/recipes/recipe_modules/windows_sdk/examples/full.py)
[DEPS](/recipes/recipe_modules/windows_sdk/examples/full.py#7): [windows\_sdk](#recipe_modules-windows_sdk), [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/windows_sdk/examples/full.py#16)(api):**
[recipe_engine/recipe_modules/assertions]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-assertions
[recipe_engine/recipe_modules/buildbucket]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-buildbucket
[recipe_engine/recipe_modules/cipd]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-cipd
[recipe_engine/recipe_modules/commit_position]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-commit_position
[recipe_engine/recipe_modules/context]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-context
[recipe_engine/recipe_modules/cq]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-cq
[recipe_engine/recipe_modules/file]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-file
[recipe_engine/recipe_modules/json]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-json
[recipe_engine/recipe_modules/led]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-led
[recipe_engine/recipe_modules/milo]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-milo
[recipe_engine/recipe_modules/path]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-path
[recipe_engine/recipe_modules/platform]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-platform
[recipe_engine/recipe_modules/properties]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-properties
[recipe_engine/recipe_modules/raw_io]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-raw_io
[recipe_engine/recipe_modules/resultdb]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-resultdb
[recipe_engine/recipe_modules/runtime]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-runtime
[recipe_engine/recipe_modules/step]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-step
[recipe_engine/recipe_modules/time]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-time
[recipe_engine/recipe_modules/url]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-url
[recipe_engine/recipe_modules/version]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/README.recipes.md#recipe_modules-version
[recipe_engine/wkt/RecipeApi]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/b63398006f3dbdd5e32a2bdc3d623e1cc0e34a1d/recipe_engine/recipe_api.py#886