aiden
/
depot_tools
mirror of https://chromium.googlesource.com/chromium/tools/depot_tools
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

bot_update recipe: Upload source manifest

Browse Source 
Bug: 772529
Change-Id: I3f833740ddfbc71be42b140c99dcd4bbfeed54fe
Reviewed-on: https://chromium-review.googlesource.com/706071
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
changes/71/706071/17
Ryan Tseng 8 years ago committed by Commit Bot
parent e6e0641b15
commit 037f9b80bb
8 changed files with 363 additions and 22 deletions
Show Stats Download Patch File Download Diff File

11
recipes/README.recipes.md
Unescape Escape View File

@ -38,7 +38,7 @@
### *recipe_modules* / [bot\_update](/recipes/recipe_modules/bot_update) ### *recipe_modules* / [bot\_update](/recipes/recipe_modules/bot_update)
[DEPS](/recipes/recipe_modules/bot_update/__init__.py#1): [depot\_tools](#recipe_modules-depot_tools), [gclient](#recipe_modules-gclient), [gerrit](#recipe_modules-gerrit), [rietveld](#recipe_modules-rietveld), [tryserver](#recipe_modules-tryserver), [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step] [DEPS](/recipes/recipe_modules/bot_update/__init__.py#1): [depot\_tools](#recipe_modules-depot_tools), [gclient](#recipe_modules-gclient), [gerrit](#recipe_modules-gerrit), [rietveld](#recipe_modules-rietveld), [tryserver](#recipe_modules-tryserver), [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/source\_manifest][recipe_engine/recipe_modules/source_manifest], [recipe\_engine/step][recipe_engine/recipe_modules/step]
Recipe module to ensure a checkout is consistent on a bot. Recipe module to ensure a checkout is consistent on a bot.
@ -50,12 +50,12 @@ Wrapper for easy calling of bot_update.
&mdash; **def [apply\_gerrit\_ref](/recipes/recipe_modules/bot_update/api.py#49)(self, root, gerrit_no_reset=False, gerrit_no_rebase_patch_ref=False, gerrit_repo=None, gerrit_ref=None, step_name='apply_gerrit', \*\*kwargs):** &mdash; **def [apply\_gerrit\_ref](/recipes/recipe_modules/bot_update/api.py#49)(self, root, gerrit_no_reset=False, gerrit_no_rebase_patch_ref=False, gerrit_repo=None, gerrit_ref=None, step_name='apply_gerrit', \*\*kwargs):**
&mdash; **def [deapply\_patch](/recipes/recipe_modules/bot_update/api.py#434)(self, bot_update_step):** &mdash; **def [deapply\_patch](/recipes/recipe_modules/bot_update/api.py#445)(self, bot_update_step):**
Deapplies a patch, taking care of DEPS and solution revisions properly. Deapplies a patch, taking care of DEPS and solution revisions properly.
&mdash; **def [ensure\_checkout](/recipes/recipe_modules/bot_update/api.py#71)(self, gclient_config=None, suffix=None, patch=True, update_presentation=True, patch_root=None, no_shallow=False, with_branch_heads=False, with_tags=False, refs=None, patch_oauth2=False, oauth2_json=False, use_site_config_creds=True, clobber=False, root_solution_revision=None, rietveld=None, issue=None, patchset=None, gerrit_no_reset=False, gerrit_no_rebase_patch_ref=False, disable_syntax_validation=False, \*\*kwargs):** &mdash; **def [ensure\_checkout](/recipes/recipe_modules/bot_update/api.py#71)(self, gclient_config=None, suffix=None, patch=True, update_presentation=True, patch_root=None, no_shallow=False, with_branch_heads=False, with_tags=False, refs=None, patch_oauth2=False, oauth2_json=False, use_site_config_creds=True, clobber=False, root_solution_revision=None, rietveld=None, issue=None, patchset=None, gerrit_no_reset=False, gerrit_no_rebase_patch_ref=False, disable_syntax_validation=False, manifest_name=None, \*\*kwargs):**
Args: Args:
use_site_config_creds: If the oauth2 credentials are in the buildbot use_site_config_creds: If the oauth2 credentials are in the buildbot
@ -71,8 +71,10 @@ Args:
disable_syntax_validation: (legacy) Disables syntax validation for DEPS. disable_syntax_validation: (legacy) Disables syntax validation for DEPS.
Needed as migration paths for recipes dealing with older revisions, Needed as migration paths for recipes dealing with older revisions,
such as bisect. such as bisect.
manifest_name: The name of the manifest to upload to LogDog. This must
be unique for the whole build.
&mdash; **def [get\_project\_revision\_properties](/recipes/recipe_modules/bot_update/api.py#411)(self, project_name, gclient_config=None):** &mdash; **def [get\_project\_revision\_properties](/recipes/recipe_modules/bot_update/api.py#422)(self, project_name, gclient_config=None):**
Returns all property names used for storing the checked-out revision of Returns all property names used for storing the checked-out revision of
a given project. a given project.
@ -839,6 +841,7 @@ Move things around in a loop!
[recipe_engine/recipe_modules/properties]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-properties [recipe_engine/recipe_modules/properties]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-properties
[recipe_engine/recipe_modules/python]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-python [recipe_engine/recipe_modules/python]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-python
[recipe_engine/recipe_modules/raw_io]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-raw_io [recipe_engine/recipe_modules/raw_io]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-raw_io
[recipe_engine/recipe_modules/source_manifest]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-source_manifest
[recipe_engine/recipe_modules/step]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-step [recipe_engine/recipe_modules/step]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-step
[recipe_engine/recipe_modules/url]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-url [recipe_engine/recipe_modules/url]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-url
[recipe_engine/wkt/RecipeApi]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/recipe_engine/recipe_api.py#992 [recipe_engine/wkt/RecipeApi]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/recipe_engine/recipe_api.py#992

1
recipes/recipe_modules/bot_update/__init__.py
Unescape Escape View File

@ -9,6 +9,7 @@ DEPS = [
'recipe_engine/properties', 'recipe_engine/properties',
'recipe_engine/python', 'recipe_engine/python',
'recipe_engine/raw_io', 'recipe_engine/raw_io',
'recipe_engine/source_manifest',
'recipe_engine/step', 'recipe_engine/step',
'rietveld', 'rietveld',
'tryserver', 'tryserver',

15
recipes/recipe_modules/bot_update/api.py
Unescape Escape View File

@ -77,7 +77,8 @@ class BotUpdateApi(recipe_api.RecipeApi):
root_solution_revision=None, rietveld=None, issue=None, root_solution_revision=None, rietveld=None, issue=None,
patchset=None, gerrit_no_reset=False, patchset=None, gerrit_no_reset=False,
gerrit_no_rebase_patch_ref=False, gerrit_no_rebase_patch_ref=False,
disable_syntax_validation=False, **kwargs): disable_syntax_validation=False, manifest_name=None,
**kwargs):
""" """
Args: Args:
use_site_config_creds: If the oauth2 credentials are in the buildbot use_site_config_creds: If the oauth2 credentials are in the buildbot
@ -93,6 +94,8 @@ class BotUpdateApi(recipe_api.RecipeApi):
disable_syntax_validation: (legacy) Disables syntax validation for DEPS. disable_syntax_validation: (legacy) Disables syntax validation for DEPS.
Needed as migration paths for recipes dealing with older revisions, Needed as migration paths for recipes dealing with older revisions,
such as bisect. such as bisect.
manifest_name: The name of the manifest to upload to LogDog. This must
be unique for the whole build.
""" """
refs = refs or [] refs = refs or []
# We can re-use the gclient spec from the gclient module, since all the # We can re-use the gclient spec from the gclient module, since all the
@ -261,7 +264,6 @@ class BotUpdateApi(recipe_api.RecipeApi):
root, first_sln, reverse_rev_map, self._fail_patch, root, first_sln, reverse_rev_map, self._fail_patch,
fixed_revisions=fixed_revisions) fixed_revisions=fixed_revisions)
# Add suffixes to the step name, if specified.
name = 'bot_update' name = 'bot_update'
if not patch: if not patch:
name += ' (without patch)' name += ' (without patch)'
@ -296,6 +298,15 @@ class BotUpdateApi(recipe_api.RecipeApi):
step_text = result['step_text'] step_text = result['step_text']
step_result.presentation.step_text = step_text step_result.presentation.step_text = step_text
# Export the step results as a Source Manifest to LogDog.
if manifest_name:
if not patch:
# The param "patched" is purely cosmetic to mean "if false, this
# bot_update run exists purely to unpatch an existing patch".
manifest_name += '_unpatched'
self.m.source_manifest.set_json_manifest(
manifest_name, result.get('manifest', {}))
# Set the "checkout" path for the main solution. # Set the "checkout" path for the main solution.
# This is used by the Chromium module to figure out where to look for # This is used by the Chromium module to figure out where to look for
# the checkout. # the checkout.

148
recipes/recipe_modules/bot_update/examples/full.expected/with_manifest_name.json
Unescape Escape View File

@ -0,0 +1,148 @@
[
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[depot_tools::bot_update]/resources/bot_update.py",
"--spec-path",
"cache_dir = '[GIT_CACHE]'\nsolutions = [{'deps_file': '.DEPS.git', 'managed': True, 'name': 'src', 'url': 'https://chromium.googlesource.com/chromium/src.git'}]",
"--patch_root",
"src",
"--revision_mapping_file",
"{\"got_angle_revision\": \"src/third_party/angle\", \"got_cr_revision\": \"src\", \"got_revision\": \"src\", \"got_v8_revision\": \"src/v8\"}",
"--git-cache-dir",
"[GIT_CACHE]",
"--cleanup-dir",
"[CLEANUP]/bot_update",
"--output_json",
"/path/to/tmp/json",
"--revision",
"src@HEAD",
"--disable-syntax-validation"
],
"env_prefixes": {
"PATH": [
"RECIPE_PACKAGE_REPO[depot_tools]"
]
},
"infra_step": true,
"name": "bot_update",
"~followup_annotations": [
"@@@STEP_TEXT@Some step text@@@",
"@@@STEP_LOG_LINE@json.output@{@@@",
"@@@STEP_LOG_LINE@json.output@ \"did_run\": true, @@@",
"@@@STEP_LOG_LINE@json.output@ \"fixed_revisions\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": \"HEAD\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"manifest\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/third_party/angle\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/third_party/angle.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/v8\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/v8.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_failure\": false, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"properties\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision_cp\": \"refs/heads/master@{#297276}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision\": \"801ada225ddc271c132c3a35f03975671d43e399\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision_cp\": \"refs/heads/master@{#43426}\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"step_text\": \"Some step text\"@@@",
"@@@STEP_LOG_LINE@json.output@}@@@",
"@@@STEP_LOG_END@json.output@@@",
"@@@SET_BUILD_PROPERTY@got_angle_revision@\"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@SET_BUILD_PROPERTY@got_angle_revision_cp@\"refs/heads/master@{#297276}\"@@@",
"@@@SET_BUILD_PROPERTY@got_cr_revision@\"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@SET_BUILD_PROPERTY@got_cr_revision_cp@\"refs/heads/master@{#170242}\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision@\"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision_cp@\"refs/heads/master@{#170242}\"@@@",
"@@@SET_BUILD_PROPERTY@got_v8_revision@\"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@SET_BUILD_PROPERTY@got_v8_revision_cp@\"refs/heads/master@{#43426}\"@@@"
]
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[depot_tools::bot_update]/resources/bot_update.py",
"--spec-path",
"cache_dir = '[GIT_CACHE]'\nsolutions = [{'deps_file': '.DEPS.git', 'managed': True, 'name': 'src', 'url': 'https://chromium.googlesource.com/chromium/src.git'}]",
"--patch_root",
"src",
"--revision_mapping_file",
"{\"got_angle_revision\": \"src/third_party/angle\", \"got_cr_revision\": \"src\", \"got_revision\": \"src\", \"got_v8_revision\": \"src/v8\"}",
"--git-cache-dir",
"[GIT_CACHE]",
"--cleanup-dir",
"[CLEANUP]/bot_update",
"--output_json",
"/path/to/tmp/json",
"--revision",
"src@f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9"
],
"env_prefixes": {
"PATH": [
"RECIPE_PACKAGE_REPO[depot_tools]"
]
},
"infra_step": true,
"name": "bot_update (without patch)",
"~followup_annotations": [
"@@@STEP_TEXT@Some step text@@@",
"@@@STEP_LOG_LINE@json.output@{@@@",
"@@@STEP_LOG_LINE@json.output@ \"did_run\": true, @@@",
"@@@STEP_LOG_LINE@json.output@ \"fixed_revisions\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"manifest\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/third_party/angle\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/third_party/angle.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/v8\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/v8.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_failure\": false, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"properties\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision_cp\": \"refs/heads/master@{#297276}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision\": \"801ada225ddc271c132c3a35f03975671d43e399\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision_cp\": \"refs/heads/master@{#43426}\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"step_text\": \"Some step text\"@@@",
"@@@STEP_LOG_LINE@json.output@}@@@",
"@@@STEP_LOG_END@json.output@@@"
]
},
{
"name": "$result",
"recipe_result": null,
"status_code": 0
}
]

82
recipes/recipe_modules/bot_update/examples/full.expected/with_manifest_name_no_patch.json
Unescape Escape View File

@ -0,0 +1,82 @@
[
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[depot_tools::bot_update]/resources/bot_update.py",
"--spec-path",
"cache_dir = '[GIT_CACHE]'\nsolutions = [{'deps_file': '.DEPS.git', 'managed': True, 'name': 'src', 'url': 'https://chromium.googlesource.com/chromium/src.git'}]",
"--patch_root",
"src",
"--revision_mapping_file",
"{\"got_angle_revision\": \"src/third_party/angle\", \"got_cr_revision\": \"src\", \"got_revision\": \"src\", \"got_v8_revision\": \"src/v8\"}",
"--git-cache-dir",
"[GIT_CACHE]",
"--cleanup-dir",
"[CLEANUP]/bot_update",
"--output_json",
"/path/to/tmp/json",
"--revision",
"src@HEAD",
"--disable-syntax-validation"
],
"env_prefixes": {
"PATH": [
"RECIPE_PACKAGE_REPO[depot_tools]"
]
},
"infra_step": true,
"name": "bot_update (without patch)",
"~followup_annotations": [
"@@@STEP_TEXT@Some step text@@@",
"@@@STEP_LOG_LINE@json.output@{@@@",
"@@@STEP_LOG_LINE@json.output@ \"did_run\": true, @@@",
"@@@STEP_LOG_LINE@json.output@ \"fixed_revisions\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": \"HEAD\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"manifest\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/third_party/angle\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/third_party/angle.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/v8\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/v8.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_failure\": false, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"properties\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision_cp\": \"refs/heads/master@{#297276}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision\": \"801ada225ddc271c132c3a35f03975671d43e399\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision_cp\": \"refs/heads/master@{#43426}\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"step_text\": \"Some step text\"@@@",
"@@@STEP_LOG_LINE@json.output@}@@@",
"@@@STEP_LOG_END@json.output@@@",
"@@@SET_BUILD_PROPERTY@got_angle_revision@\"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@SET_BUILD_PROPERTY@got_angle_revision_cp@\"refs/heads/master@{#297276}\"@@@",
"@@@SET_BUILD_PROPERTY@got_cr_revision@\"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@SET_BUILD_PROPERTY@got_cr_revision_cp@\"refs/heads/master@{#170242}\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision@\"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision_cp@\"refs/heads/master@{#170242}\"@@@",
"@@@SET_BUILD_PROPERTY@got_v8_revision@\"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@SET_BUILD_PROPERTY@got_v8_revision_cp@\"refs/heads/master@{#43426}\"@@@"
]
},
{
"name": "$result",
"recipe_result": null,
"status_code": 0
}
]

11
recipes/recipe_modules/bot_update/examples/full.py
Unescape Escape View File

@ -49,6 +49,7 @@ def RunSteps(api):
gerrit_no_reset = True if api.properties.get('gerrit_no_reset') else False gerrit_no_reset = True if api.properties.get('gerrit_no_reset') else False
gerrit_no_rebase_patch_ref = bool( gerrit_no_rebase_patch_ref = bool(
api.properties.get('gerrit_no_rebase_patch_ref')) api.properties.get('gerrit_no_rebase_patch_ref'))
manifest_name = api.properties.get('manifest_name')
if api.properties.get('test_apply_gerrit_ref'): if api.properties.get('test_apply_gerrit_ref'):
prop2arg = { prop2arg = {
@ -79,7 +80,8 @@ def RunSteps(api):
suffix=suffix, suffix=suffix,
gerrit_no_reset=gerrit_no_reset, gerrit_no_reset=gerrit_no_reset,
gerrit_no_rebase_patch_ref=gerrit_no_rebase_patch_ref, gerrit_no_rebase_patch_ref=gerrit_no_rebase_patch_ref,
disable_syntax_validation=True) disable_syntax_validation=True,
manifest_name=manifest_name)
if patch: if patch:
api.bot_update.deapply_patch(bot_update_step) api.bot_update.deapply_patch(bot_update_step)
@ -89,6 +91,13 @@ def GenTests(api):
patch=False, patch=False,
revision='abc' revision='abc'
) )
yield api.test('with_manifest_name_no_patch') + api.properties(
manifest_name='checkout',
patch=False
)
yield api.test('with_manifest_name') + api.properties(
manifest_name='checkout'
)
yield api.test('buildbot') + api.properties( yield api.test('buildbot') + api.properties(
path_config='buildbot', path_config='buildbot',
patch=False, patch=False,

85
recipes/recipe_modules/bot_update/resources/bot_update.py
Unescape Escape View File

@ -49,9 +49,6 @@ COMMIT_FOOTER_ENTRY_RE = re.compile(r'([^:]+):\s*(.*)')
COMMIT_POSITION_FOOTER_KEY = 'Cr-Commit-Position' COMMIT_POSITION_FOOTER_KEY = 'Cr-Commit-Position'
COMMIT_ORIGINAL_POSITION_FOOTER_KEY = 'Cr-Original-Commit-Position' COMMIT_ORIGINAL_POSITION_FOOTER_KEY = 'Cr-Original-Commit-Position'
# Regular expression to parse gclient's revinfo entries.
REVINFO_RE = re.compile(r'^([^:]+):\s+([^@]+)@(.+)$')
# Copied from scripts/recipes/chromium.py. # Copied from scripts/recipes/chromium.py.
GOT_REVISION_MAPPINGS = { GOT_REVISION_MAPPINGS = {
@ -364,18 +361,75 @@ def gclient_revinfo():
return call_gclient('revinfo', '-a') or '' return call_gclient('revinfo', '-a') or ''
def create_manifest(): def normalize_git_url(url):
manifest = {} """Normalize a git url to be consistent.
output = gclient_revinfo()
for line in output.strip().splitlines(): This recognizes urls to the googlesoruce.com domain. It ensures that
match = REVINFO_RE.match(line.strip()) the url:
if match: * Do not end in .git
manifest[match.group(1)] = { * Do not contain /a/ in their path.
'repository': match.group(2), """
'revision': match.group(3), try:
p = urlparse.urlparse(url)
except Exception:
# Not a url, just return it back.
return url
if not p.netloc.endswith('.googlesource.com'):
# Not a googlesource.com URL, can't normalize this, just return as is.
return url
path = p.path
if path.startswith('/a'):
path = path[len('/a'):]
if path.endswith('.git'):
path = path[:-len('.git')]
return 'https://%s%s' % (p.netloc, path)
# TODO(hinoka): Include patch revision.
def create_manifest(gclient_output, patch_root, gerrit_ref):
"""Return the JSONPB equivilent of the source manifest proto.
The source manifest proto is defined here:
https://chromium.googlesource.com/infra/luci/recipes-py/+/master/recipe_engine/source_manifest.proto
This is based off of:
* The gclient_output (from calling gclient.py --output-json) which contains
the directory -> repo:revision mapping.
* Gerrit Patch info which contains info about patched revisions.
We normalize the URLs such that if they are googlesource.com urls, they:
"""
manifest = {
'version': 0, # Currently the only valid version is 0.
}
dirs = {}
if patch_root:
patch_root = patch_root.strip('/') # Normalize directory names.
for directory, info in gclient_output.get('solutions', {}).iteritems():
directory = directory.strip('/') # Normalize the directory name.
# There are two places to the the revision from, we do it in this order:
# 1. In the "revision" field
# 2. At the end of the URL, after @
repo = ''
revision = info.get('revision', '')
# The format of the url is "https://repo.url/blah.git@abcdefabcdef" or
# just "https://repo.url/blah.git"
url_split = info.get('url', '').split('@')
if not revision and len(url_split) == 2:
revision = url_split[1]
if url_split:
repo = normalize_git_url(url_split[0])
if repo:
dirs[directory] = {
'git_checkout': {
'repo_url': repo,
'revision': revision,
}
} }
else: if patch_root == directory:
print "WARNING: Couldn't match revinfo line:\n%s" % line dirs[directory]['git_checkout']['patch_fetch_ref'] = gerrit_ref
manifest['directories'] = dirs
return manifest return manifest
@ -1104,7 +1158,8 @@ def checkout(options, git_slns, specs, revisions, step_text, shallow):
step_text=step_text, step_text=step_text,
fixed_revisions=revisions, fixed_revisions=revisions,
properties=got_revisions, properties=got_revisions,
manifest=create_manifest()) manifest=create_manifest(
gclient_output, options.patch_root, options.gerrit_ref))
def print_debug_info(): def print_debug_info():

32
tests/bot_update_coverage_test.py
Unescape Escape View File

@ -245,6 +245,38 @@ class BotUpdateUnittests(unittest.TestCase):
setattr(os, 'remove', old_os_remove) setattr(os, 'remove', old_os_remove)
self.assertTrue(os.path.join(path, lockfile) in removed) self.assertTrue(os.path.join(path, lockfile) in removed)
def testGenerateManifestsBasic(self):
gclient_output = {
'solutions': {
'breakpad/': {
'revision': None,
'scm': None,
'url': ('https://chromium.googlesource.com/breakpad/breakpad.git' +
'@5f638d532312685548d5033618c8a36f73302d0a')
},
"src/": {
'revision': 'f671d3baeb64d9dba628ad582e867cf1aebc0207',
'scm': None,
'url': 'https://chromium.googlesource.com/a/chromium/src.git'
},
}
}
out = bot_update.create_manifest(gclient_output, None, None)
self.assertEquals(len(out['directories']), 2)
print out
self.assertEquals(
out['directories']['src']['git_checkout']['revision'],
'f671d3baeb64d9dba628ad582e867cf1aebc0207')
self.assertEquals(
out['directories']['src']['git_checkout']['repo_url'],
'https://chromium.googlesource.com/chromium/src')
self.assertEquals(
out['directories']['breakpad']['git_checkout']['revision'],
'5f638d532312685548d5033618c8a36f73302d0a')
self.assertEquals(
out['directories']['breakpad']['git_checkout']['repo_url'],
'https://chromium.googlesource.com/breakpad/breakpad')
if __name__ == '__main__': if __name__ == '__main__':
unittest.main() unittest.main()

Write Preview
Loading…
Cancel
Save