sys.platform doesn't contain major version of aix.
Always return 'aix', instead of 'aix6' or 'aix7'.
R=gavinmak@google.com, kimstephanie@google.com
Change-Id: I6d2075be808503731996df7bf17f23527ebdf445
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5999752
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This CL adds a hint on how to log into GCS to the 401 error message
displayed to the user.
Bug: 381634957
Change-Id: Id72b5030c018b28a7297b7ff3f279485c6da4d9e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6056844
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Alan Cutter <alancutter@chromium.org>
Sometimes the object name can have more than one dot like
20.11.0/f9a337cfa0e2b92d3e5c671c26b454bd8e99769e.
Currently, the dotfiles would be .20_.*
This change makes it so that the dotfiles would be
.20_11_0_f9a337cfa0e2b92d3e5c671c26b454bd8e99769e.*
Bug: b/328065301
Change-Id: I043a4b5fd56e48b046cfdc9a2fde3204b1bb638c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5491861
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
The downloaded file may be a binary that needs to be executed
Bug: b/328065301, b/336843583
Change-Id: I0f6c959cbe3d5cf7e2717da72c98c51bbe98f7b3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5484340
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Allows multiple objects to use the same directory path.
This requires that each object has its own unique hash file and
unique migration file name. All object names must be unique.
Also update download_from_google_storage to check for the unique
migration file name.
Remove ConvertToGcs smoke tests since we're not converting
any git <-> gcs deps.
Example:
```
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'not llvm_force_head_revision',
'bucket': 'chromium-browser-clang',
'objects': [
{
'object_name': 'Linux_x64/clang-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz',
'sha256sum': '7b33138d8592199f97d132242d7b3e10f460c5c9655d49a3ad3767218fba7a77',
'size_bytes': 50212876,
},
{
'object_name': 'Linux_x64/llvmobjdump-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz',
'sha256sum': '14d669650cd212eb0ccb8c34a9e655338dfdee66fe2ecdaa517f6bd607c09a97',
'size_bytes': 5302312,
},
]
},
```
TODO: update .gitignore to search for *_is_first_class and *_hash
Bug: b/324418194
Change-Id: I89d34b06ee24f4c1aa316cd51530ad078e823143
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5420793
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
If is_first_class_gcs file exists with contents '1'
- GCS hooks will download content and then overwrite the file with '0'
- First class GCS deps will download as necessary by checking the hash
file.
If there is no is_first_class_gcs file or its content is '0' for a GCS hook:
- Hook will run normally and, if content is downloaded, an is_first_class_gcs
file will be written with '0'
If there is no is_first_class_gcs file or its content is '0' for a first
class gcs dep:
- Content will be downloaded and is_first_class_gcs will be set to '1'
Verified locally with migrating and unmigrating a GCS hook.
TODO: update other GCS hook scripts in chromium/src to check migration
file.
Bug: b/324418194
Change-Id: Ida8541cb70839b86e99115bcabfc38bc87e40136
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5368250
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
The check is intended to ensure paths don't traverse outside
the extracted directory. However, the check was too strict: it
banned all links, even relative links that still point inside the
target directory. This CL relaxes the requirement to allow valid
links. This is required to allow library symlinks for the
instrumented libraries. Examples:
libpcre.so -> libpcre.so.3.13.3
libpcre.so.3 -> libpcre.so.3.13.3
libpixman-1.so -> libpixman-1.so.0.38.4
libpixman-1.so.0 -> libpixman-1.so.0.38.4
libpng16.so -> libpng16.so.16.37.0
libpng16.so.16 -> libpng16.so.16.37.0
Bug: 320564950
Change-Id: I2aae18b86b1f1cc3d73a1b80c06d757af782f700
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5369132
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Auto-Submit: Thomas Anderson <thomasanderson@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Also take out GCS calling logic from download_google_storage and
into call_google_storage.
GCS deps look like:
'src/third_party/node/linux': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-nodejs/20.11.0',
'object_name': '46795170ff5df9831955f163f6966abde581c8af',
'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
},
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-browser-clang',
'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
},
Example directory for src/third_party/node/linux after gclient sync:
- tar_file.gz is the downloaded file from GCS.
- node_linux_x64/ is extracted in its path.
- `hash` contains the sha of GCS filename.
```
chromium/src/ ->
third_party/node/linux/ ->
hash, tar_file.gz, node_linux_x64/
```
Bug: b/324418194
Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
All __future__ imports (unicode_literals, print_function) are already
mandatory in py3. Also remove an outdated py2 comment in
presubmit_canned_checks.py
Bug: 1475402
Change-Id: I27cf6a8268f6dd1081f22af782c4c29a975376ae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4867135
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.
Reformatted using:
files=( $(
git ls-tree -r --name-only HEAD | \
grep -Ev -e '^(third_party|recipes)/' | \
grep '\.py$';
git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"
The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py
These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py
Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
python3 is the only supported version of python in depot_tools.
Bug: 1475402
Change-Id: I71db631b5556525dd4932134679c70cacd205a90
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4824616
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Instruct user to use gsutil.py config if they get 401 from gsutil
operation. Setting project ID is no longer applicable.
R=aravindvasudev@google.com
Bug: 1433494
Change-Id: Ibbb22a9b5bb3a92e35f689371701eb051b38c202
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4442760
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.
Bug:1262286
Change-Id: I4b8f8fc521386419a3121bbb07edc8ac83170a94
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413679
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This reverts commit 22bf605bb6.
Reason for revert: breaks gclient sync
Original change's description:
> Use pylint 2.7 for depot_tools
>
> This includes a few fixes for specific errors, and disables several new
> warnings introduced in this version, in order to allow for an incremental migration.
>
> Bug:1262286
> Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Bug: 1262286
Change-Id: Ieb946073c7886c7bf056ce843a5a48e82becf7a5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413672
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.
Bug:1262286
Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This reverts commit e8ef6259b9.
Reason for revert: https://crbug.com/1240673
High rate of bot_update failures, lots of gsutil defunct
processes.
Original change's description:
> Update gsutil to use gsutil version 4.66, python3
>
> This is a reland of 457736028d
> with following changes:
> * bump version from 4.61 to 4.66, which contains several bugfixes
>
> Original change's description:
> > Reland "Reland "Update gsutil to use gsutil version 4.61, python3.""
> >
> > This is a reland of e53a593956
> >
> > Additional bug fixes:
> > * handle race condition in gsutil when creating its directory
> > * limit to one gsutil process on darwin due to bug in python3
> >
> > Original change's description:
> > > Reland "Update gsutil to use gsutil version 4.61, python3."
> > >
> > > This reverts commit af121aeec9.
> > >
> > > Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
> > >
> > > Original change's description:
> > > > Revert "Update gsutil to use gsutil version 4.61, python3."
> > > >
> > > > This reverts commit f059ec9368.
> > > >
> > > > Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
> > > >
> > > > Original change's description:
> > > > > Update gsutil to use gsutil version 4.61, python3.
> > > > >
> > > > > This CL updates the gsutil.py wrapper to download and use
> > > > > v4.61 of GCP's gsutil, which is Python3-compatible.
> > > > >
> > > > > v4.61 appears to be fully self-contained and have all of the
> > > > > packages it needs vendored into it. So, there's no reason to
> > > > > use vpython anymore, and this CL removes that.
> > > > >
> > > > > Also, this CL removes the 'fallback' option to gsutil and
> > > > > the ability to force a version switch, as this should no
> > > > > longer be necessary (it was added for a migration back in 2014
> > > > > but apparently this code was never removed afterwards).
> > > > >
> > > > > This CL also updates download_from_google_storage.py and
> > > > > upload_to_google_storage.py to similarly not have the version flags
> > > > > and to just use regular python3, not vpython3.
> > > > >
> > > > > Bug: 1184108
> > > > > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > > > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > > > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > > > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > > > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> > > >
> > > > Bug: 1184108
> > > > Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> > > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> > > > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> > >
> > > Bug: 1184108
> > > Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
> > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> >
> > Bug: 1184108
> > Change-Id: Ibb5d886fd22e3553521ff8ad6e2b4435844ef972
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2988716
> > Reviewed-by: Dirk Pranke <dpranke@google.com>
> > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > Commit-Queue: Josip Sokcevic <sokcevic@google.com>
>
> Bug: 1184108
> Change-Id: I33787dc75f6e45d6b462706e934d7a2a37703fa7
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3088085
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bug: 1184108, 1240673
Change-Id: I74c3243ea29b99476e09b6ddb49cb052812a1e3e
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3100348
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
This is a reland of 457736028d
with following changes:
* bump version from 4.61 to 4.66, which contains several bugfixes
Original change's description:
> Reland "Reland "Update gsutil to use gsutil version 4.61, python3.""
>
> This is a reland of e53a593956
>
> Additional bug fixes:
> * handle race condition in gsutil when creating its directory
> * limit to one gsutil process on darwin due to bug in python3
>
> Original change's description:
> > Reland "Update gsutil to use gsutil version 4.61, python3."
> >
> > This reverts commit af121aeec9.
> >
> > Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
> >
> > Original change's description:
> > > Revert "Update gsutil to use gsutil version 4.61, python3."
> > >
> > > This reverts commit f059ec9368.
> > >
> > > Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
> > >
> > > Original change's description:
> > > > Update gsutil to use gsutil version 4.61, python3.
> > > >
> > > > This CL updates the gsutil.py wrapper to download and use
> > > > v4.61 of GCP's gsutil, which is Python3-compatible.
> > > >
> > > > v4.61 appears to be fully self-contained and have all of the
> > > > packages it needs vendored into it. So, there's no reason to
> > > > use vpython anymore, and this CL removes that.
> > > >
> > > > Also, this CL removes the 'fallback' option to gsutil and
> > > > the ability to force a version switch, as this should no
> > > > longer be necessary (it was added for a migration back in 2014
> > > > but apparently this code was never removed afterwards).
> > > >
> > > > This CL also updates download_from_google_storage.py and
> > > > upload_to_google_storage.py to similarly not have the version flags
> > > > and to just use regular python3, not vpython3.
> > > >
> > > > Bug: 1184108
> > > > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> > >
> > > Bug: 1184108
> > > Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> > > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> >
> > Bug: 1184108
> > Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
> > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > Commit-Queue: Dirk Pranke <dpranke@google.com>
>
> Bug: 1184108
> Change-Id: Ibb5d886fd22e3553521ff8ad6e2b4435844ef972
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2988716
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bug: 1184108
Change-Id: I33787dc75f6e45d6b462706e934d7a2a37703fa7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3088085
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This reverts commit 457736028d.
Reason for revert: crbug/1227849 - android builders aren't happy,
also a report from user about android not working properly.
Original change's description:
> Reland "Reland "Update gsutil to use gsutil version 4.61, python3.""
>
> This is a reland of e53a593956
>
> Additional bug fixes:
> * handle race condition in gsutil when creating its directory
> * limit to one gsutil process on darwin due to bug in python3
>
> Original change's description:
> > Reland "Update gsutil to use gsutil version 4.61, python3."
> >
> > This reverts commit af121aeec9.
> >
> > Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
> >
> > Original change's description:
> > > Revert "Update gsutil to use gsutil version 4.61, python3."
> > >
> > > This reverts commit f059ec9368.
> > >
> > > Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
> > >
> > > Original change's description:
> > > > Update gsutil to use gsutil version 4.61, python3.
> > > >
> > > > This CL updates the gsutil.py wrapper to download and use
> > > > v4.61 of GCP's gsutil, which is Python3-compatible.
> > > >
> > > > v4.61 appears to be fully self-contained and have all of the
> > > > packages it needs vendored into it. So, there's no reason to
> > > > use vpython anymore, and this CL removes that.
> > > >
> > > > Also, this CL removes the 'fallback' option to gsutil and
> > > > the ability to force a version switch, as this should no
> > > > longer be necessary (it was added for a migration back in 2014
> > > > but apparently this code was never removed afterwards).
> > > >
> > > > This CL also updates download_from_google_storage.py and
> > > > upload_to_google_storage.py to similarly not have the version flags
> > > > and to just use regular python3, not vpython3.
> > > >
> > > > Bug: 1184108
> > > > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> > >
> > > Bug: 1184108
> > > Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> > > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> >
> > Bug: 1184108
> > Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
> > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > Commit-Queue: Dirk Pranke <dpranke@google.com>
>
> Bug: 1184108
> Change-Id: Ibb5d886fd22e3553521ff8ad6e2b4435844ef972
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2988716
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bug: 1184108
Bug: 1227849
Change-Id: Ib4176b94861e9f85caf6ed74050111a07b1e308d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3021641
Reviewed-by: Dirk Pranke <dpranke@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This is a reland of e53a593956
Additional bug fixes:
* handle race condition in gsutil when creating its directory
* limit to one gsutil process on darwin due to bug in python3
Original change's description:
> Reland "Update gsutil to use gsutil version 4.61, python3."
>
> This reverts commit af121aeec9.
>
> Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
>
> Original change's description:
> > Revert "Update gsutil to use gsutil version 4.61, python3."
> >
> > This reverts commit f059ec9368.
> >
> > Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
> >
> > Original change's description:
> > > Update gsutil to use gsutil version 4.61, python3.
> > >
> > > This CL updates the gsutil.py wrapper to download and use
> > > v4.61 of GCP's gsutil, which is Python3-compatible.
> > >
> > > v4.61 appears to be fully self-contained and have all of the
> > > packages it needs vendored into it. So, there's no reason to
> > > use vpython anymore, and this CL removes that.
> > >
> > > Also, this CL removes the 'fallback' option to gsutil and
> > > the ability to force a version switch, as this should no
> > > longer be necessary (it was added for a migration back in 2014
> > > but apparently this code was never removed afterwards).
> > >
> > > This CL also updates download_from_google_storage.py and
> > > upload_to_google_storage.py to similarly not have the version flags
> > > and to just use regular python3, not vpython3.
> > >
> > > Bug: 1184108
> > > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> >
> > Bug: 1184108
> > Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > Commit-Queue: Dirk Pranke <dpranke@google.com>
>
> Bug: 1184108
> Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Dirk Pranke <dpranke@google.com>
Bug: 1184108
Change-Id: Ibb5d886fd22e3553521ff8ad6e2b4435844ef972
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2988716
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This reverts commit e53a593956.
Reason for revert: Speculative revert for build failures on Mac11 bots: crbug.com/1219065.
Original change's description:
> Reland "Update gsutil to use gsutil version 4.61, python3."
>
> This reverts commit af121aeec9.
>
> Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
>
> Original change's description:
> > Revert "Update gsutil to use gsutil version 4.61, python3."
> >
> > This reverts commit f059ec9368.
> >
> > Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
> >
> > Original change's description:
> > > Update gsutil to use gsutil version 4.61, python3.
> > >
> > > This CL updates the gsutil.py wrapper to download and use
> > > v4.61 of GCP's gsutil, which is Python3-compatible.
> > >
> > > v4.61 appears to be fully self-contained and have all of the
> > > packages it needs vendored into it. So, there's no reason to
> > > use vpython anymore, and this CL removes that.
> > >
> > > Also, this CL removes the 'fallback' option to gsutil and
> > > the ability to force a version switch, as this should no
> > > longer be necessary (it was added for a migration back in 2014
> > > but apparently this code was never removed afterwards).
> > >
> > > This CL also updates download_from_google_storage.py and
> > > upload_to_google_storage.py to similarly not have the version flags
> > > and to just use regular python3, not vpython3.
> > >
> > > Bug: 1184108
> > > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> >
> > Bug: 1184108
> > Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > Commit-Queue: Dirk Pranke <dpranke@google.com>
>
> Bug: 1184108
> Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Dirk Pranke <dpranke@google.com>
Bug: 1184108
Change-Id: Ib0757e0873d6e6b39621d50d327d959f0fa43720
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2957473
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
This reverts commit af121aeec9.
Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
Original change's description:
> Revert "Update gsutil to use gsutil version 4.61, python3."
>
> This reverts commit f059ec9368.
>
> Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
>
> Original change's description:
> > Update gsutil to use gsutil version 4.61, python3.
> >
> > This CL updates the gsutil.py wrapper to download and use
> > v4.61 of GCP's gsutil, which is Python3-compatible.
> >
> > v4.61 appears to be fully self-contained and have all of the
> > packages it needs vendored into it. So, there's no reason to
> > use vpython anymore, and this CL removes that.
> >
> > Also, this CL removes the 'fallback' option to gsutil and
> > the ability to force a version switch, as this should no
> > longer be necessary (it was added for a migration back in 2014
> > but apparently this code was never removed afterwards).
> >
> > This CL also updates download_from_google_storage.py and
> > upload_to_google_storage.py to similarly not have the version flags
> > and to just use regular python3, not vpython3.
> >
> > Bug: 1184108
> > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > Commit-Queue: Dirk Pranke <dpranke@google.com>
>
> Bug: 1184108
> Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Dirk Pranke <dpranke@google.com>
Bug: 1184108
Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
This reverts commit f059ec9368.
Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
Original change's description:
> Update gsutil to use gsutil version 4.61, python3.
>
> This CL updates the gsutil.py wrapper to download and use
> v4.61 of GCP's gsutil, which is Python3-compatible.
>
> v4.61 appears to be fully self-contained and have all of the
> packages it needs vendored into it. So, there's no reason to
> use vpython anymore, and this CL removes that.
>
> Also, this CL removes the 'fallback' option to gsutil and
> the ability to force a version switch, as this should no
> longer be necessary (it was added for a migration back in 2014
> but apparently this code was never removed afterwards).
>
> This CL also updates download_from_google_storage.py and
> upload_to_google_storage.py to similarly not have the version flags
> and to just use regular python3, not vpython3.
>
> Bug: 1184108
> Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Dirk Pranke <dpranke@google.com>
Bug: 1184108
Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
This CL updates the gsutil.py wrapper to download and use
v4.61 of GCP's gsutil, which is Python3-compatible.
v4.61 appears to be fully self-contained and have all of the
packages it needs vendored into it. So, there's no reason to
use vpython anymore, and this CL removes that.
Also, this CL removes the 'fallback' option to gsutil and
the ability to force a version switch, as this should no
longer be necessary (it was added for a migration back in 2014
but apparently this code was never removed afterwards).
This CL also updates download_from_google_storage.py and
upload_to_google_storage.py to similarly not have the version flags
and to just use regular python3, not vpython3.
Bug: 1184108
Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
We have vpython wheels for py3 but not for py2 on mac/arm. With this,
download_from_google_storage.py works on arm mac even without the
vpython bypass. (Also, moving from py2 to py3 seems like a good thing
in general.)
Bug: 1205263,1103236
Change-Id: I6cb08b7c56e5e6fc160fb5abc86d68e04a01f8de
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2881917
Auto-Submit: Nico Weber <thakis@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
If download from google storage gets terminated while extracting .tar.gz
archive, it won't be extracted on the next run which results in invalid
workspace. This patches places a flag file if extraction wasn't
completed. Such flag is checked on next execution and forces re-download
if found.
This does not fix currently broken workspaces.
R=dpranke@google.com, ehmladonado@chromium.org
Bug: 1189273
Change-Id: I4047d98c193493d9a0fe20a46043e1f8376e5c7e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2774085
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
xrange doesn't exist in Python 3 and only nominally improves
memory usage on a value as small as MAX_TRIES.
Change-Id: I1e7408a05de880fee293466b1548c7f35e926270
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2246611
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
This was made by running `codespell` and `scspell`
and then checking the results.
Change-Id: I169fd5b40294f83015075b4a899fbca263821f25
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2144602
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Auto-Submit: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
The code in _downloader_worker_thread() does not work with Python 3:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/rakuco/src/depot_tools/download_from_google_storage.py", line 343, in _downloader_worker_thread
elif re.search(r'executable:\s*1', out):
File "/usr/lib/python3.7/re.py", line 183, in search
return _compile(pattern, flags).search(string)
TypeError: cannot use a string pattern on a bytes-like object
Bug: 984182, 1007872
Change-Id: I09df8169d802b010596ac4f34501b4bda805e6f7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1844833
Auto-Submit: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
This throws an exception when running with Python 3:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/rakuco/src/depot_tools/download_from_google_storage.py", line 259, in _downloader_worker_thread
file_url = '%s/%s' % (base_url, input_sha1_sum.decode())
AttributeError: 'str' object has no attribute 'decode'
We are already calling bytes.decode() in enumerate_input(), whose output
is passed to _downloader_worker_thread().
Bug: 984182, 1007872
Change-Id: I55c4835192caf69a8dee9e89ae1bb5531f4bacae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1844832
Auto-Submit: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Scott Lee
Alan Cutter
Stephanie Kim
Tom Anderson
Michael Ershov
Gaby Baghdadi
Gavin Mak
Josip Sokcevic
Mike Frysinger
Joanna Wang
Aravind Vasudevan
Josip Sokcevic
Dirk Pranke
Nico Weber
Linkun Chen
Mirko Bonadei
Eli Ribble
Yuly Novikov
Quinten Yearsley
Takuto Ikuta
Sylvain Defresne
Edward Lesmes
Raphael Kubo da Costa
Edward Lemur
Victor Costan