gclient normally runs hooks in a pseudo terminal which has no
sense of the terminal size. If we want hooks to be able to change
what they output based on the terminal size, we have to explicitly
propagate the terminal size info down through the subprocess
environment.
Change-Id: I08f7c48ef78ea4eb9f5b791abb2a7e5ef8870050
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6161596
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
llvm-build GCS dependency needs to place one GCS object based on
platform. We can't have multiple top-level GCS llvm-build dependencies
as the pathname is used as a key.
This extends object property of GCS dependency to have `condition`.
The `condition` field will be AND with the top level condition.
R=jojwang@google.com, kimstephanie@google.com
Change-Id: I06a6281140a8057d598a6eac2b07c7577f0ea86c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5527474
Reviewed-by: Stephanie Kim <kimstephanie@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
So that it can be shared with code other than scm.
Bug: 339231299
Change-Id: Iba68cc7ebba78c8933598d073861290bb0157c40
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5529883
Auto-Submit: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Importing pipes gets a DeprecationWarning. shlex.quote is equivalent to
pipes.quote. Use that instead.
Bug: 1522874
Change-Id: I18b8598f2e961d58ba71d886921bda9d9ad956ca
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5243707
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Auto-Submit: Gavin Mak <gavinmak@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.
Reformatted using:
files=( $(
git ls-tree -r --name-only HEAD | \
grep -Ev -e '^(third_party|recipes)/' | \
grep '\.py$';
git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"
The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py
These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py
Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
python3 is the only supported version of python in depot_tools.
Bug: 1475402
Change-Id: I17174d7252b5cbf698700333d3cd561c6591d0a1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4809190
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This will improve the ergonomics of `git cl upload`, `git cl
description`, etc. for `vim` users.
I'm sure there are analogues for emacs, vscode, etc, but I'll leave
that as an exercise for a user of those editors :)
R=aravindvasudev@google.com
Change-Id: I194718326bc34092ef8c669a97eae9598592137d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4676559
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Detect the name of unknown operating systems using uname, if available,
since it doesn't append the operating system version.
Change-Id: Idab7bd0db65a8d424ec2fd48f06247405b6649e4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4169240
Auto-Submit: Jonas Termansen <sortie@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
There are other retries (e.g. in git wrapper, gclient_scm) so we
shouldn't be retrying that much in gclient_utils.
In ideal case, we have a dedicated place where retries are determined,
and we make a retry decision based on stderr / exit code.
R=jojwang@google.com
Bug: 1359109
Change-Id: I97daa0d991a7294635e54b7a3d85a349c03c04c6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3871979
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.
Bug:1262286
Change-Id: I4b8f8fc521386419a3121bbb07edc8ac83170a94
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413679
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This reverts commit 22bf605bb6.
Reason for revert: breaks gclient sync
Original change's description:
> Use pylint 2.7 for depot_tools
>
> This includes a few fixes for specific errors, and disables several new
> warnings introduced in this version, in order to allow for an incremental migration.
>
> Bug:1262286
> Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Bug: 1262286
Change-Id: Ieb946073c7886c7bf056ce843a5a48e82becf7a5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413672
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.
Bug:1262286
Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
The (raw_)input Python function used by `gclient_utils.AskForData`
supports additional 'elaborate line editing' [1] [2] features, such as
the ability to use backspace to fix a typo, if the readline module is
loaded. Make sure that readline is loaded by importing it locally in
`gclient_utils.AskForData`.
It does not appear to be possible to write a test that verifies this
behavior - assigning `io.StringIO` to `sys.stdin` appears to cause
(raw_)input to fall back to basic input processing.
[1] https://docs.python.org/2.7/library/functions.html#raw_input
[2] https://docs.python.org/3/library/functions.html#input
Change-Id: Ie1c25b3fa2f3a255b78426c20e47c968cd7198ab
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3306613
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Christian Flach <cmfcmf@chromium.org>
We always want gclient_utils.FileRead() to return a text
string these days, but the existing code didn't work
right for Python2 if you called it explicitly with mode='r'.
R=ehmaldonado@chromium.org
Bug: 1210746
Change-Id: If540746f31308b671f9101334b5cd9848a7e9257
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2930626
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
If git push refs/for/ errors out, inspect stdout message to detect
uploadvalidator rejection. If present, present user a solution.
R=ajp@google.com
Change-Id: I70aee7615f3d905127a31bd5679968ade75c234a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2888475
Reviewed-by: Andy Perelson <ajp@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
As reported in crbug.com/1122671, windows bots can manage to trigger
the isatty conditional even when openpty() isn't provided.
So just force that conditional to false on windows.
Bug: 1122671
Change-Id: I2c85024da40feadd97cde7f718f4e5a4231fe9d6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2386321
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
This was made by running `codespell` and `scspell`
and then checking the results.
Change-Id: I169fd5b40294f83015075b4a899fbca263821f25
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2144602
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Auto-Submit: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
gclient_utils.AskForData will use raw_input on Python 2 and input on Python 3.
Bug: 1063976
Change-Id: I93c059c4e4454d01787716b5a0a2641ad5253f53
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2116370
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Useful when passing information to a subcommand via a temporary file.
Bug: 1051631
Change-Id: I0b8deda921effd24a37109544e1e7ca22e00ba4e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2068942
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Various collections module members were removed in Python 3.9 after being deprecated and moved to collections.abc in earlier Python 3 versions.
For accessing these members alias collections_abc as:
* collections.abc on Python 3
* collections on Python 2
Traceback (most recent call last):
File "C:\Google\depot_tools\gclient.py", line 107, in <module>
import gclient_eval
File "C:\Google\depot_tools\gclient_eval.py", line 11, in <module>
import gclient_utils
File "C:\Google\depot_tools\gclient_utils.py", line 1201, in <module>
class FrozenDict(collections.Mapping):
AttributeError: module 'collections' has no attribute 'Mapping'
Traceback (most recent call last):
File "C:\Google\depot_tools\gclient.py", line 107, in <module>
import gclient_eval
File "C:\Google\depot_tools\gclient_eval.py", line 25, in <module>
class _NodeDict(collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'
Bug: 984182
Change-Id: I7a4417978b93e29397e63764e4570a598c075bc0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2043879
Auto-Submit: Raul Tambre <raul@tambre.ee>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Previously, it returned bytes if the file did not contain valid Unicode data,
and a Unicode string otherwise.
It is confusing to reason about such a function, and no current caller needs
bytes data AFAICT, so make FileRead always return Unicode strings.
Bug: 1009814
Change-Id: I89dd1935e5d4fcaf9af71585b85bda6c47695950
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1880013
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Ran:
vi $(git grep --name-only iteritems | grep -v third_party)
vi $(git grep --name-only itervalues | grep -v third_party)
vi $(git grep --name-only 'print ' | grep -v third_party)
and edited the files quickly with adhoc macros. Then ran in recipes/:
./recipes.py test train
There was only a small subset of files that had been updated to use
six.iteritems() and six.itervalues(). Since the dataset size that is
being used in gclient is small (pretty much always below 200 items),
it's better to just switch to .items() right away and take the temporary
performance hit, so that we don't need to come back to rewrite the code.
Recipe-Nontrivial-Roll: build
Bug: 984182
Change-Id: I5faf11486b66b0d73c9098ab0f2ce1b15a45c53e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1854900
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Marc-Antoine Ruel <maruel@chromium.org>
This is a reland of 5d284fdf48
Always convert the input to bytes and write to sys.stdout (in Python2)
and sys.stdout.buffer (in Python 3).
Original change's description:
> gclient_utils: buffer output as bytestrings in Annotated
>
> In Python 3 byestrings and normal strings can't be concatenated.
> To fix this we buffer as bytestrings in the Annotated wrapper.
> We can't decode to a string because the output might come byte-by-byte, so it doesn't work with Unicode characters like ✔.
>
> Also had to update gclient_test.py, where double-wrapping stdout with Annotated caused made output not work and include_zero=True working caused other unintended side-effects.
>
> Example error from "fetch chromium":
> Traceback (most recent call last):
> File "C:\Google\depot_tools\gclient_scm.py", line 1045, in _Clone
> self._Run(clone_cmd, options, cwd=self._root_dir, retry=True,
> File "C:\Google\depot_tools\gclient_scm.py", line 1370, in _Run
> gclient_utils.CheckCallAndFilter(cmd, env=env, **kwargs)
> File "C:\Google\depot_tools\gclient_utils.py", line 583, in CheckCallAndFilter
> show_header_if_necessary(needs_header, attempt)
> File "C:\Google\depot_tools\gclient_utils.py", line 533, in show_header_if_necessary
> stdout_write(header.encode())
> File "C:\Google\depot_tools\gclient_utils.py", line 391, in write
> obj[0] += out
> TypeError: can only concatenate str (not "bytes") to str
>
> Bug: 984182
> Change-Id: If7037d30e9faf524f2405258281f6e6cd0bcdcae
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1778745
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Auto-Submit: Raul Tambre <raul@tambre.ee>
Bug: 984182
Change-Id: Ifafb5e16a517c4c38dd4c9d5c6d4c3f994838bc9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1845504
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This reverts commit 5d284fdf48.
Reason for revert: breaks buildbots, crbug.com/1011982
Original change's description:
> gclient_utils: buffer output as bytestrings in Annotated
>
> In Python 3 byestrings and normal strings can't be concatenated.
> To fix this we buffer as bytestrings in the Annotated wrapper.
> We can't decode to a string because the output might come byte-by-byte, so it doesn't work with Unicode characters like ✔.
>
> Also had to update gclient_test.py, where double-wrapping stdout with Annotated caused made output not work and include_zero=True working caused other unintended side-effects.
>
> Example error from "fetch chromium":
> Traceback (most recent call last):
> File "C:\Google\depot_tools\gclient_scm.py", line 1045, in _Clone
> self._Run(clone_cmd, options, cwd=self._root_dir, retry=True,
> File "C:\Google\depot_tools\gclient_scm.py", line 1370, in _Run
> gclient_utils.CheckCallAndFilter(cmd, env=env, **kwargs)
> File "C:\Google\depot_tools\gclient_utils.py", line 583, in CheckCallAndFilter
> show_header_if_necessary(needs_header, attempt)
> File "C:\Google\depot_tools\gclient_utils.py", line 533, in show_header_if_necessary
> stdout_write(header.encode())
> File "C:\Google\depot_tools\gclient_utils.py", line 391, in write
> obj[0] += out
> TypeError: can only concatenate str (not "bytes") to str
>
> Bug: 984182
> Change-Id: If7037d30e9faf524f2405258281f6e6cd0bcdcae
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1778745
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Auto-Submit: Raul Tambre <raul@tambre.ee>
TBR=dpranke@chromium.org,ehmaldonado@chromium.org,raul@tambre.ee
Change-Id: I5ea8d3249c58a3487996649a264bb5be059fe884
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 984182
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1845500
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
In Python 3 byestrings and normal strings can't be concatenated.
To fix this we buffer as bytestrings in the Annotated wrapper.
We can't decode to a string because the output might come byte-by-byte, so it doesn't work with Unicode characters like ✔.
Also had to update gclient_test.py, where double-wrapping stdout with Annotated caused made output not work and include_zero=True working caused other unintended side-effects.
Example error from "fetch chromium":
Traceback (most recent call last):
File "C:\Google\depot_tools\gclient_scm.py", line 1045, in _Clone
self._Run(clone_cmd, options, cwd=self._root_dir, retry=True,
File "C:\Google\depot_tools\gclient_scm.py", line 1370, in _Run
gclient_utils.CheckCallAndFilter(cmd, env=env, **kwargs)
File "C:\Google\depot_tools\gclient_utils.py", line 583, in CheckCallAndFilter
show_header_if_necessary(needs_header, attempt)
File "C:\Google\depot_tools\gclient_utils.py", line 533, in show_header_if_necessary
stdout_write(header.encode())
File "C:\Google\depot_tools\gclient_utils.py", line 391, in write
obj[0] += out
TypeError: can only concatenate str (not "bytes") to str
Bug: 984182
Change-Id: If7037d30e9faf524f2405258281f6e6cd0bcdcae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1778745
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Raul Tambre <raul@tambre.ee>
execfile() does not exist in Python 3; replace it with open(), read()
and exec().
Bug: 984182
Change-Id: I2266ff2bd63bffc458bd04500c8c0d6c360f1b4a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1844831
Auto-Submit: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Header was printed using sys.stdout, but the rest of the output was
printed using sys.stdout.buffer, causing the output to be written
before the header.
Bug: 984182
Change-Id: If6aaa05a579c3b0c73ce8ffe48f3a4e777fa16fa
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1790254
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@google.com>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
It seems like line is always a string, not bytes; this change
seems to make it work with both Python 2 and 3 (at least,
"gclient sync" looks correct and doesn't throw exceptions)
Bug: 984182
Change-Id: Iaa09e08d4eda0ccb54afa5598bac6231319d6c3d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1744766
Auto-Submit: Christian Biesinger <cbiesinger@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
- Merge CheckCallAndFilter[AndHeader]
- print_stdout will cause command output to be redirected to sys.stdout.
- Made compatible with Python 3.
Bug: 984182
Change-Id: Ida30e295b872c8c1a1474a376a90aecea924f307
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1727404
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
This reverts commit 602076dcf2.
Reason for revert:
It fails on input 0xe2 0x9c 0x94 which is a utf-8 heavy check mark. Problem is that CheckCallAndFilter is processing output byte by byte, so trying to decode only one of utf-8 bytes in sequence will always fail.
Original change's description:
> depot_tools: Make gclient_utils Python 3 compatible.
>
> Bug: 984182
> Change-Id: Ifb93466dcb7541f46c31c9b298049c425dd150d6
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1715472
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
TBR=tandrii@chromium.org,ehmaldonado@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: 984182
Change-Id: Ieae421acc430a031621db3329223c1da85506e55
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1724730
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>