Commit Graph

248 Commits (main)

Author SHA1 Message Date
Dirk Pranke bf76f3d3ed Explicitly propagate terminal size to gclient hooks.
gclient normally runs hooks in a pseudo terminal which has no
sense of the terminal size. If we want hooks to be able to change
what they output based on the terminal size, we have to explicitly
propagate the terminal size info down through the subprocess
environment.

Change-Id: I08f7c48ef78ea4eb9f5b791abb2a7e5ef8870050
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6161596
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
2 months ago
L. David Baron 22ee199f19 Flush stdout more often so password prompts come after explanations.
"gclient sync" on Mac occasionally requires a password for "sudo":
https://source.chromium.org/chromium/chromium/src/+/main:build/mac_toolchain.py;l=172-174;drc=a5fb630f300a1a3fdda9bb22a7ca79dd5296def2
in order to update XCode.  It attempts to print the explanation for why
the password is needed prior to the password prompt showing up.

This change flushes stdout more frequently so that the explanation will
*actually* show up before the password prompt.

Bug: 40827853
Change-Id: I840e4eb3c9a46ddfb4dbfef1faff043e25d8e694
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5903937
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: David Baron <dbaron@chromium.org>
5 months ago
Josip Sokcevic 525b18d46b [gclient] Use lowercase and when merging conditions
python doesn't understand AND.

R=jojwang@google.com

Change-Id: Icd0a8a4440bea2e272b2e9760bcd88a65d559848
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5555884
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
9 months ago
Josip Sokcevic 96dfd2e757 [gclient] Support per-object GCS condition
llvm-build GCS dependency needs to place one GCS object based on
platform. We can't have multiple top-level GCS llvm-build dependencies
as the pathname is used as a key.

This extends object property of GCS dependency to have `condition`.
The `condition` field will be AND with the top level condition.

R=jojwang@google.com, kimstephanie@google.com

Change-Id: I06a6281140a8057d598a6eac2b07c7577f0ea86c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5527474
Reviewed-by: Stephanie Kim <kimstephanie@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
10 months ago
Yiwei Zhang 314e46cf2c move is_env_cog to a dedicated helper function in gclient_util module
So that it can be shared with code other than scm.

Bug: 339231299
Change-Id: Iba68cc7ebba78c8933598d073861290bb0157c40
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5529883
Auto-Submit: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
10 months ago
Gaby Baghdadi 9d64acedea enable fetch and gclient to run on z/OS part 1
See https://issues.chromium.org/issues/323790693#comment12 for the manual steps still required.

R=gavinmak@google.com

Bug: 323790693
Change-Id: Id0ac2d8a6027cbb5e0554574471c160d5fae807c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5277474
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Gavin Mak b808b1bcdd Use shlex instead of pipes
Importing pipes gets a DeprecationWarning. shlex.quote is equivalent to
pipes.quote. Use that instead.

Bug: 1522874
Change-Id: I18b8598f2e961d58ba71d886921bda9d9ad956ca
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5243707
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Auto-Submit: Gavin Mak <gavinmak@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
1 year ago
Gavin Mak edba22d4eb Fix multiline comment formatting
Many incorrectly formatted comments exist from the switch to
4 space indent: https://crrev.com/c/4836379

Bug: 1514505
Change-Id: I6366f9da812919bd35b999f18fa8a49b7a66c09b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5153633
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Mike Frysinger 124bb8e53c switch to 4 space indent
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.

Reformatted using:
files=( $(
	git ls-tree -r --name-only HEAD | \
		grep -Ev -e '^(third_party|recipes)/' | \
		grep '\.py$';
	git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"

The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py

These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py

Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Gavin Mak 65c49b18b9 Drop py2 support in gclient and related files
python3 is the only supported version of python in depot_tools.

Bug: 1475402
Change-Id: I17174d7252b5cbf698700333d3cd561c6591d0a1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4809190
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Robert Iannucci 15d9af9d86 [gclient_utils] Make git tools set `gitcommit` modeline in vim.
This will improve the ergonomics of `git cl upload`, `git cl
description`, etc. for `vim` users.

I'm sure there are analogues for emacs, vscode, etc, but I'll leave
that as an exercise for a user of those editors :)

R=aravindvasudev@google.com

Change-Id: I194718326bc34092ef8c669a97eae9598592137d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4676559
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
2 years ago
Josip Sokcevic 7958e30579 Revert "Move scm.py to lib/"
This reverts commit 688adfe276.

Reason for revert: breaks try jobs

Original change's description:
> Move scm.py to lib/
>
> It belongs there.
>
> Change-Id: I286b50c43c587c39557fc508175efbd4b0f51d51
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4289680
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>

Change-Id: I662907b8bb4b2e7db82409bd34c3905db7961a12
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4300261
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
2 years ago
Josip Sokcevic 688adfe276 Move scm.py to lib/
It belongs there.

Change-Id: I286b50c43c587c39557fc508175efbd4b0f51d51
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4289680
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Jonas Termansen bf7eb5292c [portability] Support unknown operating systems in gclient.
Detect the name of unknown operating systems using uname, if available,
since it doesn't append the operating system version.

Change-Id: Idab7bd0db65a8d424ec2fd48f06247405b6649e4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4169240
Auto-Submit: Jonas Termansen <sortie@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Josip Sokcevic 38d669fe6f Reduce retries in gclient_utils
There are other retries (e.g. in git wrapper, gclient_scm) so we
shouldn't be retrying that much in gclient_utils.

In ideal case, we have a dedicated place where retries are determined,
and we make a retry decision based on stderr / exit code.

R=jojwang@google.com

Bug: 1359109
Change-Id: I97daa0d991a7294635e54b7a3d85a349c03c04c6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3871979
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
3 years ago
Joanna Wang 6628661eba [no-sync] Add a skip_sync_revisions and process it before running deps.
Bug: 1339472
Change-Id: I429489f7deea035c47e27e32ada858fb8a827643
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3735487
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
3 years ago
Joanna Wang 1a977bdc2d Update gclient cloning to work with cog.
Bug:1330629
Change-Id: I522255528e36d4806eada70c22afee1277a6778d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3668429
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
3 years ago
Aravind Vasudevan c5f0cbb865 Use pylint 2.7 for depot_tools
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.

Bug:1262286
Change-Id: I4b8f8fc521386419a3121bbb07edc8ac83170a94
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413679
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Josip Sokcevic 42c5bbbc96 Revert "Use pylint 2.7 for depot_tools"
This reverts commit 22bf605bb6.

Reason for revert: breaks gclient sync

Original change's description:
> Use pylint 2.7 for depot_tools
>
> This includes a few fixes for specific errors, and disables several new
> warnings introduced in this version, in order to allow for an incremental migration.
>
> Bug:1262286
> Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>

Bug: 1262286
Change-Id: Ieb946073c7886c7bf056ce843a5a48e82becf7a5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413672
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
3 years ago
Aravind Vasudevan 22bf605bb6 Use pylint 2.7 for depot_tools
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.

Bug:1262286
Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Christian Flach e685543855 Enable 'elaborate line editing' in `gclient_utils.AskForData`
The (raw_)input Python function used by `gclient_utils.AskForData`
supports additional 'elaborate line editing' [1] [2] features, such as
the ability to use backspace to fix a typo, if the readline module is
loaded. Make sure that readline is loaded by importing it locally in
`gclient_utils.AskForData`.
It does not appear to be possible to write a test that verifies this
behavior - assigning `io.StringIO` to `sys.stdin` appears to cause
(raw_)input to fall back to basic input processing.

[1] https://docs.python.org/2.7/library/functions.html#raw_input
[2] https://docs.python.org/3/library/functions.html#input

Change-Id: Ie1c25b3fa2f3a255b78426c20e47c968cd7198ab
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3306613
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Christian Flach <cmfcmf@chromium.org>
3 years ago
sokcevic 071528093f Support UTF-8 messages in git cl upload 🪟
Bug: 1240760
Change-Id: Ib64cd9c9b8c901105f6317bd74b889daa457e4de
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3101520
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
4 years ago
Josip Sokcevic 8b3bc259a0 Fix gclient_utils.FileRead() behavior for both 2 and 3.
We always want gclient_utils.FileRead() to return a text
string these days, but the existing code didn't work
right for Python2 if you called it explicitly with mode='r'.

R=ehmaldonado@chromium.org

Bug: 1210746
Change-Id: If540746f31308b671f9101334b5cd9848a7e9257
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2930626
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
4 years ago
Josip Sokcevic 740825ecf2 Detect uploadvalidator error message
If git push refs/for/ errors out, inspect stdout message to detect
uploadvalidator rejection. If present, present user a solution.

R=ajp@google.com

Change-Id: I70aee7615f3d905127a31bd5679968ade75c234a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2888475
Reviewed-by: Andy Perelson <ajp@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
4 years ago
Milad Fa d949c910de gclient: Don't try opening pseudo-terminal pipes on AIX
Change-Id: I57228d5b9d15253ae525cb2d138bfa9e73fea0e3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2416977
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
5 years ago
Milad Fa 52fdd1ffce Adding AIX
Change-Id: Ibf023a1f6b635872c849bda885b47ea1dbf7dda3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2410733
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
5 years ago
Ben Pastene ed15af5c4e gclient: Don't try opening pseudo-terminal pipes when running hooks.
As reported in crbug.com/1122671, windows bots can manage to trigger
the isatty conditional even when openpty() isn't provided.

So just force that conditional to false on windows.

Bug: 1122671
Change-Id: I2c85024da40feadd97cde7f718f4e5a4231fe9d6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2386321
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
5 years ago
Ben Pastene d410c664e6 gclient: Preserve ANSI color codes when calling hooks.
If a hook prints error/warning output that's color-coded, gclient
will cause the coloring to be disabled since those hooks aren't
called directly from a terminal.

By emulating a terminal when launching subprocs from gclient, we can
convince them to keep the color escape codes.

LED builds with both //third_party/depot_tools rolled to this CL, as
well as depot_tools in the recipe bundle rolled to this CL:
linux: https://ci.chromium.org/swarming/task/4e40237985888310
mac: https://ci.chromium.org/swarming/task/4e4023ea0c829710
win: https://ci.chromium.org/swarming/task/4e4024612e03dc10

Bug: 1034063
Change-Id: I4150f66ef215ece06f4c32482dcd4ded14eb1bfe
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2368435
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
5 years ago
Ayu Ishii 0985861970 CodeInclusion: Rename whitelist/blacklist -> allowlist/blocklist
Migrate to more inclusive terminology.

Bug: 1097674
Change-Id: I387b385ff36c7766682c06af34ed5fc6115119d1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2268403
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Ayu Ishii <ayui@chromium.org>
5 years ago
Aaron Gable 6e7ddb698b Remove agable from depot_tools OWNERS and TODOs
Change-Id: I0667af4562208a111ab3673f3fca6aab655f5f26
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2218962
Auto-Submit: Aaron Gable <agable@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Anthony Polito <apolito@google.com>
5 years ago
Quinten Yearsley 925cedba7b Run spellchecker on depot_tools
This was made by running `codespell` and `scspell`
and then checking the results.

Change-Id: I169fd5b40294f83015075b4a899fbca263821f25
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2144602
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Auto-Submit: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Edward Lesmes ae3586b538 depot_tools: Use gclient_utils.AskForData instead of raw_input.
gclient_utils.AskForData will use raw_input on Python 2 and input on Python 3.

Bug: 1063976
Change-Id: I93c059c4e4454d01787716b5a0a2641ad5253f53
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2116370
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Edward Lemur 1773f37de6 gclient_utils: Add temporary_file method.
Useful when passing information to a subcommand via a temporary file.

Bug: 1051631
Change-Id: I0b8deda921effd24a37109544e1e7ca22e00ba4e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2068942
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Raul Tambre 6693d0929f gclient_eval, gclient_utils: Fix Python 3.9 removal of collections members
Various collections module members were removed in Python 3.9 after being deprecated and moved to collections.abc in earlier Python 3 versions.
For accessing these members alias collections_abc as:
* collections.abc on Python 3
* collections on Python 2

Traceback (most recent call last):
  File "C:\Google\depot_tools\gclient.py", line 107, in <module>
    import gclient_eval
  File "C:\Google\depot_tools\gclient_eval.py", line 11, in <module>
    import gclient_utils
  File "C:\Google\depot_tools\gclient_utils.py", line 1201, in <module>
    class FrozenDict(collections.Mapping):
AttributeError: module 'collections' has no attribute 'Mapping'

Traceback (most recent call last):
  File "C:\Google\depot_tools\gclient.py", line 107, in <module>
    import gclient_eval
  File "C:\Google\depot_tools\gclient_eval.py", line 25, in <module>
    class _NodeDict(collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'

Bug: 984182
Change-Id: I7a4417978b93e29397e63764e4570a598c075bc0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2043879
Auto-Submit: Raul Tambre <raul@tambre.ee>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Edward Lemur a81450253f depot_tools: Use mock from vpython (or unittest.mock) instead of third_party/mock
Change-Id: I3a188b34ae5f62649108afe08fe0e389a408c2ac
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1947933
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
5 years ago
Edward Lemur 419c92f1bc gclient_utils: Make FileRead always return a Unicode string.
Previously, it returned bytes if the file did not contain valid Unicode data,
and a Unicode string otherwise.
It is confusing to reason about such a function, and no current caller needs
bytes data AFAICT, so make FileRead always return Unicode strings.

Bug: 1009814
Change-Id: I89dd1935e5d4fcaf9af71585b85bda6c47695950
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1880013
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Ben Pastene 1906f405ba gclient: When deleting untracked files, gracefully handle symlinks that point to dirs.
eg: https://logs.chromium.org/logs/chrome/buildbucket/cr-buildbucket.appspot.com/8898770399362199536/+/steps/bot_update/0/stdout

bot_update ends up clobbering the checkout in that build due to:
"Error: 161> Called rmtree(/b/s/w/ir/cache/builder/src/......) in non-directory"

That's because we're trying to rmtree a file. This will instead unlink
it.

Bug: 991276
Change-Id: Icffa18c27ddbaced20fb3410a91ca8fcc849d5e1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1877727
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
5 years ago
Marc-Antoine Ruel 8e57b4bc55 python3 improvements
Ran:
  vi $(git grep --name-only iteritems | grep -v third_party)
  vi $(git grep --name-only itervalues | grep -v third_party)
  vi $(git grep --name-only 'print ' | grep -v third_party)

and edited the files quickly with adhoc macros. Then ran in recipes/:
  ./recipes.py test train

There was only a small subset of files that had been updated to use
six.iteritems() and six.itervalues(). Since the dataset size that is
being used in gclient is small (pretty much always below 200 items),
it's better to just switch to .items() right away and take the temporary
performance hit, so that we don't need to come back to rewrite the code.

Recipe-Nontrivial-Roll: build
Bug: 984182
Change-Id: I5faf11486b66b0d73c9098ab0f2ce1b15a45c53e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1854900
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Marc-Antoine Ruel <maruel@chromium.org>
5 years ago
Edward Lemur cb1eb48966 Reland "gclient_utils: buffer output as bytestrings in Annotated"
This is a reland of 5d284fdf48

Always convert the input to bytes and write to sys.stdout (in Python2)
and sys.stdout.buffer (in Python 3).

Original change's description:
> gclient_utils: buffer output as bytestrings in Annotated
>
> In Python 3 byestrings and normal strings can't be concatenated.
> To fix this we buffer as bytestrings in the Annotated wrapper.
> We can't decode to a string because the output might come byte-by-byte, so it doesn't work with Unicode characters like ✔.
>
> Also had to update gclient_test.py, where double-wrapping stdout with Annotated caused made output not work and include_zero=True working caused other unintended side-effects.
>
> Example error from "fetch chromium":
> Traceback (most recent call last):
>   File "C:\Google\depot_tools\gclient_scm.py", line 1045, in _Clone
>     self._Run(clone_cmd, options, cwd=self._root_dir, retry=True,
>   File "C:\Google\depot_tools\gclient_scm.py", line 1370, in _Run
>     gclient_utils.CheckCallAndFilter(cmd, env=env, **kwargs)
>   File "C:\Google\depot_tools\gclient_utils.py", line 583, in CheckCallAndFilter
>     show_header_if_necessary(needs_header, attempt)
>   File "C:\Google\depot_tools\gclient_utils.py", line 533, in show_header_if_necessary
>     stdout_write(header.encode())
>   File "C:\Google\depot_tools\gclient_utils.py", line 391, in write
>     obj[0] += out
> TypeError: can only concatenate str (not "bytes") to str
>
> Bug: 984182
> Change-Id: If7037d30e9faf524f2405258281f6e6cd0bcdcae
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1778745
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Auto-Submit: Raul Tambre <raul@tambre.ee>

Bug: 984182
Change-Id: Ifafb5e16a517c4c38dd4c9d5c6d4c3f994838bc9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1845504
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Bob Haarman eeafa0eaec Revert "gclient_utils: buffer output as bytestrings in Annotated"
This reverts commit 5d284fdf48.

Reason for revert: breaks buildbots, crbug.com/1011982

Original change's description:
> gclient_utils: buffer output as bytestrings in Annotated
> 
> In Python 3 byestrings and normal strings can't be concatenated.
> To fix this we buffer as bytestrings in the Annotated wrapper.
> We can't decode to a string because the output might come byte-by-byte, so it doesn't work with Unicode characters like ✔.
> 
> Also had to update gclient_test.py, where double-wrapping stdout with Annotated caused made output not work and include_zero=True working caused other unintended side-effects.
> 
> Example error from "fetch chromium":
> Traceback (most recent call last):
>   File "C:\Google\depot_tools\gclient_scm.py", line 1045, in _Clone
>     self._Run(clone_cmd, options, cwd=self._root_dir, retry=True,
>   File "C:\Google\depot_tools\gclient_scm.py", line 1370, in _Run
>     gclient_utils.CheckCallAndFilter(cmd, env=env, **kwargs)
>   File "C:\Google\depot_tools\gclient_utils.py", line 583, in CheckCallAndFilter
>     show_header_if_necessary(needs_header, attempt)
>   File "C:\Google\depot_tools\gclient_utils.py", line 533, in show_header_if_necessary
>     stdout_write(header.encode())
>   File "C:\Google\depot_tools\gclient_utils.py", line 391, in write
>     obj[0] += out
> TypeError: can only concatenate str (not "bytes") to str
> 
> Bug: 984182
> Change-Id: If7037d30e9faf524f2405258281f6e6cd0bcdcae
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1778745
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Auto-Submit: Raul Tambre <raul@tambre.ee>

TBR=dpranke@chromium.org,ehmaldonado@chromium.org,raul@tambre.ee

Change-Id: I5ea8d3249c58a3487996649a264bb5be059fe884
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 984182
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1845500
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Raul Tambre 5d284fdf48 gclient_utils: buffer output as bytestrings in Annotated
In Python 3 byestrings and normal strings can't be concatenated.
To fix this we buffer as bytestrings in the Annotated wrapper.
We can't decode to a string because the output might come byte-by-byte, so it doesn't work with Unicode characters like ✔.

Also had to update gclient_test.py, where double-wrapping stdout with Annotated caused made output not work and include_zero=True working caused other unintended side-effects.

Example error from "fetch chromium":
Traceback (most recent call last):
  File "C:\Google\depot_tools\gclient_scm.py", line 1045, in _Clone
    self._Run(clone_cmd, options, cwd=self._root_dir, retry=True,
  File "C:\Google\depot_tools\gclient_scm.py", line 1370, in _Run
    gclient_utils.CheckCallAndFilter(cmd, env=env, **kwargs)
  File "C:\Google\depot_tools\gclient_utils.py", line 583, in CheckCallAndFilter
    show_header_if_necessary(needs_header, attempt)
  File "C:\Google\depot_tools\gclient_utils.py", line 533, in show_header_if_necessary
    stdout_write(header.encode())
  File "C:\Google\depot_tools\gclient_utils.py", line 391, in write
    obj[0] += out
TypeError: can only concatenate str (not "bytes") to str

Bug: 984182
Change-Id: If7037d30e9faf524f2405258281f6e6cd0bcdcae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1778745
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Raul Tambre <raul@tambre.ee>
5 years ago
Raphael Kubo da Costa 107c97c837 gclient_utils: Stop using execfile()
execfile() does not exist in Python 3; replace it with open(), read()
and exec().

Bug: 984182
Change-Id: I2266ff2bd63bffc458bd04500c8c0d6c360f1b4a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1844831
Auto-Submit: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Raul Tambre 383f6cf02d gclient_utils: Actually pass the include_zero parameter to Annotated constructor
Change-Id: I3d1e155ed2b6c4b4d6549bfb609358538dcdcd0a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1815868
Auto-Submit: Raul Tambre <raul@tambre.ee>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
6 years ago
Edward Lemur efce0d1b76 gclient_utils: Fix bug in CheckCallAndFilter in Python 3.
Header was printed using sys.stdout, but the rest of the output was
printed using sys.stdout.buffer, causing the output to be written
before the header.

Bug: 984182
Change-Id: If6aaa05a579c3b0c73ce8ffe48f3a4e777fa16fa
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1790254
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@google.com>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
6 years ago
Edward Lemur a877ee62b1 depot_tools: Run some tests on Python 3 on the bots.
Bug: 984182
Change-Id: I0b7f1e3e056548a27a416c9b4078e54c9e5d60d0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1769400
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@google.com>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
6 years ago
Edward Lemur 26a8b9f356 depot_tools: Make gclient Python3 compatible.
Bug: 984182
Change-Id: Idc38678acbfca9cc6b89d77db28fdc24784f2b7d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1753742
Commit-Queue: Edward Lemur <ehmaldonado@google.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
6 years ago
Christian Biesinger 1b4c7e9f38 Make gclient_utils work with Python 3
It seems like line is always a string, not bytes; this change
seems to make it work with both Python 2 and 3 (at least,
"gclient sync" looks correct and doesn't throw exceptions)

Bug: 984182
Change-Id: Iaa09e08d4eda0ccb54afa5598bac6231319d6c3d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1744766
Auto-Submit: Christian Biesinger <cbiesinger@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
6 years ago
Edward Lemur 24146be833 depot_tools: Simplify CheckCallAndFilter[AndHeader]
- Merge CheckCallAndFilter[AndHeader]
- print_stdout will cause command output to be redirected to sys.stdout.
- Made compatible with Python 3.

Bug: 984182
Change-Id: Ida30e295b872c8c1a1474a376a90aecea924f307
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1727404
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
6 years ago
Edward Lesmes c0758331ea Revert "depot_tools: Make gclient_utils Python 3 compatible."
This reverts commit 602076dcf2.

Reason for revert:

It fails on input 0xe2 0x9c 0x94 which is a utf-8 heavy check mark. Problem is that CheckCallAndFilter is processing output byte by byte, so trying to decode only one of utf-8 bytes in sequence will always fail.

Original change's description:
> depot_tools: Make gclient_utils Python 3 compatible.
> 
> Bug: 984182
> Change-Id: Ifb93466dcb7541f46c31c9b298049c425dd150d6
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1715472
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

TBR=tandrii@chromium.org,ehmaldonado@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: 984182
Change-Id: Ieae421acc430a031621db3329223c1da85506e55
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1724730
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
6 years ago
Edward Lemur df746d0e6f depot_tools: Make git_cache Python 3 compatible.
Bug: 984182
Change-Id: Ia413435af80d03dcfa01bb2ab7480cd6a37b4161
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1717496
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
6 years ago