Commit Graph

224 Commits (03918630fa596dd86f1a8f9f2e4c924b5bb33097)

Author SHA1 Message Date
Milad Fa d949c910de gclient: Don't try opening pseudo-terminal pipes on AIX
Change-Id: I57228d5b9d15253ae525cb2d138bfa9e73fea0e3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2416977
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
5 years ago
Milad Fa 52fdd1ffce Adding AIX
Change-Id: Ibf023a1f6b635872c849bda885b47ea1dbf7dda3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2410733
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
5 years ago
Ben Pastene ed15af5c4e gclient: Don't try opening pseudo-terminal pipes when running hooks.
As reported in crbug.com/1122671, windows bots can manage to trigger
the isatty conditional even when openpty() isn't provided.

So just force that conditional to false on windows.

Bug: 1122671
Change-Id: I2c85024da40feadd97cde7f718f4e5a4231fe9d6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2386321
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
5 years ago
Ben Pastene d410c664e6 gclient: Preserve ANSI color codes when calling hooks.
If a hook prints error/warning output that's color-coded, gclient
will cause the coloring to be disabled since those hooks aren't
called directly from a terminal.

By emulating a terminal when launching subprocs from gclient, we can
convince them to keep the color escape codes.

LED builds with both //third_party/depot_tools rolled to this CL, as
well as depot_tools in the recipe bundle rolled to this CL:
linux: https://ci.chromium.org/swarming/task/4e40237985888310
mac: https://ci.chromium.org/swarming/task/4e4023ea0c829710
win: https://ci.chromium.org/swarming/task/4e4024612e03dc10

Bug: 1034063
Change-Id: I4150f66ef215ece06f4c32482dcd4ded14eb1bfe
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2368435
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
5 years ago
Ayu Ishii 0985861970 CodeInclusion: Rename whitelist/blacklist -> allowlist/blocklist
Migrate to more inclusive terminology.

Bug: 1097674
Change-Id: I387b385ff36c7766682c06af34ed5fc6115119d1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2268403
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Ayu Ishii <ayui@chromium.org>
5 years ago
Aaron Gable 6e7ddb698b Remove agable from depot_tools OWNERS and TODOs
Change-Id: I0667af4562208a111ab3673f3fca6aab655f5f26
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2218962
Auto-Submit: Aaron Gable <agable@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Anthony Polito <apolito@google.com>
5 years ago
Quinten Yearsley 925cedba7b Run spellchecker on depot_tools
This was made by running `codespell` and `scspell`
and then checking the results.

Change-Id: I169fd5b40294f83015075b4a899fbca263821f25
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2144602
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Auto-Submit: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Edward Lesmes ae3586b538 depot_tools: Use gclient_utils.AskForData instead of raw_input.
gclient_utils.AskForData will use raw_input on Python 2 and input on Python 3.

Bug: 1063976
Change-Id: I93c059c4e4454d01787716b5a0a2641ad5253f53
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2116370
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Edward Lemur 1773f37de6 gclient_utils: Add temporary_file method.
Useful when passing information to a subcommand via a temporary file.

Bug: 1051631
Change-Id: I0b8deda921effd24a37109544e1e7ca22e00ba4e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2068942
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Raul Tambre 6693d0929f gclient_eval, gclient_utils: Fix Python 3.9 removal of collections members
Various collections module members were removed in Python 3.9 after being deprecated and moved to collections.abc in earlier Python 3 versions.
For accessing these members alias collections_abc as:
* collections.abc on Python 3
* collections on Python 2

Traceback (most recent call last):
  File "C:\Google\depot_tools\gclient.py", line 107, in <module>
    import gclient_eval
  File "C:\Google\depot_tools\gclient_eval.py", line 11, in <module>
    import gclient_utils
  File "C:\Google\depot_tools\gclient_utils.py", line 1201, in <module>
    class FrozenDict(collections.Mapping):
AttributeError: module 'collections' has no attribute 'Mapping'

Traceback (most recent call last):
  File "C:\Google\depot_tools\gclient.py", line 107, in <module>
    import gclient_eval
  File "C:\Google\depot_tools\gclient_eval.py", line 25, in <module>
    class _NodeDict(collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'

Bug: 984182
Change-Id: I7a4417978b93e29397e63764e4570a598c075bc0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2043879
Auto-Submit: Raul Tambre <raul@tambre.ee>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Edward Lemur a81450253f depot_tools: Use mock from vpython (or unittest.mock) instead of third_party/mock
Change-Id: I3a188b34ae5f62649108afe08fe0e389a408c2ac
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1947933
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
5 years ago
Edward Lemur 419c92f1bc gclient_utils: Make FileRead always return a Unicode string.
Previously, it returned bytes if the file did not contain valid Unicode data,
and a Unicode string otherwise.
It is confusing to reason about such a function, and no current caller needs
bytes data AFAICT, so make FileRead always return Unicode strings.

Bug: 1009814
Change-Id: I89dd1935e5d4fcaf9af71585b85bda6c47695950
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1880013
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Ben Pastene 1906f405ba gclient: When deleting untracked files, gracefully handle symlinks that point to dirs.
eg: https://logs.chromium.org/logs/chrome/buildbucket/cr-buildbucket.appspot.com/8898770399362199536/+/steps/bot_update/0/stdout

bot_update ends up clobbering the checkout in that build due to:
"Error: 161> Called rmtree(/b/s/w/ir/cache/builder/src/......) in non-directory"

That's because we're trying to rmtree a file. This will instead unlink
it.

Bug: 991276
Change-Id: Icffa18c27ddbaced20fb3410a91ca8fcc849d5e1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1877727
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
5 years ago
Marc-Antoine Ruel 8e57b4bc55 python3 improvements
Ran:
  vi $(git grep --name-only iteritems | grep -v third_party)
  vi $(git grep --name-only itervalues | grep -v third_party)
  vi $(git grep --name-only 'print ' | grep -v third_party)

and edited the files quickly with adhoc macros. Then ran in recipes/:
  ./recipes.py test train

There was only a small subset of files that had been updated to use
six.iteritems() and six.itervalues(). Since the dataset size that is
being used in gclient is small (pretty much always below 200 items),
it's better to just switch to .items() right away and take the temporary
performance hit, so that we don't need to come back to rewrite the code.

Recipe-Nontrivial-Roll: build
Bug: 984182
Change-Id: I5faf11486b66b0d73c9098ab0f2ce1b15a45c53e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1854900
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Marc-Antoine Ruel <maruel@chromium.org>
5 years ago
Edward Lemur cb1eb48966 Reland "gclient_utils: buffer output as bytestrings in Annotated"
This is a reland of 5d284fdf48

Always convert the input to bytes and write to sys.stdout (in Python2)
and sys.stdout.buffer (in Python 3).

Original change's description:
> gclient_utils: buffer output as bytestrings in Annotated
>
> In Python 3 byestrings and normal strings can't be concatenated.
> To fix this we buffer as bytestrings in the Annotated wrapper.
> We can't decode to a string because the output might come byte-by-byte, so it doesn't work with Unicode characters like ✔.
>
> Also had to update gclient_test.py, where double-wrapping stdout with Annotated caused made output not work and include_zero=True working caused other unintended side-effects.
>
> Example error from "fetch chromium":
> Traceback (most recent call last):
>   File "C:\Google\depot_tools\gclient_scm.py", line 1045, in _Clone
>     self._Run(clone_cmd, options, cwd=self._root_dir, retry=True,
>   File "C:\Google\depot_tools\gclient_scm.py", line 1370, in _Run
>     gclient_utils.CheckCallAndFilter(cmd, env=env, **kwargs)
>   File "C:\Google\depot_tools\gclient_utils.py", line 583, in CheckCallAndFilter
>     show_header_if_necessary(needs_header, attempt)
>   File "C:\Google\depot_tools\gclient_utils.py", line 533, in show_header_if_necessary
>     stdout_write(header.encode())
>   File "C:\Google\depot_tools\gclient_utils.py", line 391, in write
>     obj[0] += out
> TypeError: can only concatenate str (not "bytes") to str
>
> Bug: 984182
> Change-Id: If7037d30e9faf524f2405258281f6e6cd0bcdcae
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1778745
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Auto-Submit: Raul Tambre <raul@tambre.ee>

Bug: 984182
Change-Id: Ifafb5e16a517c4c38dd4c9d5c6d4c3f994838bc9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1845504
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Bob Haarman eeafa0eaec Revert "gclient_utils: buffer output as bytestrings in Annotated"
This reverts commit 5d284fdf48.

Reason for revert: breaks buildbots, crbug.com/1011982

Original change's description:
> gclient_utils: buffer output as bytestrings in Annotated
> 
> In Python 3 byestrings and normal strings can't be concatenated.
> To fix this we buffer as bytestrings in the Annotated wrapper.
> We can't decode to a string because the output might come byte-by-byte, so it doesn't work with Unicode characters like ✔.
> 
> Also had to update gclient_test.py, where double-wrapping stdout with Annotated caused made output not work and include_zero=True working caused other unintended side-effects.
> 
> Example error from "fetch chromium":
> Traceback (most recent call last):
>   File "C:\Google\depot_tools\gclient_scm.py", line 1045, in _Clone
>     self._Run(clone_cmd, options, cwd=self._root_dir, retry=True,
>   File "C:\Google\depot_tools\gclient_scm.py", line 1370, in _Run
>     gclient_utils.CheckCallAndFilter(cmd, env=env, **kwargs)
>   File "C:\Google\depot_tools\gclient_utils.py", line 583, in CheckCallAndFilter
>     show_header_if_necessary(needs_header, attempt)
>   File "C:\Google\depot_tools\gclient_utils.py", line 533, in show_header_if_necessary
>     stdout_write(header.encode())
>   File "C:\Google\depot_tools\gclient_utils.py", line 391, in write
>     obj[0] += out
> TypeError: can only concatenate str (not "bytes") to str
> 
> Bug: 984182
> Change-Id: If7037d30e9faf524f2405258281f6e6cd0bcdcae
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1778745
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Auto-Submit: Raul Tambre <raul@tambre.ee>

TBR=dpranke@chromium.org,ehmaldonado@chromium.org,raul@tambre.ee

Change-Id: I5ea8d3249c58a3487996649a264bb5be059fe884
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 984182
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1845500
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Raul Tambre 5d284fdf48 gclient_utils: buffer output as bytestrings in Annotated
In Python 3 byestrings and normal strings can't be concatenated.
To fix this we buffer as bytestrings in the Annotated wrapper.
We can't decode to a string because the output might come byte-by-byte, so it doesn't work with Unicode characters like ✔.

Also had to update gclient_test.py, where double-wrapping stdout with Annotated caused made output not work and include_zero=True working caused other unintended side-effects.

Example error from "fetch chromium":
Traceback (most recent call last):
  File "C:\Google\depot_tools\gclient_scm.py", line 1045, in _Clone
    self._Run(clone_cmd, options, cwd=self._root_dir, retry=True,
  File "C:\Google\depot_tools\gclient_scm.py", line 1370, in _Run
    gclient_utils.CheckCallAndFilter(cmd, env=env, **kwargs)
  File "C:\Google\depot_tools\gclient_utils.py", line 583, in CheckCallAndFilter
    show_header_if_necessary(needs_header, attempt)
  File "C:\Google\depot_tools\gclient_utils.py", line 533, in show_header_if_necessary
    stdout_write(header.encode())
  File "C:\Google\depot_tools\gclient_utils.py", line 391, in write
    obj[0] += out
TypeError: can only concatenate str (not "bytes") to str

Bug: 984182
Change-Id: If7037d30e9faf524f2405258281f6e6cd0bcdcae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1778745
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Raul Tambre <raul@tambre.ee>
5 years ago
Raphael Kubo da Costa 107c97c837 gclient_utils: Stop using execfile()
execfile() does not exist in Python 3; replace it with open(), read()
and exec().

Bug: 984182
Change-Id: I2266ff2bd63bffc458bd04500c8c0d6c360f1b4a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1844831
Auto-Submit: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Raul Tambre 383f6cf02d gclient_utils: Actually pass the include_zero parameter to Annotated constructor
Change-Id: I3d1e155ed2b6c4b4d6549bfb609358538dcdcd0a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1815868
Auto-Submit: Raul Tambre <raul@tambre.ee>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
6 years ago
Edward Lemur efce0d1b76 gclient_utils: Fix bug in CheckCallAndFilter in Python 3.
Header was printed using sys.stdout, but the rest of the output was
printed using sys.stdout.buffer, causing the output to be written
before the header.

Bug: 984182
Change-Id: If6aaa05a579c3b0c73ce8ffe48f3a4e777fa16fa
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1790254
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@google.com>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
6 years ago
Edward Lemur a877ee62b1 depot_tools: Run some tests on Python 3 on the bots.
Bug: 984182
Change-Id: I0b7f1e3e056548a27a416c9b4078e54c9e5d60d0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1769400
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@google.com>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
6 years ago
Edward Lemur 26a8b9f356 depot_tools: Make gclient Python3 compatible.
Bug: 984182
Change-Id: Idc38678acbfca9cc6b89d77db28fdc24784f2b7d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1753742
Commit-Queue: Edward Lemur <ehmaldonado@google.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
6 years ago
Christian Biesinger 1b4c7e9f38 Make gclient_utils work with Python 3
It seems like line is always a string, not bytes; this change
seems to make it work with both Python 2 and 3 (at least,
"gclient sync" looks correct and doesn't throw exceptions)

Bug: 984182
Change-Id: Iaa09e08d4eda0ccb54afa5598bac6231319d6c3d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1744766
Auto-Submit: Christian Biesinger <cbiesinger@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
6 years ago
Edward Lemur 24146be833 depot_tools: Simplify CheckCallAndFilter[AndHeader]
- Merge CheckCallAndFilter[AndHeader]
- print_stdout will cause command output to be redirected to sys.stdout.
- Made compatible with Python 3.

Bug: 984182
Change-Id: Ida30e295b872c8c1a1474a376a90aecea924f307
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1727404
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
6 years ago
Edward Lesmes c0758331ea Revert "depot_tools: Make gclient_utils Python 3 compatible."
This reverts commit 602076dcf2.

Reason for revert:

It fails on input 0xe2 0x9c 0x94 which is a utf-8 heavy check mark. Problem is that CheckCallAndFilter is processing output byte by byte, so trying to decode only one of utf-8 bytes in sequence will always fail.

Original change's description:
> depot_tools: Make gclient_utils Python 3 compatible.
> 
> Bug: 984182
> Change-Id: Ifb93466dcb7541f46c31c9b298049c425dd150d6
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1715472
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

TBR=tandrii@chromium.org,ehmaldonado@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: 984182
Change-Id: Ieae421acc430a031621db3329223c1da85506e55
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1724730
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
6 years ago
Edward Lemur df746d0e6f depot_tools: Make git_cache Python 3 compatible.
Bug: 984182
Change-Id: Ia413435af80d03dcfa01bb2ab7480cd6a37b4161
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1717496
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
6 years ago
Edward Lemur 602076dcf2 depot_tools: Make gclient_utils Python 3 compatible.
Bug: 984182
Change-Id: Ifb93466dcb7541f46c31c9b298049c425dd150d6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1715472
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
6 years ago
Raul Tambre 25eb8e4680 Update colorama to 0.4.1
Fixes colorama unintentionally overriding multiple sys.stdout wrappers of our own.
And provides a fix for a SyntaxWarning on Python 3.8:
C:\Google\depot_tools\third_party\colorama\ansitowin32.py:43: SyntaxWarning: invalid escape sequence \[
  ANSI_RE = re.compile('\033\[((?:\d|;)*)([a-zA-Z])')

Tests are updated to account for the annotation wrapper no longer being overriden.
A fix by agable for the annotation wrapper working with carriage returns is also included (https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1592612).

Bug: 958138, 958321
Change-Id: I2fe1def85c66cfe5229a1c25c2f677e498593eea
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1591513
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Commit-Queue: Raul Tambre <raul@tambre.ee>
Auto-Submit: Raul Tambre <raul@tambre.ee>
6 years ago
Raul Tambre 1fb046306b Fix UTF-8 output in gclient_utils.CheckCallAndFilter
Hooks for Electron output UTF-8 characters.
Example error for when "✔" is output:
  File "/home/markus/depot_tools/metrics.py", line 266, in print_notice_and_exit
    yield
  File "/home/markus/depot_tools/gclient.py", line 3112, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/home/markus/depot_tools/gclient.py", line 3098, in main
    return dispatcher.execute(OptionParser(), argv)
  File "/home/markus/depot_tools/subcommand.py", line 252, in execute
    return command(parser, args[1:])
  File "/home/markus/depot_tools/gclient.py", line 2677, in CMDsync
    ret = client.RunOnDeps('update', args)
  File "/home/markus/depot_tools/gclient.py", line 1746, in RunOnDeps
    self.RunHooksRecursively(self._options, pm)
  File "/home/markus/depot_tools/gclient.py", line 1052, in RunHooksRecursively
    hook.run()
  File "/home/markus/depot_tools/gclient.py", line 245, in run
    cmd, cwd=self.effective_cwd, always=self._verbose)
  File "/home/markus/depot_tools/gclient_utils.py", line 344, in CheckCallAndFilterAndHeader
    return CheckCallAndFilter(args, **kwargs)
  File "/home/markus/depot_tools/gclient_utils.py", line 576, in CheckCallAndFilter
    stdout.write(in_byte.decode())
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe2 in position 0: unexpected end of data

This issue was introduced in CL:1524583.

Bug: 942522
Change-Id: I3c4355b925b34398c800d142f942531a829e0297
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1541334
Auto-Submit: Raul Tambre <raul@tambre.ee>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Raul Tambre <raul@tambre.ee>
6 years ago
Raul Tambre b946b2389c Improve gclient Python 3 compatibility
This enables gclient sync and gclient runhooks to run, barring hook script failures.
git cl upload also now works.

The scripts still work with Python 2.
There are no intended behaviour changes.

Bug: 942522
Change-Id: I2ac587b5f803ba7f5bb5e412337ce049f4b1a741
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1524583
Commit-Queue: Raul Tambre <raul@tambre.ee>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
6 years ago
Raul Tambre c2f74c1698 Make fetch compatible with Python 3
The scripts still work with Python 2.
There are no intended behaviour changes.

Bug: 939847
Change-Id: Icada60c5b2cf351d62aead26b7364fcef2c2a3e5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1524486
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Raul Tambre <raul@tambre.ee>
6 years ago
Nico Weber 09e0b38f0f Move gclient path access to separate module.
gclient_utils.py is a kitchen sink and is for that reason expensive
to import. Move the comparatively cheap and simple path routines
to a new gclient_paths module and use that in gn.py, clang_format.py,
dart_format.py.

(To be able to move FindGclientRoot() to gclient_paths.py,
make it use io.open() instead of FileRead(). FileRead() tries
to paper over invalid utf-8, but that was added for presubmits,
not for .gclient files, so this is hopefully fine.)

Cuts gn.py overhead in half (on my Windows laptop from 0.6s to 0.25s,
still high; on my Mac laptop from 0.1s to 0.05s), and probably helps
the other two too.

Completely remove PathDifference() since it's unused.

Bug: 939959
Change-Id: I6a70f6e4c16062b622fb2df8778e8a598d4cc956
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1512058
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
6 years ago
Edward Lesmes 6f64a0522b Reland "gclient: Add commands to edit dependencies and variables in DEPS"
This is a reland of 7f4c905fc5

Original change's description:
> gclient: Add commands to edit dependencies and variables in DEPS
> 
> Adds 'gclient setvar' and 'gclient setdep' commands to edit variables
> and dependencies in a DEPS file.
> 
> Bug: 760633
> Change-Id: I6c0712cc079dbbbaee6541b7eda71f4b4813b77b
> Reviewed-on: https://chromium-review.googlesource.com/950405
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Aaron Gable <agable@chromium.org>

Bug: 760633
Change-Id: Ia46c74d02e5cc3b67517dfa248f597cb3d98ef3d
Reviewed-on: https://chromium-review.googlesource.com/969457
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
7 years ago
Edward Lemur 231f5eab45 Print git output when cloning a repo.
This will allow us to show the progress when cloning a new repo when running
commands like gclient sync.

R=agable@chromium.org

Bug: 722686
Change-Id: I268cba343ea4c3c024292c9341d5009aa112b184
Reviewed-on: https://chromium-review.googlesource.com/890524
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
7 years ago
Daniel Cheng a0c5f0850a Reland "gclient runhooks: now with less spammy progress notifications"
This is a reland of 5fd1defbcc
Original change's description:
> gclient runhooks: now with less spammy progress notifications
> 
> Bug: 772741
> Change-Id: I62ad64219ba27ee4ea26ceb603b1133dff4a2e04
> Reviewed-on: https://chromium-review.googlesource.com/727194
> Reviewed-by: Nico Weber <thakis@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Commit-Queue: Daniel Cheng <dcheng@chromium.org>

Bug: 772741
Change-Id: Idd6fcd46a8c97716d68ed0975940705a90540a4d
Reviewed-on: https://chromium-review.googlesource.com/728961
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Daniel Cheng <dcheng@chromium.org>
7 years ago
Daniel Cheng a21b5b314b Revert "gclient runhooks: now with less spammy progress notifications"
This reverts commit 5fd1defbcc.

Reason for revert: breaks v8 because hook names are optional

Original change's description:
> gclient runhooks: now with less spammy progress notifications
> 
> Bug: 772741
> Change-Id: I62ad64219ba27ee4ea26ceb603b1133dff4a2e04
> Reviewed-on: https://chromium-review.googlesource.com/727194
> Reviewed-by: Nico Weber <thakis@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Commit-Queue: Daniel Cheng <dcheng@chromium.org>

TBR=dcheng@chromium.org,thakis@chromium.org,dpranke@chromium.org

Change-Id: I7dee0ca188f5ca34232f16e6496249fc9f49fa96
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 772741
Reviewed-on: https://chromium-review.googlesource.com/728959
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Daniel Cheng <dcheng@chromium.org>
7 years ago
Daniel Cheng 5fd1defbcc gclient runhooks: now with less spammy progress notifications
Bug: 772741
Change-Id: I62ad64219ba27ee4ea26ceb603b1133dff4a2e04
Reviewed-on: https://chromium-review.googlesource.com/727194
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Daniel Cheng <dcheng@chromium.org>
7 years ago
Nico Weber 8db10a6fb1 gclient: Stop printing newline in front of "________ running hook"
The newline is probably here to protect against a hook that doesn't print
a trailing newline.  I've never seen a hook that did that, and if we found one,
we could make the logic look like "print a trailing newline if it's not there"
-- or just fix the hook.

The newline has been around since depot_tools was created
(https://codereview.chromium.org/92087, gclient.py), back then this was in
a general "run stuff" function, not in hooks-specific code.  Maybe it made
more sense back then.

This is part of a few changes to make `gclient runhooks` less noisy.

Bug: 772741
Change-Id: I285f76dc3f01c5acf5bbaa0be4db9f6edb9c0366
Reviewed-on: https://chromium-review.googlesource.com/706914
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
8 years ago
Paweł Hajdan, Jr 5ec7713bed gclient flatten: fixes for --pin-all-deps
Consistently pin all dependencies - including deps_os and ones
with shortened shas.

We add --process-all-deps switch so that users can easily
tell gclient to check out all affected dependencies locally.

Bug: 570091
Change-Id: If68db98000c569ae35dd7d0a4b695eb80a589213
Reviewed-on: https://chromium-review.googlesource.com/617224
Reviewed-by: Michael Moss <mmoss@chromium.org>
Commit-Queue: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
8 years ago
Bruce Dawson 1c5c118cfd Clean up a confusing gclient_utils.py message
While setting up a Chromium repo on a new machine I ended up with a
missing .gclient_entries file. This triggered a warning message but it
led me in the wrong direction. I think that the new message - saying
exactly what problem was detected - will be slightly more helpful.

Change-Id: I8c91861f843e3d30881b1b4933a02ad966c114ef
Reviewed-on: https://chromium-review.googlesource.com/533314
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
8 years ago
Paweł Hajdan, Jr 7e50261c2c gclient: freeze vars to prevent accidental modification
This will also be useful for other values (deps_os, hooks_os)
we may want to store and keep constant.

Freeze code copied from luci/recipes-py @ 944125e6d1e8c831d09517bde658a38d8f81db37

Bug: 570091
Change-Id: I3365cf2b267c478316870bbb3fd41e9955aa4ddf
Reviewed-on: https://chromium-review.googlesource.com/531166
Commit-Queue: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
8 years ago
Paweł Hajdan, Jr 7e9303ba19 gclient: remove support for From() (reland #1)
Original CL: https://chromium-review.googlesource.com/509693

This feature appears unused, and removing it will simplify the codebase.

Bug: 661382
Change-Id: I0d83b537b57d0b9ca65c7101d13e0ad6c1642a29
Reviewed-on: https://chromium-review.googlesource.com/512842
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
8 years ago
Andrii Shyshkalov b4f9d90473 Revert "gclient: remove support for From()"
This reverts commit bf72b593a2.

Reason for revert: maybe caused outage.

Original change's description:
> gclient: remove support for From()
> 
> This feature appears unused, and removing it will simplify the codebase.
> 
> Bug: 661382
> Change-Id: I545befb2c592eea53c54552018ce2d3dda7670f5
> Reviewed-on: https://chromium-review.googlesource.com/509693
> Commit-Queue: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> 

TBR=phajdan.jr@chromium.org,dpranke@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Bug: 661382

Change-Id: I4db9554a0a3a64a3a69908560b6da2a9963518f2
Reviewed-on: https://chromium-review.googlesource.com/512343
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
8 years ago
Paweł Hajdan, Jr bf72b593a2 gclient: remove support for From()
This feature appears unused, and removing it will simplify the codebase.

Bug: 661382
Change-Id: I545befb2c592eea53c54552018ce2d3dda7670f5
Reviewed-on: https://chromium-review.googlesource.com/509693
Commit-Queue: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
8 years ago
Andrii Shyshkalov 351c61da20 Fix git cl on windows for git-numberer repos.
Git cl decides if git-numberer is enabled on a repository by writing
Gerrit's project.config from refs/meta/config into a tempfile, which is
then queried using `git config -f tempfile --get ...`.  The file itself
is only flushed, but not closed after writing because Python's
tempfile.NamedTemporaryFile is deleted on closing. This worked fine on
Linix/Mac, but not on Windows, where `git config` apparently doesn't see
file or its contents.

This CL rewrites the above using yet another contexmanager temp
directory into which a file is written and closed before git config is
ran.

R=machenbach@chromium.org,grt@chromium.org
BUG=683202

Change-Id: I7974d66b1b2b0478ab4b6f7ac04e547a4981c46c
Reviewed-on: https://chromium-review.googlesource.com/430719
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
8 years ago
Quinten Yearsley b2cc4a94da depot_tools: Replace pylint error numbers with symbolic names.
This affects a bunch of files, but only changes comments,
and shouldn't make any difference to behavior.

The purpose is to slightly improve readability of pylint
disable comments.

Change-Id: Ic6cd0f8de792b31d91c6125f6da2616450b30f11
Reviewed-on: https://chromium-review.googlesource.com/420412
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
8 years ago
agable 41e3a6c553 Remove SVN support from gclient_utils and gclient_scm
This removes SVN support (most notably the SVNWrapper class, and the git-svn
logic in GitWrapper.GetUsableRev) from gclient_scm. It also removes some
references to SVN from comments in gclient_utils.

R=maruel@chromium.org
BUG=641588

Review-Url: https://chromiumcodereview.appspot.com/2393773003
8 years ago
tandrii 64103db728 Revert of gclient: kill git fetch operation that hangs. (patchset #12 id:220001 of https://codereview.chromium.org/2241843002/ )
Reason for revert:
Didn't help.

Original issue's description:
> gclient: kill git fetch operation that hangs.
>
> This provides env variable GCLIENT_KILL_GIT_FETCH_AFTER
> that kills git fetch if it produces no output for that
> many seconds.
>
> Note that this is not final patch, but an experiment.
> See http://crbug.com/635641#c24 for the deployment plan.
>
> BUG=635641
> R=hinoka@chromium.org
>
> Committed: f8757b7e02

TBR=hinoka@chromium.org,hinoka@google.com
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG=635641

Review-Url: https://codereview.chromium.org/2410853002
9 years ago
tandrii 30d956272d Revert of bot_update/gclient: kill git fetch after timeout regardless of output. (patchset #2 id:20001 of https://codereview.chromium.org/2293013002/ )
Reason for revert:
didn't work.

Original issue's description:
> bot_update/gclient: kill git fetch after timeout regardless of output.
>
> Also spits out whatever output was produced by git fetch for debugging.
>
> BUG=635641
> R=machenbach@chromium.org,hinoka@chromium.org
>
> Committed: db8b839320

TBR=hinoka@chromium.org,machenbach@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG=635641

Review-Url: https://codereview.chromium.org/2410053002
9 years ago
tandrii db8b839320 bot_update/gclient: kill git fetch after timeout regardless of output.
Also spits out whatever output was produced by git fetch for debugging.

BUG=635641
R=machenbach@chromium.org,hinoka@chromium.org

Review-Url: https://codereview.chromium.org/2293013002
9 years ago