It turns out python's built-in hash function uses a random seed
that changes per-process, so branch names based on that hash
aren't actually deterministic. This changes the hashing algorithm
to use a stable hash so we can get consistent results.
Bug: 389069356
Change-Id: I1d3241b922005a7bff4d8621dc79dc4551bf264e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6258544
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Devon Loehr <dloehr@google.com>
During testing, I discovered that the try/catch block surrounding
most of SplitCl is broken: subprocess errors seem to return bytes
more often than strings. When I fixed that, however, I found that
the errors were much more comprehensible if we simply re-raised
the initial error and got all the information it contained; this
both gives us a stack trace and makes it very obvious an error
occurred.
Specifically, we also had the problem that stderr was sometimes
empty even if the process failed (because it wrote to stdout instead),
and if we just wrote the output then it wasn't clear that an error
happened until you actually read the output, as opposed to the
exception printing where the stack trace makes it obvious.
Since re-raising without additional work is pointless, I just removed
the block entirely.
Bug: 389069356
Change-Id: I1d1bec765469d0b70b463f7214f47a25316a37c3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6258516
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Devon Loehr <dloehr@google.com>
This CL implements the proposal (from the design doc in the cited bug)
to allow `git cl split` to save the generated splitting to a
human-readable and editable text file. This allows users to save the
splitting in case something goes wrong and they need to retry, and also
lets them tweak a generated splitting if they get something mostly, but
not quite, usable.
It also allows users to edit the generated splitting interactively, like
they do for e.g. CL descriptions during `git cl upload`.
Bug: 389069356
Change-Id: I8ae21f2eb7c022ba181ae5af273ce09605b5acec
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6235727
Commit-Queue: Devon Loehr <dloehr@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This CL changes the way `git cl split` generates branch names during
upload. Specifically, it attempts to derive a common directory for all
the files in each CL, and includes that dirname as part of the branch
name, in addition to a hash of the files in the CL. This makes it a bit
easier to guess what's in a branch, given its name.
Since the hash is still included, branch names should remain unique.
Finding a common directory should be deterministic, so we can still rely
on branches having the same name if the script is re-run with the same
splitting.
Bug: 389069356
Change-Id: I70490258755f13962f3db5c835619caa24176af9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6243377
Commit-Queue: Devon Loehr <dloehr@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
`git cl split` operates by creating a new git branch for each CL
it uploads. It names this branch based on the directory of the files
in the CL. It uses this convention when resuming an interrupted upload:
it skips branches that already exist.
However, this approach is brittle for several reasons:
- The splitting is nondeterministic, so there's no guarantee that CLs
with the same directories will be generated when re-run. Even if they
are, there's no guarantee they'll have exactly the same files.
- For CLs with multiple directories, it uses an arbitrary one, so it
may not choose the same name a second time
- It bakes in an assumption that every CL is defined by the directories
it draws files from.
This CL alleviates this issues by instead naming branches using the hash
of their contained files. This ensures that, with extremely high
probability, branches will have the same name if and only if they have
the same (sorted) list of files. Thus resumed uploads can safely skip
existing branches.
Bug: 389069356
Change-Id: I19bf7cd6ff8b0127f5906384f7f43b3e40ac8fee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6236622
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Devon Loehr <dloehr@google.com>
This is a preparatory CL which refactors some internal logic to make
writing future CLs easier. In particular, it changes from treating a
CL as "a list of files with the same reviewer" to treating them as a
triple of (reviewers, files, directories). It does so using a dataclass
to make it easy to work with.
It also adds an EmitWarning convenience function, which will be more
heavily used in future CLs.
Bug: 389069356
Change-Id: I842d59ba69f5db5e3745fa39832794cdc1eb222e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6190430
Commit-Queue: Devon Loehr <dloehr@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
`git cl split` has the potential to do a lot of operations at once,
which is problematic if it didn't end up splitting things in a way
the user approves of. This changes it to print a summary of the
splitting, and asking the user to review it briefly before proceeding
with the upload.
Bug: 389069356, 40642562, 40269201
Change-Id: I90bead0147badbaa3afcce2264d805ae498f101c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6163905
Commit-Queue: Devon Loehr <dloehr@google.com>
Reviewed-by: Peter Kotwicz <pkotwicz@chromium.org>
Reviewed-by: Andy Perelson <ajp@google.com>
When adding tests, the LoadDescription function accidentally got called
with the wrong arguments in the main function. This fixes that; a future
CL (uploading shortly) adds more robust tests that would have caught this.
Bug: 389568463, 389069356
Change-Id: Icaf5e83cd8caa9f3975173f8c8ee7d92ef44ee56
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6170638
Commit-Queue: Andy Perelson <ajp@google.com>
Reviewed-by: Brian Ryner <bryner@google.com>
Reviewed-by: Andy Perelson <ajp@google.com>
Auto-Submit: Devon Loehr <dloehr@google.com>
I frequently find myself running `git cl split` just to see how the
files are going to be split up, without actually intending to submit the
resulting CLs immediately. Doing so requires creating a file, putting a
dummy description in it, and passing that to the command line, which is
unnecessary work. This CL simply allows dry runs to use a dummy
description if none is provided, to make checking the tool's output
easier.
Bug: None
Change-Id: I47e06c6e6da26701e07dcae81ab605edac2e2ca6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6163904
Reviewed-by: Andy Perelson <ajp@google.com>
Commit-Queue: Devon Loehr <dloehr@google.com>
Reviewed-by: Brian Ryner <bryner@google.com>
In source, line-crossing string fragments are joined with no spaces in
between. This CL trivially adds spaces s.t. the warning doesn't get
printed like:
```
won'tbreak anything...to cancelyour jobs
```
Change-Id: I17b4a6c0d22fa63ee4ea66d5b32f7fc3221bc2be
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5352388
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Auto-Submit: Kalvin Lee <kdlee@chromium.org>
This CL updates the warning printed when more than 10 CLs will be
created to be shown whether or not the CLs are being sent to the CQ
and provides guidance on adjusting the number of CLs using the max-depth
flag. This makes the tool more user friendly by preventing a new user
from unknowingly creating many CLs without realizing there is a
supported option to customize the depth.
Additionally, it sets the default for the cq-dry-run option to False
instead of None and logs the value of cq_dry_run when running a git cl
split dry run.
Change-Id: I6feba24d06dd5e88d9005345d5b0a1e15c53e197
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5234543
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Mike Dougherty <michaeldo@chromium.org>
All __future__ imports (unicode_literals, print_function) are already
mandatory in py3. Also remove an outdated py2 comment in
presubmit_canned_checks.py
Bug: 1475402
Change-Id: I27cf6a8268f6dd1081f22af782c4c29a975376ae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4867135
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.
Reformatted using:
files=( $(
git ls-tree -r --name-only HEAD | \
grep -Ev -e '^(third_party|recipes)/' | \
grep '\.py$';
git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"
The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py
These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py
Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This CL:
- Adds tests for split_cl.UploadCl()
- Splits out logic checking bug links so that it can be tested
Bug:None
Change-Id: I3c08b129e0cfda67a3d93a2e01acef86d33e92b8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4743773
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Peter Kotwicz <pkotwicz@chromium.org>
This CL stops "git cl split" from printing which directories could
not be uploaded if upload fails. This information is redundant with
previous lines printed by "git cl split"
Bug:None
Change-Id: I39bc47e3b29d4ce777a65b6bdac28b19b672339f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4743774
Commit-Queue: Peter Kotwicz <pkotwicz@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This CL prevents split_cl.py from uploading multiple CLs for the same
reviewer set.
BUG=1468350
Change-Id: I9c328589f7facfe10ee5066cc3d1cda007dd1d2a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4726781
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Peter Kotwicz <pkotwicz@chromium.org>
For some types of changes, git cl split generates too many small CLs.
--max-depth provides one way of generating larger CLs when the author
judges that the larger CLs do not adversely affect reviewability (e.g.
20x 1 line CLs packed into 1x 20 line CL is generally fine).
Fixed: 777781
Change-Id: I64426ff4723fbc412fbc47f3cc12767433aeb8ae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3933974
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Daniel Cheng <dcheng@chromium.org>
This change updates the output format of `git cl split` when
`description_file` has no footers.
Bug: 1215852
Change-Id: I69764885337ec31134f2b5e2d861930e0bc8cd2d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2936161
Reviewed-by: Jochen Eisinger <jochen@chromium.org>
Reviewed-by: François Doray <fdoray@chromium.org>
Commit-Queue: Fangzhen Song <songfangzhen@bytedance.com>
Without this change, git cl split will create a separate CL for every
file that has per-file owners rules. This is almost always the wrong
thing to do. With this change, it will at least aggregate all such files
to their parent directory. This may under-split, but is perhaps nearer
the mark.
Change-Id: Iea7997b36d053713fa00f1261ea032e09d9c2818
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2555729
Auto-Submit: Sigurður Ásgeirsson <siggi@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
gclient_utils.AskForData will use raw_input on Python 2 and input on Python 3.
Bug: 1063976
Change-Id: I93c059c4e4454d01787716b5a0a2641ad5253f53
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2116370
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Also refactor code to eliminate multiple calls to _create_description_from_log.
Change-Id: I113134fbd90f396bdb6d561ed0369ea5ee9c78ad
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2090448
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This reverts commit 684096347b.
Reason for revert:
Breaks git-cl split.
https://bugs.chromium.org/p/chromium/issues/detail?id=1054888
Original change's description:
> Improve git cl split
>
> This CL changes the behavior of `git cl split` to split the change
> by the size of the resulting CLs. For now, this is based on the number
> of bytes changed, and not by the number of changed lines. Depending
> on the shape of change, this may still produce more CLs than expected
> (and possibly more than before).
>
> A future change will switch the split to be based on the number
> of affected lines, and also introduce a mode to base the split
> on the number of affected files.
>
> Bug: 998922
> Change-Id: I49f868972a61b89b426ef9e2ceedc733eacb4350
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1778744
> Commit-Queue: Yannic Bonenberger <yannic.bonenberger@gmail.com>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
TBR=fdoray@chromium.org,dpranke@chromium.org,yannic.bonenberger@gmail.com,infra-scoped@luci-project-accounts.iam.gserviceaccount.com
Bug: 998922
Change-Id: I466e1245d5f1c934443d850c48c42d3e694e71c6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2080205
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Useful when passing information to a subcommand via a temporary file.
Bug: 1051631
Change-Id: I0b8deda921effd24a37109544e1e7ca22e00ba4e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2068942
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This CL changes the behavior of `git cl split` to split the change
by the size of the resulting CLs. For now, this is based on the number
of bytes changed, and not by the number of changed lines. Depending
on the shape of change, this may still produce more CLs than expected
(and possibly more than before).
A future change will switch the split to be based on the number
of affected lines, and also introduce a mode to base the split
on the number of affected files.
Bug: 998922
Change-Id: I49f868972a61b89b426ef9e2ceedc733eacb4350
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1778744
Commit-Queue: Yannic Bonenberger <yannic.bonenberger@gmail.com>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Ran:
vi $(git grep --name-only iteritems | grep -v third_party)
vi $(git grep --name-only itervalues | grep -v third_party)
vi $(git grep --name-only 'print ' | grep -v third_party)
and edited the files quickly with adhoc macros. Then ran in recipes/:
./recipes.py test train
There was only a small subset of files that had been updated to use
six.iteritems() and six.itervalues(). Since the dataset size that is
being used in gclient is small (pretty much always below 200 items),
it's better to just switch to .items() right away and take the temporary
performance hit, so that we don't need to come back to rewrite the code.
Recipe-Nontrivial-Roll: build
Bug: 984182
Change-Id: I5faf11486b66b0d73c9098ab0f2ce1b15a45c53e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1854900
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Marc-Antoine Ruel <maruel@chromium.org>
Ran "2to3 -w -n -f print ./" and manually added imports.
Ran "^\s*print " and "\s+print " to find batch/shell scripts, comments and the like with embedded code, and updated them manually.
Also manually added imports to files, which used print as a function, but were missing the import.
The scripts still work with Python 2.
There are no intended behaviour changes.
Bug: 942522
Change-Id: Id777e4d4df4adcdfdab1b18bde89f235ef491b9f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1595684
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Auto-Submit: Raul Tambre <raul@tambre.ee>
`git cl split` currently runs a cq dry run for every uploaded CL. This
has overloaded our infrastructure a few times in the past. This CL changes
the command to not dry run by default, and adds a flag to enable this. The
flag has a warning about doing this, and tells the user to email
infra-dev@chromium.org if they're going to be generating >~10 CLs.
Bug: 878117
Change-Id: Ic865c09b188b8d4f202785f4763f7b7b8910c9cf
Reviewed-on: https://chromium-review.googlesource.com/1191927
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Devon Loehr
Chris Fredrickson
Takuto Ikuta
Kalvin Lee
Mike Dougherty
Gavin Mak
Mike Frysinger
Aravind Vasudevan
Peter Kotwicz
Anne Redulla
Rachael Newitt
Josip Sokcevic
Thiago Perrotta
Daniel Cheng
Josip Sokcevic
Song Fangzhen
Olivier Li
Edward Lesmes
Sigurdur Asgeirsson
Anthony Polito
Edward Lesmes
Yannic Bonenberger
Marc-Antoine Ruel
Raul Tambre
Stephen Martinis