The `-Y` flag was added to `df` only in the latest macOS version, so we
should guard its usage with a version check.
Change-Id: Ifafe47de5f9e86ce6a4605a1adb26994db5f33d9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5345461
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Philipp Wollermann <philwo@chromium.org>
Auto-Submit: Philipp Wollermann <philwo@chromium.org>
Use the log directory for the hash used to determime the socket used, so
developers get a distinct socket per build instead of per build
directory. Should allow concurrent builds against the same output
directory as much as ninja allows it.
Bug:b/321554715
Change-Id: I714dd2d32e2c651075055134e1568423a43eabdb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5290983
Auto-Submit: Michael Savigny <msavigny@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
Change is that racing directory is deleted atexit instead of being
cleaned up in a sweep at the start, whick wouldn't work anyway for the
actual use case the change was for (in-use racing directories would get
deleted, breaking concurrent builds).
Bug:b/324877315
Change-Id: I3baf25b07c13061e9b4c14a3a701ed9eceb9bbb4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5288415
Auto-Submit: Michael Savigny <msavigny@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
This reverts commit 566c3772a0.
Reason for revert: Seems to be causing flaky failures in win-trace angle try builder, see b/323830218 for details.
Original change's description:
> Changing location of temp racing directory.
>
> Put temporary racing artifacts into
> .reproxy_tmp/racing/<timestamp>-<hash> in prep for permitting
> simultaneous builds against the same output directory.
>
> Bug: b/321554715
> Change-Id: Iae9af9aed03197184e6d11da010d0bb10779ed23
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5273209
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Commit-Queue: Michael Savigny <msavigny@google.com>
> Reviewed-by: Junji Watanabe <jwata@google.com>
> Reviewed-by: Ben Segall <bentekkie@google.com>
Bug: b/321554715
Change-Id: Ib6fa285bb435d7c0d82960633364669e6575d73b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5286354
Commit-Queue: Michael Savigny <msavigny@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
Auto-Submit: Michael Savigny <msavigny@google.com>
Put temporary racing artifacts into
.reproxy_tmp/racing/<timestamp>-<hash> in prep for permitting
simultaneous builds against the same output directory.
Bug: b/321554715
Change-Id: Iae9af9aed03197184e6d11da010d0bb10779ed23
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5273209
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
Turns on deps cache, and increases it's size, for mac developers. In
some cases, it shows good improvements on chrome builds (cutting 1-4
minutes off a full chrome build, once populated).
Bug: b/322801289
Change-Id: I9cd2b35b5429228c07b0703d6c3344963e4fb9e5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5258563
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Reduces resources used for local compilation in racing for windows.
Makes machine running the build more responsive during the build.
Testing seems to show it doesn't impact overall build performance, and might even improve it slightly.
Bug: b/309819402
Change-Id: I6f4f41887b165c753508a57fcaa1a1a6931feee3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254559
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Auto-Submit: Michael Savigny <msavigny@google.com>
On my windows cloudtop, without deps cache a build of chrome took around 2130s, with it enabled (and populated) a build took around 1665s, which is about a 7-8 minute improvement.
Bug: b/320457907
Change-Id: I82232cad57152f1a10d1e262125290d54bf8ab33
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5249053
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Set cas_concurrency for mac builds to 100. It doesn't affect
performance on high-speed networks, but has shown an improvement on
networks prone to congestion.
Note that these adjustments have been checked out for builds on macs, and the cas_concurrency value being a default of 100 for mac doesn't mean it should be the default for other oses.
Bug: b/304616237
Change-Id: I9431fdb35737f9f2b236296625af2f2a31e52754
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5075857
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
The code rotating the old build logs expects to delete directories
(since it uses rmtree) but tries to delete all items returned by
os.listdir(...) which may includes symbolic links or regular files.
Change it to only consider the old build logs directories instead.
Based on crrev/5028323, with fix to unit test.
Fixed: b/310900283
Change-Id: I4c618e6618c0193331c063028ebf02d8c4e7baee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5038467
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Ramy Medhat <abdelaal@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
The catch clause was specific to one error type, but other errors may also happen such as socket.herror. In any case, the point is to fallback anyways.
Bug: b/310216577
Change-Id: I55561f8f40446f90bee27ea2aab1a4e60c03e7e6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5024019
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Anas Sulaiman <mrahs@google.com>
Reviewed-by: Michael Savigny <msavigny@google.com>
This is already covered in the PDD for googlers. We are already gathering hostnames in cloud monitoring mertrics
Bug: b/294945709
Change-Id: Idfb5b8d3b7cf64edd60092f6bd49a3f9f1950d15
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4848094
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Ben Segall <bentekkie@google.com>
This will help analyze startup metrics
Bug: b/294945709
Change-Id: Ia397dbebef7cc30b49c614dda51e56f482b6145b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4847325
Commit-Queue: Ben Segall <bentekkie@google.com>
Auto-Submit: Ben Segall <bentekkie@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.
Reformatted using:
files=( $(
git ls-tree -r --name-only HEAD | \
grep -Ev -e '^(third_party|recipes)/' | \
grep '\.py$';
git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"
The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py
These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py
Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
`Developer builds with reclient are not yet supported` is misleading.
This CL updates the message to help users to resolve the issues.
Bug: b/294933473
Change-Id: Ibce2854e25b8994af58011f4e7661f467fe910e6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4755000
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Junji Watanabe <jwata@google.com>
On NOP or incremental builds the reproxy startup/shutdown times can be a
significant portion of the total build time and it is important to
acknowledge these times.
Bug: b/294443556
Change-Id: Ic63331683632c8f447858b498f53d254b56ffe89
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4753472
Reviewed-by: Philipp Wollermann <philwo@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
local_resource_fraction is set to 0.2 as goma limited the max local
racing subprocesses to num_cpu/5
racing_bias is set to 0.95 as experimentally that ensures reasonable
behaviour for cache hits.
This cl does not enable racing by default. These values are only used if
RBE_exec_strategy=racing is set
Test: Tested builds on my mac and linux cloudtop and got mostly remote exec
and cache hits with some local executions
Bug: b/288564971
Change-Id: Idb71c03f4addd23d0253dc5bf3564058cd212284
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4713186
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
This is caused by the previous reproxy instance still having the log files open for writing. Bootstrap will kill that instance on startup of the next build but that only happens after we try and create the log dir
Bug: b/290230382
Change-Id: Idd88cfa9c571c0bca078f8bf0078fdb3941603ca
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4670328
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Auto-Submit: Ben Segall <bentekkie@google.com>
mdproxy/bin/prodcertstatus confuses the automatic auth logic in reproxy
This will be fixed in the next release but for now we remove mdproxy
from the path for autoninja builds
Test: Installed mdproxy locally and verified that gcert is not called
Change-Id: I0912a3c8b00a0cda41a85a16f51b8f54b20f1974
Bug: b/292523514
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4712465
Auto-Submit: Ben Segall <bentekkie@google.com>
Reviewed-by: Ramy Medhat <abdelaal@google.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
This reverts commit 3aa7489c3e.
Reason for revert: reproxy 0.110.0 includes a flag for fixing hardlink issue
Original change's description:
> Revert "reclient: Enable racing for chrome developers"
>
> This reverts commit 24ac2543c7.
>
> Reason for revert: hard links dont work on tmpfs b/288997481
>
> Original change's description:
> > reclient: Enable racing for chrome developers
> >
> > racing_bias=0.95 is a best guess based on benchmarks on my glinux and gwindows cloudtops, and my gmac laptop. I will monitor build performance data and adjust the bias based on that.
> >
> > Bug: b/288285261
> > Change-Id: Ie8652281f7bc815e64978bcd56a75e5ec28e2132
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4633180
> > Commit-Queue: Ben Segall <bentekkie@google.com>
> > Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
>
> Bug: b/288285261
> Change-Id: I7b262f63d4edd68447fcec0a781ef55923635d2c
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4651968
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Ben Segall <bentekkie@google.com>
Bug: b/288285261
Change-Id: I8fb65c32adc7b22a747c30f408f1f34b2bb5f8f6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4693625
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Ben Segall <bentekkie@google.com>
This reverts commit 24ac2543c7.
Reason for revert: hard links dont work on tmpfs b/288997481
Original change's description:
> reclient: Enable racing for chrome developers
>
> racing_bias=0.95 is a best guess based on benchmarks on my glinux and gwindows cloudtops, and my gmac laptop. I will monitor build performance data and adjust the bias based on that.
>
> Bug: b/288285261
> Change-Id: Ie8652281f7bc815e64978bcd56a75e5ec28e2132
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4633180
> Commit-Queue: Ben Segall <bentekkie@google.com>
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Bug: b/288285261
Change-Id: I7b262f63d4edd68447fcec0a781ef55923635d2c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4651968
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
racing_bias=0.95 is a best guess based on benchmarks on my glinux and gwindows cloudtops, and my gmac laptop. I will monitor build performance data and adjust the bias based on that.
Bug: b/288285261
Change-Id: Ie8652281f7bc815e64978bcd56a75e5ec28e2132
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4633180
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
This will allow us to segment siso and ninja metrics for analysis
Bug: b/281504726
Change-Id: If45b272004b236d634c717ea9e28e38e49c6231a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4568610
Commit-Queue: Ben Segall <bentekkie@google.com>
Auto-Submit: Ben Segall <bentekkie@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
This cl will be submitted after the cl to add the metadata to the
reproxy.cfg file has been submitted to chromium/src: crrev/c/4513215
Bug: b/281504726
Change-Id: Ifa6d5f56d4a85ccb9ec8e4f70207d8b6b9382e89
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4516023
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This speeds up small builds as we will still persist the credential
cache and it will speed up all builds as we will keep the deps cache
and compiler info cache.
Bug: b/278498980
Change-Id: I2fe26395a7493c461860b34ef3f109b2147abb5e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4454889
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
Developers will be able to use `autosiso` command to invoke siso builds.
This CL extracts reclient management logic from `ninja_reclient` to be reused in both Ninja and Siso builds.
Bug: b/278675516
Change-Id: I3e64a3188db184f4d3f851063a0feef7d3a73d6d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4445366
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Philipp Wollermann
Michael Savigny
Gavin Mak
Anas Sulaiman
Takuto Ikuta
Ben Segall
Mike Frysinger
Junji Watanabe
Bruce Dawson
Fumitoshi Ukai