This allows us to configure the get_use_siso_default to give a notification suggesting people to switch.
Bug: 393209062
Change-Id: I1bced33272a6dd19d4e71f96f80ac6de61b48d9d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6217099
Commit-Queue: Matt Stark <msta@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Somewhere deep in the Google Cloud Go SDK is a call to user.Current() to
get the user's home directory, in order to find some configuration files
necessary for TLS Client Certificate authentication.
The user.Current() function in Go versions prior to the yet unreleased
Go 1.24 can be very slow on Windows, as it causes a network call to the
Active Directory server to be made if the machine is part of a Windows
domain.
As we don't use TLS Client Certificate at Google, we can safely disable
this code path by setting GOOGLE_API_USE_CLIENT_CERTIFICATE=false in the
environment prior to launching reproxy and other tools using the SDK.
As I'm not sure whether this would cause issues for people actually
using TLS Client Certificate auth with their REAPI backend, let's only
apply this workaround on Google corp machines. It's highly unlikely that
others are affected by this particular issue anyway, and when Go 1.24
is released in February, it will fix the underlying issue anyway.
Bug: 388195531
Change-Id: Ia9d4b74c251b303b66b19fb59957ff5a922d883b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6191410
Auto-Submit: Philipp Wollermann <philwo@google.com>
Commit-Queue: Philipp Wollermann <philwo@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
- Starts the build server when the build starts.
- Writes tty filename to env variable.
- Tells the build server about the current build so it does not exit
until autoninja does, even if idle.
- Cancels pending tasks on Ctrl+c.
Change-Id: I86bb9852bd0975f381b049b9ff21c38eef7cef9d
Bug: 370589852
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5917985
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Mohamed Heikal <mheikal@chromium.org>
Commit-Queue: Mohamed Heikal <mheikal@chromium.org>
These were introduced in crrev.com/c/6024739 and currently break
autoninja for non-Googlers who manually enable Siso by setting the
use_siso=true GN arg.
$ gn gen --args='use_siso=true' out/Debug
$ autoninja -C out/Debug base
Traceback (most recent call last):
File "/home/philwo/depot_tools/autoninja.py", line 504, in <module>
sys.exit(main(sys.argv))
^^^^^^^^^^^^^^
File "/home/philwo/depot_tools/autoninja.py", line 493, in main
exit_code = _main_inner(input_args, build_id, should_collect_logs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/philwo/depot_tools/autoninja.py", line 249, in _main_inner
assert use_siso != False
^^^^^^^^^^^^^^^^^
AssertionError
R=ukai@google.com
Change-Id: I6536576d39b4ef094daa4e7ec80f3d743ecefd85
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6058992
Auto-Submit: Philipp Wollermann <philwo@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
If use_reclient_value.py is not found and args.gn only has
use_remotexec=true, the autoninja fails to start reproxy.
Change-Id: I6608ea48b66864bf2ab89176b6b01a79e07b7564
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6059221
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Aleksei Khoroshilov <akhoroshilov@brave.com>
https://crrev.com/c/6048393 provides build/toolchain/use_reclient_value.py
to get value for use_reclient.
Use it in autoninja if it exists.
Bug: 341167943
Change-Id: Ia6207ea044cc109fa156deff3c3357475368576f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6024739
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
ninja tools don't need an RBE environment active, so avoid an
unnecessary delay and noisy output when something like autoninja -t
commands is invoked.
Change-Id: Ieb58dee56b15968dcac9483ff41be25544f02470
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5991650
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
ninja tools don't need an RBE environment active, so avoid an
unnecessary delay and noisy output when something like autoninja -t
commands is invoked.
Change-Id: I28fe5275a4a130251d5c0feac5841ba2c410527d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5987270
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
External developers may use RBE without RBE_instance set.
Autoninja should not fail in this case.
Change-Id: I2dbafded690246132bb0ff1cd080caa0b758b4f6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5851434
Auto-Submit: Aleksei Khoroshilov <akhoroshilov@brave.com>
Commit-Queue: Aleksei Khoroshilov <akhoroshilov@brave.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Account check would become too slow.
We'll check RBE project to use instead.
On corp machine, our policy to use @google.com account
and rbe-chrome-untrusted to build chromium/chrome.
We don't allow rbe-chromium-untrusted with @chromium.org
on corp machine.
On non-corp machine, user couldn't use rbe-chrome-untrusted
because it's @google.com only, and corp security policy
doesn't allow @google.com account on non-corp machine.
Bug: b/364318216
Change-Id: I0f3a19e105b050aef6a62e1b25b45b1722382a34
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5848450
Reviewed-by: Scott Lee <ddoman@chromium.org>
Reviewed-by: Michael Savigny <msavigny@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
If user forgets to set `use_reclient=false`,
```
ukai@ukai /google/cog/cloud/ukai/siso-test/chromium
% ~/depot_tools/autoninja -C out/Default base
WARNING: You're not using Siso's built-in remote execution. The build will be slow.
You should set the following in args.gn to get better performance:
use_remoteexec=true
use_reclient=false
use_siso=true
Proxy started successfully.
```
Bug: b/362301140
Change-Id: I40972b96fd2576c32232f88e6248a1000708c3e2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5816108
Reviewed-by: Junji Watanabe <jwata@google.com>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Gary Tong <gatong@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@google.com>
The main process exits right after ninja_uploader.py subprocess starts,
which may print ResourceWarning from subprocess.
e319f774f9/Lib/subprocess.py (L946-L947)
This CL ignores the warning message.
Change-Id: I800387194763e60a16437b3bd41fa282b605dda7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5782795
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
I ran into this from having use_siso=true in an include(), and then
trying to turn it off again via an assignment
Change-Id: Iba70ade68c7a3699755d6a0070816283e0538c6f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5769918
Commit-Queue: Junji Watanabe <jwata@google.com>
Auto-Submit: Andrew Grieve <agrieve@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Currently, {autoninja, autoninja.bat} scripts generate AUTONINJA_BUILD_ID by calling `python3 -c "import uuid; print(uuid.uuid4())"`.
This CL moves the logic at the beginning `autoninja.py`.
This also reduces the overhead of starting Python interpreter, which
takes around 60ms on my Windows workstaion.
```
> hyperfine python3
Benchmark 1: python3
Time (mean ± σ): 63.4 ms ± 7.8 ms [User: 25.7 ms, System: 33.7 ms]
Range (min … max): 51.0 ms … 80.2 ms 41 runs
```
Change-Id: I93d5b65f3c5542c3a93a3de1f27a5aa5d06c09a6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5737673
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Junji Watanabe <jwata@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
This CL unifies the opt-in/opt-out handling for build telemetry collections about Reclient and Ninjalog.
The user consent message will be displayed only once at the beginning of a build.
```
❯ autoninja -C out/deterministic-andorid-dbg base
*** NOTICE ***
Google-internal telemetry (including build logs, username, and hostname) is collected on corp machines to diagnose performance and fix build issues. This reminder will be shown 9 more times. See http://go/chrome-build-telemetry for details. Hide this notice or opt out by running: build_telemetry [opt-in] [opt-out]
*** END NOTICE ***
Proxy started successfully.
...
```
Bug: 345113094
Change-Id: Ie5886287c4bd20262be0ff247508ac3869441eb1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5669094
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Michael Savigny <msavigny@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
- Remove reclient_helper.py to avoid calling it directly.
- Cleans up autoninja.py
I just want to do this cleanup before merging the user notice messages
for logs/metrics collection.
Bug: 345113094
Change-Id: I0c76426b3cb387eae6dede031727c107e57d5d1a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5668282
Reviewed-by: Philipp Wollermann <philwo@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Ben Segall <bentekkie@google.com>
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
during build, lots of python scripts run and generates
lots of pycache *pyc in workspace.
To keep workspace clean, set PYTHONDONTWRITEBYTECODE=1
in autoninja.py
Change-Id: I5ce4c904aa11dd8172672ff9a8e9a06ebf69c9fe
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5599922
Reviewed-by: Erik Staab <estaab@chromium.org>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gary Tong <gatong@chromium.org>
Reviewed-by: Gregory Guterman <guterman@google.com>
need to split input_args[1] even if it doesn't contains " ",
e.g. when input_args[1] == ""
Bug: b/343172216
Change-Id: I32d9346063534ea2b4f891fd596c7ec02ab10dbf
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5583415
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Richard Wang <richardwa@google.com>
Reviewed-by: Richard Wang <richardwa@google.com>
This CL allows autoninja to use Siso's builtin RBE client instead of using Reclient.
Developers have to specify `use_remoteexec=true use_reclient=false` GN arg to use this mode.
This should be no-op for all developers unless `use_reclient=false` is set.
Bug: 342038479
Change-Id: I8a67f114c056be6ddb0cd26cccf0157e0b184c5a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5567811
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Include a luci-auth check of the account logged in.
This is to check that a user isn't using a non-google account on a corp machine. Also this check is only performed every 12 hours, so it doesn't impact every build. This check was already performed with other auth mechanisms, this is just making sure it is being done for luci-auth as well.
Bug: b/330339907
Change-Id: I8ea97c8de0f2d74c2d735fa959d8227cec35df46
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5379037
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
This doesn't catch all the variants of --help that getopt allows but it
does catch --help, which is probably the most common.
Bug: 329317224
Change-Id: I2dde63df84bd2c2ade5f9d2c1ab10c3dc13fba70
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5370829
Auto-Submit: David Baron <dbaron@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Notes that --offline is for disabling remote execution in general not just for goma.
Bug: b/329317224
Change-Id: I5538c9fac6d5161e98e1d156e61c512c6a1e8496
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5370094
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
This reverts commit 6b84fbfb20.
Reason for revert: Jumped the gun before final resolution for b/316921158
Original change's description:
> Remove ability to use_goma in autoninja (chromeos excepted)
>
> Won't run a build if use_goma=true and target_os!="chromeos".
>
> Bug: b/277197166
> Test: Manual tests and updates to unit tests.
> Change-Id: I753618e6ad11cb623d9926a4af00a07339c02c43
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5329167
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Commit-Queue: Michael Savigny <msavigny@google.com>
Bug: b/277197166
Change-Id: Iaf506a10dfdc1c1672b2927636adf5773edc67c0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5333176
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Auto-Submit: Michael Savigny <msavigny@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Won't run a build if use_goma=true and target_os!="chromeos".
Bug: b/277197166
Test: Manual tests and updates to unit tests.
Change-Id: I753618e6ad11cb623d9926a4af00a07339c02c43
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5329167
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
This reverts commit 1592a89c9f.
Reason for revert:
b/325856422
Original change's description:
> Remove ability to use_goma via autoninja.
>
> Doesn't run a build if use_goma is true, point to reclient docs. Turns
> off use_goma for linux and mac builds. Windows will follow in 2-3 weeks.
>
> Bug: b/277197166
> Change-Id: Icf000ee5e4bdfeba82cadbbf2e02e7cd728e89b0
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5272474
> Reviewed-by: Ben Segall <bentekkie@google.com>
> Reviewed-by: Ramy Medhat <abdelaal@google.com>
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Commit-Queue: Michael Savigny <msavigny@google.com>
Bug: b/277197166 b/325856422
Change-Id: Iaec83b254af002f420bfd2df1154ce94e18fac15
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5308674
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Owners-Override: David Munro <davidmunro@google.com>
Reviewed-by: David Munro <davidmunro@google.com>
Doesn't run a build if use_goma is true, point to reclient docs. Turns
off use_goma for linux and mac builds. Windows will follow in 2-3 weeks.
Bug: b/277197166
Change-Id: Icf000ee5e4bdfeba82cadbbf2e02e7cd728e89b0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5272474
Reviewed-by: Ben Segall <bentekkie@google.com>
Reviewed-by: Ramy Medhat <abdelaal@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
Update the warning message for linux and mac goma builds, and add a 5
second delay.
Do not submit until Jan 22, 2024
Bug: b/277197166
Change-Id: Id93822b1aa32b05746b8fad4b9dc332cf61d291c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5215930
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
This is for some internal infrastructures which want to use service
accounts from corp machines.
Bug: b/320639792
Change-Id: Ie90241fe89e0bb0fb5526844e2e3651cc316d945
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5226384
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Richard Wang <richardwa@google.com>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Update use_goma warning for linux and mac, to add date of removal of
use_goma (Feb 7th) and a warning that delays will be added (Jan 22).
Bug: b/277197166
Change-Id: Id567e5bf704c5d80e5122663c6d8194ac510d992
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5200850
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Adds the same warning to every use_goma build that is currently printed
on a gn gen with use_goma=true.
Bug:b/277197166
Change-Id: Icf9781f814416dfbeb2826fc88372465155fcd2b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5123231
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
This is to remove 1~3 seconds from no-op build using autoninja.
Bug: b/309720176
Change-Id: I2c15a43517d5a99eae794015c18c884a3f3fdef4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5092847
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
This disallows Googler to use non-google account with reclient and
siso from corp machine.
Bug: b/309720176
Change-Id: I8077eff8bbf47f579462e1fefa3609a5d492a013
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035266
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Fumitoshi Ukai
Matt Stark
Philipp Wollermann
Mohamed Heikal
Aleksey Khoroshilov
Tomasz Śniatowski
Mark Mentovai
Max Ihlenfeldt
Junji Watanabe
Andrew Grieve
Takuto Ikuta
Michael Savigny
L. David Baron