Chromium does not use CMake, however ninja with Goma is used by LLVM
team.
Change-Id: I38349d0574048b22b32365e45c587e604e045442
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2572805
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Vitaly Buka <vitalybuka@chromium.org>
In addition to setting GOMA_DISABLED, this CL also sets the
RBE_remote_disabled environment variable to autoninja.
Bug: 1149386
Change-Id: I9aa8be005a9cd473b7371eadcab83d6068e0e070
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2565550
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
It is sometimes helpful to do a full build of Chrome using goma while
online and then do incremental builds without goma while offline or with
poor network bandwidth. Turning off goma in gn args triggers a full
rebuild but the GOMA_DISABLED environment variable can be used to
disable goma without triggering a rebuild.
In order to make this feature easier to discover and use this change
adds support for -o/--offline to autoninja. autoninja -h will mention
this flag and autoninja.bat/autoninja.py handle it appropriately. This
means setting the environment variable in autoninja.bat, and using it
to adjust the -j value and stripping it out in autoninja.py.
The bash script that wraps autoninja.py on Linux and OSX did not need
updating because the Python script can emit 'GOMA_DISABLED=1 ninja ...'
and this is executed by bash.
This was produced during pair programming with jessemckenna@
Bug: b/172039612
Change-Id: Ifcfbc598ac20f23e5fe013e02979b5b8c2851b01
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2510818
Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Here we add slightly improved detection for the goma flag.
The detection of goma in autoninja does not work for the use_goma flag
when it does not simply appear at the beginning of the line.
Below are examples of multi-arg lines we now support:
is_debug=false use_goma=true is_official_build=false
use_goma=false# use_goma=true This comment is ignored
Also adding linux nice process launch feature to improve the user
experience when running a large build. This feature is disabled by
default like other autoninja features.
R=brucedawson@chromium.org
Change-Id: I8a04e7480c5e8d37d57fb71143c6066d46abb004
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2479683
Commit-Queue: Peter McNeeley <petermcneeley@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
The goma-on-borg backend is turning down. While chromium can
continue using the goma client for build acceleration, the
backend is swapped out to instead use Remote Build Execution,
a gcloud service.
Other downstream projects (e.g. Nest) switched out the goma
frontend as well as the backend to RBE. These projects use a
|use_rbe| gn flag to enable build acceleration through RBE.
This change allows autoninja to recognize this flag and treat it
as if it was using goma for the purposes of passing an
appropriate -j flag.
Bug: None
Test: run on an out/ directory with use_rbe enabled
Change-Id: I52d5771e92e9163234811b154fd36816639bf1b8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2391009
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Simeon Anfinrud <sanfin@chromium.org>
This was made by running `codespell` and `scspell`
and then checking the results.
Change-Id: I169fd5b40294f83015075b4a899fbca263821f25
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2144602
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Auto-Submit: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
This updates the comments and code regarding GOMA_DISABLED to more
precisely match its behavior and therefore avoid confusion.
Bug: b/140312943
Change-Id: I6208de282bd5ebf1d88f98c1f905cfc626de792e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1786676
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
The NINJA_SUMMARIZE_BUILD already triggers two types of build
performance data - setting NINJA_STATUS to give more details during the
build and running post_build_ninja_summary.py after the build.
One of the most common causes of slow builds is when CreateProcess
becomes slow, usually due to anti-virus overhead. Neither of these
reports make this problem easily visible, but ninja -d stats does. In
particular the StartEdge data is an excellent summary of CreateProcess
timing. If the StartEdge average (avg) is more than about 6,000 us
(6.0 ms) it might be worth investigating why.
Therefore this change makes it so that autoninja.py adds -d stats to the
command line when NINJA_SUMMARIZE_BUILD is set to 1.
Change-Id: I1b6b990708c784a2cd71d153713b422f090775e6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1788399
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
On the wasm team we build several CMake-based projects with ninja
(LLVM, binaryen, etc), and its useful for us (me at least) to be able
to use autoninja with these project too.
Change-Id: I7e213448dbbe95ffe3d249c9c6a3d4baa41f50d1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1682742
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Sam Clegg <sbc@chromium.org>
This incorporates a fix to psutil's calculation of
the number of processors on Windows systems with
more than 64 cores.
Windows systems with more than 64 logical
processors divide the processors into groups, with
most applications using only one group, i.e. a
portion of the actual processing power available.
Prior to v5.4.4, psutil only counted the number of
processors in the first processor group. This
resulted in only partial utilization of high-
processing-power systems.
For example, on the 72-core P920, this change has
the following effect:
Before:
> autoninja -C out\nogoma chrome
"c:\src\depot_tools\ninja.exe" -C out\nogoma base -j 38
After:
> autoninja -C out\nogoma chrome
"c:\src\depot_tools\ninja.exe" -C out\nogoma base -j 74
Using this new version of psutil doubles the
number of processors used for building Chrome on
the P920 (when not using goma).
A similar bug exists in ninja.exe, so using
autoninja will be particularly important after
this fix for users of the P920 and other systems
with >64 cores. More fixes will be needed
elsewhere for similar bugs - see crbug.com/980967
for general progress.
Bug: 980270
Change-Id: I8de61a72cf95acf28ef1bcef1b0057b7b1225832
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1686081
Commit-Queue: Jesse McKenna <jessemckenna@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Also make autoninja.py a vpython script to use reliable version of
psutil.
Note: this change also makes autoninja always make a decision about
-j; there's no longer a default where it lets ninja pick. The code is
simpler this way and I think it's better because it lets developers
always see which -j is in effect when using autoninja (and that's its
exact purpose, if you wanted default you shouldn't have used autoninja).
R=dpranke@chromium.org
Bug: 976265
Change-Id: Ic9d12469729e4bf58da1ec1bd70437329519fc46
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1663904
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Gabriel Charette <gab@chromium.org>
Auto-Submit: Gabriel Charette <gab@chromium.org>
Ran "2to3 -w -n -f print ./" and manually added imports.
Ran "^\s*print " and "\s+print " to find batch/shell scripts, comments and the like with embedded code, and updated them manually.
Also manually added imports to files, which used print as a function, but were missing the import.
The scripts still work with Python 2.
There are no intended behaviour changes.
Bug: 942522
Change-Id: Id777e4d4df4adcdfdab1b18bde89f235ef491b9f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1595684
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Auto-Submit: Raul Tambre <raul@tambre.ee>
Regex strings need to be raw or else double-escape all the \ characters,
but Python 2.7 doesn't reliably enforce this, so bugs have crept in.
The buggy regex from create_installer_archive.py that this was copied
from was deleted a while ago. However this bad pattern is pervasive in
Chrome and depot_tools, as crudely found by searching for:
re\.[a-z]*\('.*\\[dsw\(\.]
An earlier instance of this was found when running some of our scripts
with Python 3.8.
Bug: 958138
Change-Id: If7ded5ae13f8cc36a5f6277c6ae0a2f54f88c3e4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1590191
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Large parallelism causes too many open files error.
Let me limit to 500 for Mac now.
Bug: 936864
Change-Id: I2fccc8cf14483c6f34d84c84d82c44df6e4f3177
Reviewed-on: https://chromium-review.googlesource.com/c/1496675
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
On Windows, without quote, whole return value would be treated as
a command, and the execution would fail.
On Linux and Mac, without quote, if depot_tools is settled under
directories with ' ', command execution would fail because paths are
separated in a wrong way.
To make such a return value work on Linux and Mac, the shell script
started to use eval.
Bug: 902930
Change-Id: I9bb74585294af565988c0b844b6b113a5c685530
Reviewed-on: https://chromium-review.googlesource.com/c/1325249
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
Since ninja uses getopt_long to parse the argument, we need to suppose
-j100 and -tclean.
Please see ReadFlags function in:
https://github.com/ninja-build/ninja/blob/master/src/ninja.cc
Change-Id: If61f05e66be546a591549f6f153cce994c972309
Reviewed-on: https://chromium-review.googlesource.com/c/1335428
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
When people execute autoninja, I believe people would expect ninja
in the autoninja directory would be used. However, the original code
find ninja from PATH, and ninja in other directory could be used.
People usually do not notice this because having depot_tools in
PATH is recommended for Chromium developers. However, in some
environments, an old version ninja is pre-installed, and unexpected
version ninja could be used upon PATH environment.
Also, in most of bots, depot_tools directory is not included in PATH.
Autoninja execution would fail there because the system cannot find
ninja.
Bug: b/77176746
Change-Id: Iad8bd952dc1e34a9d303fd5b493c555156369a17
Reviewed-on: https://chromium-review.googlesource.com/c/1319489
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@chromium.org>
Reviewed-by: Shinya Kawanaka <shinyak@chromium.org>
Commit-Queue: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
Auto-Submit: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
autoninja automatically sets -l <num_cores>. -l option makes ninja
not invoke a new command if current CPU load average is above -l.
However, as far as I investigated, -l <num_cores> make the build
much slower than without -l especially on Linux & OSX machines with
small numbers of cores. I should say -l decreases the build
performance. When I build Chromium with Goma with the same -j with
autoninja without -l, the load average goes more than the number of
cores while keeping the machine working as usual.
Also, ninja can invoke commands until the spike of command invocation
is reflected to load average, -l might not mitigate for a machine
to get stuck by too high load.
Note that from what I understand from the implementation, Windows
ninja's posix-compatible load average is always less than num_cores.
i.e. -l <num_cores> won't limit the process invocation.
5984986459/src/util.cc (L479)
Let me make autoninja not set -l and bring better build performance
on machines with small number of cores.
Bug: b/117810340
Change-Id: I50f231f1a8976f8ecfc3a0c778f0f1ac98d3827f
Reviewed-on: https://chromium-review.googlesource.com/c/1290611
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
Nina supports -C out/Default and -Cout/Default to specify the build
directory so autoninja should also. This change adds that support.
Bug: 890744
Change-Id: I5e824242ed4b333ac99f1ee9a649ffcfa03a812e
Reviewed-on: https://chromium-review.googlesource.com/c/1257586
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
When GOMA_DISABLED is set goma will use the local compiler. Autoninja
needs to understand this in order to avoid requesting too much
parallelism.
Change-Id: Ic124893dd583a401d0d9ad7fbd27ee9b6715fcfe
Reviewed-on: https://chromium-review.googlesource.com/1015402
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
This is the second attempt at fixing autoninja to allow passing '^^' to
it to specify that ninja should build the outputs of the specified file
instead of building that file itself. The problem is that '^' is a
special character and when extra layers of indirection are added the
number of '^' characters needed grows exponentially in some poorly
understood way. The first fix attempt just quoted the arguments that
autoninja.bat passed to autoninja.py, but that meant they came in as
one argument. This fix expands on that by modifying autoninja.py to
understand how to deal with the monolithic argument. With this change
this once again works:
autoninja -C out\debug_component ..\..\base\win\enum_variant.cc^^
It can be convenient to have a ninja.bat file which starts goma and lets
users keep typing the same build commands. However even with this fix
the previously recommended ninja.bat file must be invoked with four
'^' characters. If that is too much then the new recommended ninja.bat
is to copy autoninja.bat and modify as needed, perhaps like this:
@echo off
call python c:\goma\goma-win64\goma_ctl.py ensure_start >nul
FOR /f "usebackq tokens=*" %%a in (`python c:\src\depot_tools\autoninja.py "%*"`) do echo %%a & %%a
BUG: 758725
Change-Id: Ieee9cf343ee5f22e9988a1969cb7a7a90687666b
Reviewed-on: https://chromium-review.googlesource.com/656478
Reviewed-by: Sébastien Marchand <sebmarchand@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
The -t tools in ninja fail if -j or -l are specified. So, autoninja.py
needs to watch for -t and omit -j and -l if it is noticed.
R=sebmarchand@chromium.org
BUG=740227
Change-Id: I1418193daeab154178d15be60ab09551bacaf3af
Reviewed-on: https://chromium-review.googlesource.com/563775
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Using goma requires the developer to remember which build directories
use goma and which don't so that they can pass an appropriate -j number.
Getting this wrong makes builds slower, either by under utilizing
resources or by causing a self-inflicted DOS attack. Usage:
autoninja -C out/debug
autoninja looks at the settings for the specified build directory and
then selects either -j num_cores*20 or no -j flag based on the
use_goma setting.
You can set the NINJA_CORE_MULTIPLIER variable to change from the
default 20* multiplier. You can also use NINJA_CORE_ADDITION if you
want non-goma builds to specify -j with an offset to the number of
cores, such as this Linux command:
NINJA_CORE_ADDITION=-2 autoninja -C out/release base
This will tell autoninja to pass -j to ninja with num_cores-2 as the
parameter.
On Windows you can have a ninja.bat file (ahead of ninja on the path)
such that autoninja will automatically be used. It should contain this:
@call autoninja.bat %*
Change-Id: I4003e3fc323d1cbab612999c945b5a8dc5bc6655
Reviewed-on: https://chromium-review.googlesource.com/517662
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Vitaly Buka
Michael Savigny
Bruce Dawson
Peter McNeeley
Simeon Anfinrud
Takuto Ikuta
Quinten Yearsley
Sam Clegg
Jesse McKenna
Gabriel Charette
Raul Tambre
Yoshisato Yanagisawa
Allen Bauer