aiden
/
depot_tools
mirror of https://chromium.googlesource.com/chromium/tools/depot_tools
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
main
chrome/4147
chrome/3987
chrome/3865
chrome/3904
infra/config
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'main'
${ noResults }
depot_tools/autoninja.bat

35 lines
1.3 KiB
Batchfile
Raw Permalink Normal View History Unescape Escape

autoninja - ninja wrapper to make goma easier Using goma requires the developer to remember which build directories use goma and which don't so that they can pass an appropriate -j number. Getting this wrong makes builds slower, either by under utilizing resources or by causing a self-inflicted DOS attack. Usage: autoninja -C out/debug autoninja looks at the settings for the specified build directory and then selects either -j num_cores*20 or no -j flag based on the use_goma setting. You can set the NINJA_CORE_MULTIPLIER variable to change from the default 20* multiplier. You can also use NINJA_CORE_ADDITION if you want non-goma builds to specify -j with an offset to the number of cores, such as this Linux command: NINJA_CORE_ADDITION=-2 autoninja -C out/release base This will tell autoninja to pass -j to ninja with num_cores-2 as the parameter. On Windows you can have a ninja.bat file (ahead of ninja on the path) such that autoninja will automatically be used. It should contain this: @call autoninja.bat %* Change-Id: I4003e3fc323d1cbab612999c945b5a8dc5bc6655 Reviewed-on: https://chromium-review.googlesource.com/517662 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
8 years ago
@echo off
:: Copyright 2017 The Chromium Authors. All rights reserved.
:: Use of this source code is governed by a BSD-style license that can be
:: found in the LICENSE file.
Set NINJA_STATUS for better build performance summarizing When NINJA_SUMMARIZE_BUILD=1 then autoninja prints a summary of build performance at the end. This change makes it so that ninja prints more detailed build statistics during the build. In particular, slow process creation is a common but difficult to see bottleneck. This change makes it so that if NINJA_SUMMARIZE_BUILD=1 then the number of running build processes will be displayed. Bug: 787983 Change-Id: Ic0907e23e7f762e23e4795059b37085e0cb8c4ea Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1582802 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
6 years ago
setlocal
Reland "Check whether goma is running when it is enabled" This reverts commit 428143ee24c5f084c8dfb38cd17f07b4f7ba9bf7. Reason for revert: Fixing the issues revealed by the original change by avoiding python3 and by checking for the existence of gomacc[.exe] before running it. This also relands 2241db8a1f6c05325c8a0c9eb5426d96cd6fa984 - "Avoid capture_output to support Python 3.6", to simplify relanding and any possible reverts. Original change's description: > Revert "Check whether goma is running when it is enabled" > > This reverts commit b7ddc5a0091bcd4d070fcd91027d7099338e84b9. > > Reason for revert: > This broke the builder where depot_tools is not in PATH. > https://logs.chromium.org/logs/infra-internal/buildbucket/cr-buildbucket.appspot.com/8858077852309878080/+/u/build/stdout > > Original change's description: > > Check whether goma is running when it is enabled > > > > One of the mistakes one can make when running ninja is having goma > > enabled (use_goma=true in args.gn) but not having goma running. This can > > lead to ~1,000 failed compile steps, which is messy. > > > > This change teaches autoninja.py to check whether goma is running. If > > not then it tells autoninja to just print a warning message. The > > check costs roughly 30 ms which seems reasonable. > > > > In fact, because this change also switches away from vpython (necessary > > to use python3 to use subprocess.run) it actually runs about 600 ms > > _faster_ than before this change. > > > > If build acceleration is requested through use_rbe then no checking for > > whether the service is running is done. That could be added in the > > future. > > > > autoninja.py could auto-start goma but that is error prone and has > > limited additional value. > > > > This was tested on Linux, OSX, and Windows. > > > > Bug: 868590, b/174673874 > > Change-Id: Ie773e574878471e5136b9b82d52f86af3d848318 > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2627014 > > Commit-Queue: Bruce Dawson <brucedawson@chromium.org> > > Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@google.com> > > TBR=yyanagisawa@google.com,dpranke@google.com,brucedawson@chromium.org,sanfin@chromium.org,infra-scoped@luci-project-accounts.iam.gserviceaccount.com > > Change-Id: I57a6c73ea853259f3d1ec7ad0ce51e495acc96db > No-Presubmit: true > No-Tree-Checks: true > No-Try: true > Bug: 868590 > Bug: b/174673874 > Bug: 1167064 > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2632018 > Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@google.com> > Commit-Queue: Yoshisato Yanagisawa <yyanagisawa@google.com> TBR=yyanagisawa@google.com,dpranke@google.com,brucedawson@chromium.org,sanfin@chromium.org,infra-scoped@luci-project-accounts.iam.gserviceaccount.com # Not skipping CQ checks because this is a reland. Bug: 868590 Bug: b/174673874 Bug: 1167064 Change-Id: I8aa6830259bc18f8e7926cd0bf5c62e671c74a2d Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2634201 Reviewed-by: Bruce Dawson <brucedawson@chromium.org> Reviewed-by: Dirk Pranke <dpranke@google.com> Reviewed-by: Fumitoshi Ukai <ukai@google.com> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@google.com> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
4 years ago
set scriptdir=%~dp0
Handle goma_ctl/autoninja /? With nested batch files it is very common for /? arguments to end up being interpreted as a request for help on the batch file "call" command. That is, if a user invokes "goma_ctl /?" then this turns into something like this: @call ...\python3.bat %~dp0\.cipd_bin\goma_ctl.py %* The name of the script to be invoked is ignored and this is treated like: @call /? This is particularly problematic for commands like goma_ctl and autoninja which don't print help if no arguments are passed, thus making finding the correct help incantation slightly challenging. This special-cases /? in these two batch files. Some other common batch files were tested and found to be not affected, so while this is not a complete fix it does hit some of the most important places. Change-Id: Ic9b0455c2f8b085666862bc4495d9bad445dfeaf Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4080991 Reviewed-by: Dirk Pranke <dpranke@google.com> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
if "%*" == "/?" (
rem Handle "autoninja /?" which will otherwise give help on the "call" command
autoninja: check RBE project, not account Account check would become too slow. We'll check RBE project to use instead. On corp machine, our policy to use @google.com account and rbe-chrome-untrusted to build chromium/chrome. We don't allow rbe-chromium-untrusted with @chromium.org on corp machine. On non-corp machine, user couldn't use rbe-chrome-untrusted because it's @google.com only, and corp security policy doesn't allow @google.com account on non-corp machine. Bug: b/364318216 Change-Id: I0f3a19e105b050aef6a62e1b25b45b1722382a34 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5848450 Reviewed-by: Scott Lee <ddoman@chromium.org> Reviewed-by: Michael Savigny <msavigny@google.com> Commit-Queue: Fumitoshi Ukai <ukai@google.com> Reviewed-by: Junji Watanabe <jwata@google.com> Reviewed-by: Philipp Wollermann <philwo@google.com>
6 months ago
@call %scriptdir%python-bin\python3.bat %~dp0\ninja.py --help
Handle goma_ctl/autoninja /? With nested batch files it is very common for /? arguments to end up being interpreted as a request for help on the batch file "call" command. That is, if a user invokes "goma_ctl /?" then this turns into something like this: @call ...\python3.bat %~dp0\.cipd_bin\goma_ctl.py %* The name of the script to be invoked is ignored and this is treated like: @call /? This is particularly problematic for commands like goma_ctl and autoninja which don't print help if no arguments are passed, thus making finding the correct help incantation slightly challenging. This special-cases /? in these two batch files. Some other common batch files were tested and found to be not affected, so while this is not a complete fix it does hit some of the most important places. Change-Id: Ic9b0455c2f8b085666862bc4495d9bad445dfeaf Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4080991 Reviewed-by: Dirk Pranke <dpranke@google.com> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
exit /b
)
Replace all rem comments with :: comments in batch files Though the Microsoft docs recommend using rem to write comments, it has a shortcoming - redirection characters (< or >) or pipes (|) aren't allowed. However, :: though undocumented, is a better alternative as it allows such characters. Signed-off-by: Darshan Sen <raisinten@gmail.com> Change-Id: I95a0445168527ab5087246238f0216d5f6177046 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400399 Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Reviewed-by: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>
3 years ago
:: If a build performance summary has been requested then also set NINJA_STATUS
:: to trigger more verbose status updates. In particular this makes it possible
:: to see how quickly process creation is happening - often a critical clue on
:: Windows. The trailing space is intentional.
autoninja.bat: have space suffix in ninja status This improves output from ninja a bit on Windows when NINJA_SUMMARIZE_BUILD is set. The trailing space was removed in https://crrev.com/c/3400399. Change-Id: Ifb14129a436a2673969f529d0c9143118dd87d6b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3833272 Auto-Submit: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Bruce Dawson <brucedawson@chromium.org> Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
3 years ago
if "%NINJA_SUMMARIZE_BUILD%" == "1" set "NINJA_STATUS=[%%r processes, %%f/%%t @ %%o/s : %%es ] "
Set NINJA_STATUS for better build performance summarizing When NINJA_SUMMARIZE_BUILD=1 then autoninja prints a summary of build performance at the end. This change makes it so that ninja prints more detailed build statistics during the build. In particular, slow process creation is a common but difficult to see bottleneck. This change makes it so that if NINJA_SUMMARIZE_BUILD=1 then the number of running build processes will be displayed. Bug: 787983 Change-Id: Ic0907e23e7f762e23e4795059b37085e0cb8c4ea Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1582802 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
6 years ago
Execute Ninja / Siso directly from autoninja.py. Instead of printing a command-line, we just directly call into the respective main functions from Python. This saves spawning another interpreter and prevents things that can go wrong from having to quote, unquote, split and tunnel arguments through shells. Part of my bigger auto{ninja,siso} refactoring. Tested: - Handling of the ^^ suffix on Windows still works correctly. - Handling of error codes - i.e.; making sure "autoninja base_unittests && base_unittests.exe" behaves properly in the success/failure case. - Make sure the command prompt title is reliably reset on exit. I tested autoninja with all combinations of these: - Host platform: Linux, macOS, Windows - Remote GN args: <none>, use_goma=true, use_remoteexec=true - Siso GN args: <none>, use_siso=true - Targets: base, ../../base/types/expected_macros_unittest.cc^ (on Linux) and ../../base/types/expected_macros_unittest.cc^^ (on Windows) R=brucedawson@chromium.org, jwata@google.com, tikuta@chromium.org Bug: b/293657720 Change-Id: I275a775fdc5abb6555f79d4beab76cd0914d4bd6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4924185 Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Junji Watanabe <jwata@google.com> Commit-Queue: Philipp Wollermann <philwo@chromium.org>
1 year ago
:: Execute autoninja.py and pass all arguments to it.
autoninja: check RBE project, not account Account check would become too slow. We'll check RBE project to use instead. On corp machine, our policy to use @google.com account and rbe-chrome-untrusted to build chromium/chrome. We don't allow rbe-chromium-untrusted with @chromium.org on corp machine. On non-corp machine, user couldn't use rbe-chrome-untrusted because it's @google.com only, and corp security policy doesn't allow @google.com account on non-corp machine. Bug: b/364318216 Change-Id: I0f3a19e105b050aef6a62e1b25b45b1722382a34 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5848450 Reviewed-by: Scott Lee <ddoman@chromium.org> Reviewed-by: Michael Savigny <msavigny@google.com> Commit-Queue: Fumitoshi Ukai <ukai@google.com> Reviewed-by: Junji Watanabe <jwata@google.com> Reviewed-by: Philipp Wollermann <philwo@google.com>
6 months ago
@call %scriptdir%python-bin\python3.bat %scriptdir%autoninja.py "%%*"
Script to summarize performance of ninja builds post_build_ninja_summary.py analyzes the .ninja_log file in an output directory to summarize build performance. It lists the most expensive build steps, and build-step types, printing both their elapsed time and, more importantly, their "weighted" time which is a measure of their contribution to the total build time. For each time segment within the build the running build steps accumulate weighted time that is proportional to the elapsed time divided by the number of tasks running. If a thousand compilation steps are running in parallel then they will each be "charged" for 1/1,000th of the elapsed time. If a single link step is running alone then it is charged with all of the elapsed time. Bug: chromium:787983 Change-Id: Id5aea715f798a16415dd0365a27f0051202668e5 Reviewed-on: https://chromium-review.googlesource.com/871988 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Erik Staab <estaab@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
7 years ago
@if errorlevel 1 goto buildfailure
introduce ninjalog_uploader to autoninja.bat This is windows version of https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1345255 Changed to use call to invoke python because python is bat file. Bug: 900161 Change-Id: I0332306387614153be7ae6dbde05fb2bc93559b1 Reviewed-on: https://chromium-review.googlesource.com/c/1355641 Commit-Queue: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org>
6 years ago
Replace all rem comments with :: comments in batch files Though the Microsoft docs recommend using rem to write comments, it has a shortcoming - redirection characters (< or >) or pipes (|) aren't allowed. However, :: though undocumented, is a better alternative as it allows such characters. Signed-off-by: Darshan Sen <raisinten@gmail.com> Change-Id: I95a0445168527ab5087246238f0216d5f6177046 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400399 Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Reviewed-by: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>
3 years ago
:: Use call to invoke python script here, because we use python via python3.bat.
Have autoninja.bat run depot_tools python3 This changes autoninja.bat so that it refers explicitly to the depot_tools version of Python rather than depending on what is in the path. This helps to make autoninja behave more consistently and makes it easier for us to remove python.bat from the path. Bug: 777069 Change-Id: I6d878f58e2f8be70d06a182d059caa9fbedaedbd Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2753458 Reviewed-by: Dirk Pranke <dpranke@google.com> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
4 years ago
@if "%NINJA_SUMMARIZE_BUILD%" == "1" call %scriptdir%python-bin\python3.bat %scriptdir%post_build_ninja_summary.py %*
introduce ninjalog_uploader to autoninja.bat This is windows version of https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1345255 Changed to use call to invoke python because python is bat file. Bug: 900161 Change-Id: I0332306387614153be7ae6dbde05fb2bc93559b1 Reviewed-on: https://chromium-review.googlesource.com/c/1355641 Commit-Queue: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org>
6 years ago
Set errorlevel in .bat wrappers on exit /b where appropriate. Fixed goma_auth/ctl return code on Windows which was always zero. Fixed other places where exit /b was used without %errorlevel%. Change-Id: Ib0a647818c595f843a95762db023b111a447a66a Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3532153 Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Josip Sokcevic <sokcevic@google.com> Auto-Submit: Aleksey Khoroshilov <akhoroshilov@brave.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
3 years ago
exit /b %ERRORLEVEL%
Script to summarize performance of ninja builds post_build_ninja_summary.py analyzes the .ninja_log file in an output directory to summarize build performance. It lists the most expensive build steps, and build-step types, printing both their elapsed time and, more importantly, their "weighted" time which is a measure of their contribution to the total build time. For each time segment within the build the running build steps accumulate weighted time that is proportional to the elapsed time divided by the number of tasks running. If a thousand compilation steps are running in parallel then they will each be "charged" for 1/1,000th of the elapsed time. If a single link step is running alone then it is charged with all of the elapsed time. Bug: chromium:787983 Change-Id: Id5aea715f798a16415dd0365a27f0051202668e5 Reviewed-on: https://chromium-review.googlesource.com/871988 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Erik Staab <estaab@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
7 years ago
:buildfailure
[autoninja] upload ninjalog even in failed build I noticed current autoninja does not upload build log in failed build. Change-Id: Ie58646b483e130769ad22113953c0b95b3145548 Reviewed-on: https://chromium-review.googlesource.com/c/1441892 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
6 years ago
Replace all rem comments with :: comments in batch files Though the Microsoft docs recommend using rem to write comments, it has a shortcoming - redirection characters (< or >) or pipes (|) aren't allowed. However, :: though undocumented, is a better alternative as it allows such characters. Signed-off-by: Darshan Sen <raisinten@gmail.com> Change-Id: I95a0445168527ab5087246238f0216d5f6177046 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400399 Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Reviewed-by: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>
3 years ago
:: Return an error code of 1 so that if a developer types:
:: "autoninja chrome && chrome" then chrome won't run if the build fails.
Fix autoninja && chrome on failed builds A handy usage pattern for autoninja.bat that I was not aware of is to go autoninja -C out\Default chrome && chrome This will build chrome and then run it, but only run it if the build succeeds. The addition of post_build_ninja_summary.py broke this by losing the error code. This change fixes it by using black magic to set an error code in the failure case. Bug: chromium:787983 Change-Id: Ib87fd1799816e19d56de76e08e0f9688be903d80 Reviewed-on: https://chromium-review.googlesource.com/916705 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
7 years ago
cmd /c exit 1