This is to make sure that the command runs as intended and make more
verbose if it has some error.
Bug: b/312632221
Change-Id: I44b372452b37b744f6c02c5a2e34fafe3717e351
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5051585
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Siso is going to generate non-empty .ninja_log for the analysis tools.
https://crrev.com/c/4907677
This CL fixes the comment about empty ninja_log.
It's not compatible with Ninja's .ninja_log. So `gn clean` will still
be required.
Bug: b/298594790
Change-Id: Ib4c60f3ed22f516d6f7e2847aaf57e228121eccf
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4937692
Auto-Submit: Junji Watanabe <jwata@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
siso doesn't generate .ninja_log files so post_build_ninja_summary.py
doesn't work, but it _does_ generate siso_metrics.json files which can
be used to generate similar reports to post_build_ninja_summary.py.
Therefore, when a siso_metrics.json file is detected the siso metrics
summary command is invoked, thus preserving (more or less) the old
behavior.
Bug: b/293657720
Change-Id: I084402ca4dca9895b502ab336fa7b45b770f4768
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4854377
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
These are currently produced by Siso, until b/298594790 is addressed.
Before:
$ post_build_ninja_summary.py -C out/fastbuild-siso-reclient
Traceback (most recent call last):
File "/usr/local/google/home/philwo/depot_tools/post_build_ninja_summary.py", line 366, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/google/home/philwo/depot_tools/post_build_ninja_summary.py", line 356, in main
entries = ReadTargets(log, False)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/google/home/philwo/depot_tools/post_build_ninja_summary.py", line 123, in ReadTargets
assert header == '# ninja log v5\n', \
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: unrecognized ninja log version ''
After:
$ post_build_ninja_summary.py -C out/fastbuild-siso-reclient
<nothing>
Bug: b/298594790
Fixed: b/297349353
Change-Id: I10d4613e7386707276003fe0fd05cb5b0914be46
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4846349
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Philipp Wollermann <philwo@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.
Reformatted using:
files=( $(
git ls-tree -r --name-only HEAD | \
grep -Ev -e '^(third_party|recipes)/' | \
grep '\.py$';
git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"
The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py
These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py
Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
post_build_ninja_summary.py normally sorts its output by weighted time,
which is elapsed time divided by how many steps are running. Sometimes
it is more appropriate to sort by elapsed (wall clock) time instead.
This adds a -e/--elapsed_time_sorting option to do this.
Change-Id: I13a1b094f8fd7f6795a871bee91932ce0ddab79d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4285697
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Makes it easier to run on linux/mac.
Change-Id: I0ebf512cd576a5c8aafbddc7c6b6f8e2bd6919d9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3971922
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Auto-Submit: Andrew Grieve <agrieve@chromium.org>
In Python 3 dict.values() is not a sortable type so it must be converted
to a list prior to returning targets data from the ReadTargets function.
In Python 3 you cannot compare arbitrary objects. This could happen when
sorting task_start_stop_times if the time and types were identical. This
is fixed by using the first two entries in the tuple as the sorting key.
When a .ninja_log file is corrupt it triggers a diagnostic message which
has only caused confusion. This changes that message to suggest the most
likely root cause (corruption) since that is the only cause I have seen.
If a build step generates multiple targets with long names then wrapping
of lines may occur. To avoid that the target description size is capped.
Bug: 941669
Change-Id: I2a808d2629a639e231ce2dbf2dab41251110b156
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2432409
Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Increase the number of long times by extension to report from 5 to 10
since java build steps produce a number of different extension types. It
is difficult to distinguish between android vs non-android builds, so
increase this default for all platforms.
Prioritize .jar outputs for java targets and allow up to two levels of
extensions to be recognized.
e.g.
file.interface.jar would have .interface.jar as its full extension.
file.javac.jar would have .javac.jar as its full extension.
Some of these build steps require .jar outputs, which is why these
targets use a secondary extension to differentiate.
Example output: https://crrev.com/c/2142395
Bug: chromium:1067273
Change-Id: Id5780980f60841c3384d91bf96121c6dec3e8bed
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2142163
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Peter Wen <wnwen@chromium.org>
Auto-Submit: Peter Wen <wnwen@chromium.org>
The post_build_ninja_summary.py script, which autoninja automatically
runs if NINJA_SUMMARIZE_BUILD=1 is set, summarizes build time by file
extension. This is a good default summary but it can be handy to create
additional output categories. This change allows using fn_match style
wildcards to create additional categories which will then be summarized.
For instance:
python post_build_ninja_summary.py -C out/Def -s prec*.obj;blink*.obj
In addition the chromium_step_types environment variable can be set so
that when autoninja runs this script there will be additional summary
categories.
This has been used most recently when measuring the cost of creating and
using blink precompiled header files.
Bug: 1061326
Change-Id: I080b4ddd06d045a2351220e202cd9eec111bf00e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2093145
Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
The build summary script was overly aggressive about summarizing .mojom
files under the mojo category. This was intended to measure the cost of
generating source from mojo files but it accidentally pulled in the
.mojom object files.
Categorizing different types of object files might be worthwhile but
that should be done intentionally and consistently.
Change-Id: Iab6b7e94797ce7f1ed46805034b4f274c88617e0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2067647
Commit-Queue: Aaron Gable <agable@chromium.org>
Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
The "CPU time" values printed by post_build_ninja_summary.py are not
actually measure of CPU time, they are measures of wall-clock time or
elapsed time. This just changes the labeling of the numbers - the actual
numbers printed are unchanged.
The "weighted time" number continues to be the more important number
since it is an approximation of how much the build step(s) are affecting
build time. See the comments in the script for details.
Change-Id: Ibdb4efdd327ece34492ab10337c234a826514197
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1900019
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Ran "2to3 -w -n -f print ./" and manually added imports.
Ran "^\s*print " and "\s+print " to find batch/shell scripts, comments and the like with embedded code, and updated them manually.
Also manually added imports to files, which used print as a function, but were missing the import.
The scripts still work with Python 2.
There are no intended behaviour changes.
Bug: 942522
Change-Id: Id777e4d4df4adcdfdab1b18bde89f235ef491b9f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1595684
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Auto-Submit: Raul Tambre <raul@tambre.ee>
post_build_ninja_summary.py detects a new incremental build by looking
for the end time stamps to go backwards. This means that repeated builds
that have a single long step will not be reliably detected and will, in
some cases, be missed entirely. This hits me sometimes when doing link
testing - delete an output DLL, rebuild, and the old results may be
displayed again.
This change updates the script to check for a different duration for an
existing record as well as an earlier end time should handle almost all
cases.
This also renames the targets local variable to avoid confusion due
to the targets member variable of the Targets class.
Bug: chromium:787983
Change-Id: I60a371df75d6cb7a55bd46b38156cb109feb8f15
Reviewed-on: https://chromium-review.googlesource.com/1061413
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Sometimes the log files are not named .ninja_log and then it's
good if you can give the script the actual name and path.
Change-Id: I59e9812c81a83ffac3ecdc8b29887e99bf71b60f
Reviewed-on: https://chromium-review.googlesource.com/1012112
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Daniel Bratell <bratell@opera.com>
post_build_ninja_summary.py gives a summary of a ninja build. It can be
run standalone or it can be run automatically by autoninja. This CL
updates the Python script and the autoninja bash script to make this
work on Linux. This includes removing a zero-value assert, and ensuring
that .so files get categorized as such.
Change-Id: I2d59ab129f5ce70117beeb119719f8432bfbab7c
Reviewed-on: https://chromium-review.googlesource.com/915053
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
post_build_ninja_summary.py analyzes the .ninja_log file in an output
directory to summarize build performance. It lists the most expensive
build steps, and build-step types, printing both their elapsed time and,
more importantly, their "weighted" time which is a measure of their
contribution to the total build time.
For each time segment within the build the running build steps
accumulate weighted time that is proportional to the elapsed time
divided by the number of tasks running. If a thousand compilation steps
are running in parallel then they will each be "charged" for 1/1,000th
of the elapsed time. If a single link step is running alone then it is
charged with all of the elapsed time.
Bug: chromium:787983
Change-Id: Id5aea715f798a16415dd0365a27f0051202668e5
Reviewed-on: https://chromium-review.googlesource.com/871988
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Erik Staab <estaab@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Fumitoshi Ukai
Takuto Ikuta
Ho Cheung
Junji Watanabe
Bruce Dawson
Philipp Wollermann
Mike Frysinger
Andrew Grieve
Peter Wen
Edward Lemur
Raul Tambre
Daniel Bratell