Commit Graph

136 Commits (51690612da20de7e5f5ac612b8fdb8b6262f1e46)

Author SHA1 Message Date
Bruce Dawson 04206434e7 Improve .boto message visibility
An out of date .boto file can prevent Google employees from being able
to download the Chromium toolchain from Google storage. An error message
was added to gsutil.py to help developers diagnose this, but the message
was swallowed by get_toolchain_if_necessary.py. This change prints the
error message, and adds '*' * 80 separators to make it more visible.

An obsolete import is also removed.

Bug: 1414152
Change-Id: I69d5a786672304ed363c291e0a8ef89f8be0f2e0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4310059
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Ho Cheung d437c34c1c [win_toolchain] Update toolchain version
Chromium has now switched to MSVS2022+SDK22621 toolchain.
We should update some parameters and annotations in the file
to avoid inconsistent situations.

After applying this patch locally, no abnormal situation
is found. We can pack the toolchain normally.

Change-Id: I10ca115a076001efaa2b020251f2e448622598dc
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4297895
Auto-Submit: Ho Cheung <uioptt24@gmail.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Bruce Dawson 0115386a26 Switch from VS 2017/2019 support to VS 2019/2022
This drops support for packaging toolchains based on VS 2017 and adds
support for VS 2022. This will allow us to package a toolchain for
building Chrome using the latest tools.

Bug: 1273169
Change-Id: I1981c935d1a9f3f4ae25e4643b9956fbdd705be1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4054349
Reviewed-by: Etienne Pierre-Doray <etiennep@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Josip Sokcevic 6a8680a80e Use python3 for cpplint and get_toolchain on Win
R=gavinmak@google.com

Change-Id: I7d9ca9530640ffa4b8e60c247f809b0ff477d88e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3673716
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
3 years ago
David Dorwin 7d2693ad8d Remove scottmg from OWNERS
depot_tools version of https://crrev.com/c/3526975.

Change-Id: I9c46525322ebfa36ebde67bb5fe00249cdf51fca
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3591072
Reviewed-by: Scott Graham <scottmg@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: David Dorwin <ddorwin@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Josip Sokcevic 4de5deacd4 Explicitly run everything with python3
R=aravindvasudev@google.com, gavinmak@google.com

Change-Id: Iaa5e960c6226dea3a0814c7cb778d952eafb1502
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3525960
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
3 years ago
Chris Blume bd68291e20 Reland "Support multiple VS installations" (with fix)
This reverts commit 9ce8be3339.

Reason for revert: To apply fix

Original change's description:
> Revert "Support multiple VS installations"
>
> This reverts commit 36d41ceff8.
>
> Reason for revert: Script references VS_VERSION variable that was
> renamed in the prior change, so it doesn't work.
>
> Original change's description:
> > Support multiple VS installations
> >
> > Currently, package_from_installed.py assumes only one version of VS is
> > installed. It takes the path of the first installation.
> >
> > This could be incorrect in several ways:
> > - Maybe both 2017 and 2019 (the supported versions) are installed and
> >   although the user specified using 2019, the 2017 path comes first.
> > - Maybe 2019 and 2022 are installed, and the 2022 path is used even
> >   though it isn't supported.
> >
> > This CL fixes that issue by parsing the vswhere.exe output to confirm
> > the VS version matches what the user specified, using its corresponding
> > path.
> >
> > Change-Id: I2029a4f7126d0a45b5370ad58ab257df55571b3b
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3458722
> > Reviewed-by: Gavin Mak <gavinmak@google.com>
> > Reviewed-by: Chris Blume <cblume@chromium.org>
> > Commit-Queue: Chris Blume <cblume@chromium.org>
> > Auto-Submit: Chris Blume <cblume@chromium.org>
>
> Change-Id: I3d9147a7786f7f54f861087d16967b75d4afe2c5
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3504193
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>

Change-Id: Ica90cb8d5ce08b8b127da64969401cb40d4aee63
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3497899
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Chris Blume <cblume@chromium.org>
3 years ago
Chris Blume fba3a30684 Only call GetVSPath() once
Currently, GetVSPath() is called twice. The result if that call is used
locally in both situations.

This commit makes only one call, passing the result to where it is
needed.

Change-Id: I186bd7f0387091ad16c2791de99d1de8a89f266c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3510528
Auto-Submit: Chris Blume <cblume@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Bruce Dawson 9ce8be3339 Revert "Support multiple VS installations"
This reverts commit 36d41ceff8.

Reason for revert: Script references VS_VERSION variable that was
renamed in the prior change, so it doesn't work.

Original change's description:
> Support multiple VS installations
>
> Currently, package_from_installed.py assumes only one version of VS is
> installed. It takes the path of the first installation.
>
> This could be incorrect in several ways:
> - Maybe both 2017 and 2019 (the supported versions) are installed and
>   although the user specified using 2019, the 2017 path comes first.
> - Maybe 2019 and 2022 are installed, and the 2022 path is used even
>   though it isn't supported.
>
> This CL fixes that issue by parsing the vswhere.exe output to confirm
> the VS version matches what the user specified, using its corresponding
> path.
>
> Change-Id: I2029a4f7126d0a45b5370ad58ab257df55571b3b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3458722
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Reviewed-by: Chris Blume <cblume@chromium.org>
> Commit-Queue: Chris Blume <cblume@chromium.org>
> Auto-Submit: Chris Blume <cblume@chromium.org>

Change-Id: I3d9147a7786f7f54f861087d16967b75d4afe2c5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3504193
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Chris Blume 36d41ceff8 Support multiple VS installations
Currently, package_from_installed.py assumes only one version of VS is
installed. It takes the path of the first installation.

This could be incorrect in several ways:
- Maybe both 2017 and 2019 (the supported versions) are installed and
  although the user specified using 2019, the 2017 path comes first.
- Maybe 2019 and 2022 are installed, and the 2022 path is used even
  though it isn't supported.

This CL fixes that issue by parsing the vswhere.exe output to confirm
the VS version matches what the user specified, using its corresponding
path.

Change-Id: I2029a4f7126d0a45b5370ad58ab257df55571b3b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3458722
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Chris Blume <cblume@chromium.org>
Commit-Queue: Chris Blume <cblume@chromium.org>
Auto-Submit: Chris Blume <cblume@chromium.org>
3 years ago
Chris Blume 0cfa90e203 Adjust variable names to match style guide
Currently, several non-consts are named as if they are const. The style
guide has a naming convention that fits these more accurately.

This CL changes the names to match the style guide.

Change-Id: Ib3425b76208e853b9049abe0e1be40e179124c57
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3472135
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Chris Blume <cblume@chromium.org>
Auto-Submit: Chris Blume <cblume@chromium.org>
3 years ago
Chris Blume d6d7a05085 Replace hard-coded '\\' with os.path.sep
Currently, the Windows-specific path separated is hard-coded in many locations.

This CL replaces several of them with os.path.sep to be less fragile.

Change-Id: Ib8ffca5743254949808a6b67e4915f8f01e044e8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3470977
Auto-Submit: Chris Blume <cblume@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Chris Blume b3c6512be6 Correct trailing path separator
In [1], I landed a change that contained a bug. It meant to trim off a
trailing path separator. Instead of trimming the end, it trimmed
everything but the first char (since the path separator is 1 char long).

This CL fixes that bug.

[1] https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3451982

Change-Id: Ie4aebfc1042908437bd603da1c20681f19c50f73
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3473285
Auto-Submit: Chris Blume <cblume@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Chris Blume 1ba4135766 Replace hard-coded Windows SDK path
Currently, the Windows SDK path is hard-coded. This does not work if the
SDK is installed in a different path or drive.

This CL fetches the actual Windows SDK path and uses it instead.

Change-Id: I31808ba0dbe76f47253fc165ccbd8f027bf396ec
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3451982
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Chris Blume <cblume@chromium.org>
Auto-Submit: Chris Blume <cblume@chromium.org>
3 years ago
Viktor Samun 0aabd63461 Fix python2/3 compatibility in DownloadUsingHttp function
Change-Id: I59ed965d066b7875c1739408054498b328652be5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2988315
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
4 years ago
Dirk Pranke 1fc595b55c Fix py3 compat issue in get_toolchain_if_necessary.
Bug: 1207012
Change-Id: I6ee0cb6be8e86490ec0ecf5a4f2e236546bb92ee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2900993
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
4 years ago
Robert Liao f5c054ce53 Pretty-Print the win_toolchain JSON Output
This makes the file easier to read.

Change-Id: Ie5eac66582bd6f3ce3c31def6f591e001de9a79a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2893667
Auto-Submit: Robert Liao <robliao@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
4 years ago
Dirk Pranke 3b9d2d8557 Fix a Python3-compatibility issue in get_toolchain_if_necessary.
Bug: 1207012
Change-Id: I9205089c3a7d9f2d3e2eb9aac7ef6da9ae9b0876
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2890330
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
4 years ago
Nico Weber 3f562c04e0 win_toolchain: Write json files only if different from what is on disk
Bug: chromium:1188672
Change-Id: I10aed281afd636acc0c680a7da486ae74aa637df
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2807968
Auto-Submit: Nico Weber <thakis@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
4 years ago
Nico Weber 399c5918bf win: minor behavior-preserving tweaks to get_toolchain_if_necessary.py
- add run line (also to package_from_installed.py while here)
- use startswith() to check if a string starts with another
- switch from optparse to argparse
  - ...and make desired-hash a required positional argument
- drop win8 sdk support (not behavior preserving, but also unused
  for years)

Bug: none
Change-Id: I73056184208a48b9d9610f330c56f3a324763195
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2679295
Auto-Submit: Nico Weber <thakis@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
4 years ago
Nico Weber 3fe941c45e win toolchain packager: Put win sdk in "Windows Kits/10/" instead of "win_sdk/"
This is so that the packaged toolchain can be used with clang-cl's
new /winsysroot flag added in https://reviews.llvm.org/rG82847436e.
No impact yet on the chrome build yet -- the tooling should
transparently get the new Windows Sdk path via the generated SetEnv
json file and use it with -imsvc. (I tested this locally by tweaking
my installed hermetic win package to look like the one generated
by this CL.)

Once this is deployed, we can switch the chrome build to use
/winsysroot if we want to -- it'd make compile command lines
a bit shorter and easier to work with.

Depends on https://chromium-review.googlesource.com/c/chromium/src/+/2665866

Bug: 1173176
Change-Id: I04c435f2323f26e3c26ed82656929809a7e0b5e0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2655836
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Auto-Submit: Nico Weber <thakis@chromium.org>
4 years ago
Nico Weber 474cd16266 win toolchain: The SDK path should be checked per toolchain dir
This is a follow-up to crev.com/c/2668127. That patch incorrectly
checked the windows sdk path style for the target toolchain, but
there might be other, older toolchains with the other path style.
We need to check this per toolchain directory.

Also make sure ignored_dirs is compared case-insensitively.

Bug: 1173176
Change-Id: I005eb1b3200b11597978936a970f50f101708bea
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2669048
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Auto-Submit: Nico Weber <thakis@chromium.org>
4 years ago
Nico Weber 499a2fd61b Reland "win toolchain: Prepare downloader for windows sdk dir switch"
This is a reland of e72789f5b4

Original change's description:
> win toolchain: Prepare downloader for windows sdk dir switch
>
> crrev.com/c/2655836 tries to move the Windows SDK from
> "win_sdk" to "Windows Kits/10".
>
> get_toolchain_if_necessary.py (in depot_tools) saves the path to the SDK to
> third_party/depot_tools/win_toolchain/data.json which then gets copied
> by a script in the chromium repo to build/win_toolchain.json.
> For the SDK move to work, chromium's pinned depot_tools
> must write the new SDK path when rolling in the new toolchain package.
> This change makes depot_tools handle win packages that have the
> windows sdk either win "win_sdk" or in "Windows Kits\10".
>
> The plan is:
>
> 1. Land this change, which can handle both path styles
> 2. Wait for depot_tools in chromium to update
> 3. Then roll to a win toolchain package with the new layout
>
> In a few years, when we no longer need the old layout,
> we can remove this detection code again and assume the new layout.
>
> Bug: 1173176
> Change-Id: Iaefc5c16685d3dbfff87a3e50a7b20b457366e44
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2666429
> Commit-Queue: Nico Weber <thakis@chromium.org>
> Auto-Submit: Nico Weber <thakis@chromium.org>
> Reviewed-by: Bruce Dawson <brucedawson@chromium.org>

Bug: 1173176,1173393
Change-Id: Ic706f694f8f0260208fa637864e62d7cc4f7ce93
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2668127
Commit-Queue: Nico Weber <thakis@chromium.org>
Auto-Submit: Nico Weber <thakis@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
4 years ago
Takuto Ikuta 98cc028bbe Revert "win toolchain: Prepare downloader for windows sdk dir switch"
This reverts commit e72789f5b4.

Reason for revert: 
This broke goma builder.

Original change's description:
> win toolchain: Prepare downloader for windows sdk dir switch
>
> crrev.com/c/2655836 tries to move the Windows SDK from
> "win_sdk" to "Windows Kits/10".
>
> get_toolchain_if_necessary.py (in depot_tools) saves the path to the SDK to
> third_party/depot_tools/win_toolchain/data.json which then gets copied
> by a script in the chromium repo to build/win_toolchain.json.
> For the SDK move to work, chromium's pinned depot_tools
> must write the new SDK path when rolling in the new toolchain package.
> This change makes depot_tools handle win packages that have the
> windows sdk either win "win_sdk" or in "Windows Kits\10".
>
> The plan is:
>
> 1. Land this change, which can handle both path styles
> 2. Wait for depot_tools in chromium to update
> 3. Then roll to a win toolchain package with the new layout
>
> In a few years, when we no longer need the old layout,
> we can remove this detection code again and assume the new layout.
>
> Bug: 1173176
> Change-Id: Iaefc5c16685d3dbfff87a3e50a7b20b457366e44
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2666429
> Commit-Queue: Nico Weber <thakis@chromium.org>
> Auto-Submit: Nico Weber <thakis@chromium.org>
> Reviewed-by: Bruce Dawson <brucedawson@chromium.org>

TBR=thakis@chromium.org,brucedawson@chromium.org,infra-scoped@luci-project-accounts.iam.gserviceaccount.com

Change-Id: I8d133f4fa199f81978f5245bdb2155c62bc9cc88
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 1173176
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2666651
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
4 years ago
Nico Weber e72789f5b4 win toolchain: Prepare downloader for windows sdk dir switch
crrev.com/c/2655836 tries to move the Windows SDK from
"win_sdk" to "Windows Kits/10".

get_toolchain_if_necessary.py (in depot_tools) saves the path to the SDK to
third_party/depot_tools/win_toolchain/data.json which then gets copied
by a script in the chromium repo to build/win_toolchain.json.
For the SDK move to work, chromium's pinned depot_tools
must write the new SDK path when rolling in the new toolchain package.
This change makes depot_tools handle win packages that have the
windows sdk either win "win_sdk" or in "Windows Kits\10".

The plan is:

1. Land this change, which can handle both path styles
2. Wait for depot_tools in chromium to update
3. Then roll to a win toolchain package with the new layout

In a few years, when we no longer need the old layout,
we can remove this detection code again and assume the new layout.

Bug: 1173176
Change-Id: Iaefc5c16685d3dbfff87a3e50a7b20b457366e44
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2666429
Commit-Queue: Nico Weber <thakis@chromium.org>
Auto-Submit: Nico Weber <thakis@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
4 years ago
Nico Weber cb07c525c0 win toolchain: Remove some old MSVS 2013 references.
We used to use this many years ago.

Bug: none
Change-Id: I1df268c793bbfe7423ecf86ea99ebbe918e280d0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2658676
Auto-Submit: Nico Weber <thakis@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
4 years ago
Josip Sokcevic 9c0dc3026e Replace non-inclusive terms
Bug: 1118410, 1118413, 1118422, 1118424, 1118426
Bug: 1118428, 1118430, 1118434, 1118438, 1118442
Bug: 1118453, 1118454
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: build
Change-Id: I43e6305f8c3136774f01b57f12bb442bcb863371
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2551388
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
4 years ago
Bruce Dawson 21de33a93c Print packaging file size/count stats always
When doing a dry run of packaging the toolchain we have always printed
stats on the size and number of files. It is odd and annoying that we
don't print those stats when doing the actual packaging, so this changes
that.

Change-Id: Ic60c3f4960dacaea39c048dc0c5f4c50b551f68a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2380231
Reviewed-by: Mirko Bonadei <mbonadei@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Bruce Dawson 7872beb489 Shorten hash to avoid MAX_PATH limits
The Windows SDK contains some very long paths. When those are added to
the 40-character hash directory and the other path components it is easy
to hit the Windows 260-character MAX_PATH limit. In addition, the 40
character hash makes paths unwieldy in VsChromium search results and
elsewhere.

A ten character hash should be more than enough to avoid collisions - if
we upload a million toolchain packages then there will be about a 50%
chance of a name collision - we won't upload more than a thousand.

This was tested by copying the current toolchain file on Google storage
to a truncated name and then changing to that hash in vs_toolchain.py.
It was also necessary to copy the updates from depot_tools to
third_party\depot_tools. This means that we can't actually start using
short hashes until depot_tools has rolled.

We probably won't use a shorter hash until we next roll the toolchain,
just because changing the toolchain hash is mildly disruptive.

Bug: 1120785
Change-Id: I878b058857cbe9cb72a72b535864404eede33f3f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2376030
Reviewed-by: Sébastien Marchand <sebmarchand@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Bruce Dawson e95b5d6ad5 Avoid ..\.. in packaged environment variables
win_sdk\bin\setenv.cmd is used to configure the build environment for
the packaged toolchain. This has always been done by creating paths such
as %~dp0..\..\VC, which leads to extra long paths that are less readable
than direct paths. These extra long paths make us more vulnerable to
MAX_PATH issues.

This change generates the paths relative to the grandparent directory.
It does that by changing the current directory to the grandparent
directory of the script and then generating paths relative to %cd%.
The paths are still absolute they are just shorter and simpler.

The paths are also stored in SetEnv.*.json and this change alters the
relative-root used for these paths. crrev.com/c/2370604 detects the
needed relative-root to handle toolchains packaged before and after this
change.

This change was tested using crrev.com/c/2370604 (temporarily modified
to use a toolchain created by this change) to ensure that it works.

Bug: 1120785
Change-Id: Icd72e36766d7ec92a491d16dbfd9ad7fc1b2ebc5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2372727
Reviewed-by: Scott Graham <scottmg@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Bruce Dawson f0e672e732 Filter out Assessment and Deployment Kit
When packaging the VC++ and Windows SDK toolchain it is important to
filter out unneeded directories both to avoid unnecessary size and to
avoid including excessively long paths. The Assessment and Deployment
Kit was installed on my test machine, is 3.5 GB, and has some very
long file paths, so now I filter it out.

Bug: 1120785
Change-Id: I9e0c16cea57cf4a687db1e4c315440fd5a50c9f7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2370864
Reviewed-by: Xiaohan Wang <xhwang@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Bruce Dawson 355e6449e7 Python 3 support for get_toolchain_if_necessary.py
get_toolchain_if_necessary.py would fail on Python 3 when passing a
string to digest.update - this string needs to be encoded to bytes.
This even stops packaging of new toolchains from working.

With this change digest.update get_toolchain_if_necessary.py works with
Python 2 and Python 3.

As a test the current toolchain directory was repackaged - the same hash
was generated.

Change-Id: Ia682d15be42521a35f4df2e14d1f715d9e42d96f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2234586
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Bruce Dawson e65444f91d Fix Python3 bugs in package_from_installed.py
The package_from_installed.py script didn't run under Python 3. This
change fixes that and this script now _requires_ Python 3. Requiring
Python 3 was simpler than supporting both, and there is no need to
maintain Python 2.7 support. This script is only ever invoked manually
and all machines it is run on should have Python 3.

Changes included using universal_newlines=True so that
subprocess.check_output returns a string instead of bytes, and
by explicitly converting dict.items() to a list.

Similarly, open(name, 'wb') was changed to open(name,'wt', newline='')
to avoid byte/str problems while still avoiding CR/LF translations.

This also changes the default Windows 10 SDK version to something more
recent - the current version that we build with.

To test:

  python3 package_from_installed.py -d 2019

The use of '-d' makes the script run in test mode which is much faster.

Change-Id: I2e3fc4d464f2914ba5ae36f0a78570ea7e5d4f03
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2224506
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Dale Curtis 0e7788d16b Skip sources downloaded by WinDbg when checking toolchain status.
This avoids the toolchain being wiped after using WinDbg.

Bug: None
Change-Id: I989ef7744b46254bab4c5a707724dc38a9b9a548
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2159998
Commit-Queue: Dale Curtis <dalecurtis@chromium.org>
Commit-Queue: Scott Graham <scottmg@chromium.org>
Auto-Submit: Dale Curtis <dalecurtis@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
5 years ago
Yuly Novikov e93101895e Also set LIBPATH in Windows toolchain environment
Needed to find *.winmd files for UWP build.

This CL and crrev.com/c/2007881 were used to generate the recent
toolchains with UWP support:
b92fff97f2e0323e99803f37f4b77b843bdbf69d in crrev.com/c/2015984 and
9ff60e43ba91947baca460d0ca3b1b980c3a2c23 in crrev.com/c/2025528.

Bug: chromium:1032635
Change-Id: I1ab0b6e11a30600deb377078815c268d8f9027e4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2013579
Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Yuly Novikov 852d4aa1ba Don't drop windows.winmd when packaging Windows toolchain
It is needed for UWP build and is the same on at least 2 machines in
C:\Program Files (x86)\Windows Kits\10\UnionMetadata\10.0.18362.0\Windows.winmd

Bug: chromium:1032635
Change-Id: I9d7148a8dc11ea85f1b930f63ae19eaa58ac2bf6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2007881
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
5 years ago
Marc-Antoine Ruel 8e57b4bc55 python3 improvements
Ran:
  vi $(git grep --name-only iteritems | grep -v third_party)
  vi $(git grep --name-only itervalues | grep -v third_party)
  vi $(git grep --name-only 'print ' | grep -v third_party)

and edited the files quickly with adhoc macros. Then ran in recipes/:
  ./recipes.py test train

There was only a small subset of files that had been updated to use
six.iteritems() and six.itervalues(). Since the dataset size that is
being used in gclient is small (pretty much always below 200 items),
it's better to just switch to .items() right away and take the temporary
performance hit, so that we don't need to come back to rewrite the code.

Recipe-Nontrivial-Roll: build
Bug: 984182
Change-Id: I5faf11486b66b0d73c9098ab0f2ce1b15a45c53e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1854900
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Marc-Antoine Ruel <maruel@chromium.org>
5 years ago
Mirko Bonadei cbff54b4ac Fail to repackage if win_sdk\Debuggers doesn't exist.
This is easy to miss and a toolchain package without win_sdk\Debuggers
will not work as expected so this CL adds a check to stop the
packaging/repacking process as soon as possible.

Bug: 1006238
Change-Id: I272fbaf3e1cbde48a3e260e38e34ffc10d6c86df
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1829272
Commit-Queue: Mirko Bonadei <mbonadei@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
6 years ago
Mirko Bonadei e4174f483e Add VS 2019 support to package_from_installed.py.
This CL also removes VS 2015 support.

Bug: 1006238
Change-Id: Ib05b3d211341fcd5805c3acca3bc4c0aa894831d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1823961
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Mirko Bonadei <mbonadei@chromium.org>
6 years ago
Daniel Bratell f411acf552 Make it possible to build a Windows SDK package without arm libs
Not everyone builds Chromium for arm so then it's practical to be
able to use a smaller SDK package without arm support included.

This also removes some samples that should not be needed.

Recipe-Nontrivial-Roll: chromiumos
Change-Id: I10f64d8907600fe79d3051083d100a7562d3a797
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1752202
Commit-Queue: Daniel Bratell <bratell@opera.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
6 years ago
Raul Tambre 80ee78e7fa Convert print statements to Python 3 style
Ran "2to3 -w -n -f print ./" and manually added imports.
Ran "^\s*print " and "\s+print " to find batch/shell scripts, comments and the like with embedded code, and updated them manually.
Also manually added imports to files, which used print as a function, but were missing the import.

The scripts still work with Python 2.
There are no intended behaviour changes.

Bug: 942522
Change-Id: Id777e4d4df4adcdfdab1b18bde89f235ef491b9f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1595684
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Auto-Submit: Raul Tambre <raul@tambre.ee>
6 years ago
Tom Anderson e893454f79 Fix a typo
BUG=None
R=dpranke

Change-Id: Ia16a3a30bc8339d81847725ed9a23b2694e4169e
Reviewed-on: https://chromium-review.googlesource.com/c/1464772
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Auto-Submit: Thomas Anderson <thomasanderson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
6 years ago
Lei Zhang b9ae2ca9a5 Fix get_toolchain_if_necessary.py --nodownload message formatting.
And indent properly as well.

Change-Id: I78e1f67d7820120bf809f1ac3ab64ce48c74e804
Reviewed-on: https://chromium-review.googlesource.com/c/1458784
Commit-Queue: Lei Zhang <thestig@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
6 years ago
Bruce Dawson b19e8dff15 Add a --nodownload option to get_toolchain_if_necessary.py
The Windows toolchain should be downloaded during gclient runhooks,
however because get_toolchain_if_necessary.py can either configure an
existing toolchain or download a toolchain there have been problems when
the toolchain is downloaded at an unexpected point. In particular, a
toolchain download during gn gen can lead to parallel downloads and
configuration failures.

The --nodownload option will be passed (after a subsequent change) by
build\vs_toolchain.py. A --nodownload option is needed instead of a more
positive --allowdownload option because of backwards compatibility. This
change will have no effect until the build\vs_toolchain.py change is
landed.

Bug: 584393, 662325
Change-Id: I9a0950e066744c0310e49e45d001a9113f1831cf
Reviewed-on: https://chromium-review.googlesource.com/c/1439805
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
6 years ago
Bruce Dawson eba46fc12e Improve error message when a toolchain can't be found
When non-Googlers try to build Chromium for the first time they get an
error saying:

  Please follow the instructions at https://chromium.googlesource.com/chromium/src/+/master/docs/windows_build_instructions.md

However those instructions are quite long and it's not obvious to some
users which part of the instructions they have forgotten. The most
likely is that they didn't set DEPOT_TOOLS_WIN_TOOLCHAIN=0, so this
changes the error message to suggest that directly.

Auto-detecting the correct state for DEPOT_TOOLS_WIN_TOOLCHAIN was
tried in crrev.com/c/1374833 but deemed impractical.

Bug: 907300, angleproject:2712
Change-Id: I45a9f59babe90bbe2137d49c555884d7bbbcf170
Reviewed-on: https://chromium-review.googlesource.com/c/1382942
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
6 years ago
Bruce Dawson fed2cb39c3 Fix ucrt source path when packaging the Windows toolchain
Recent versions of the 10.0.17763 SDK have rearranged the redist
directory. This change adjusts the packaging so that we can package the
SDK with and without this change. The script looks first in the
versioned directory and if that doesn't exist then it falls back to the
unversioned directory.

See also http://crrev.com/c/1371017 for the build\vs_toolchain.py
version of this change.

Change-Id: I835bf45f613a0aebb56eb1e8e63e6ffa5252edf0
Reviewed-on: https://chromium-review.googlesource.com/c/1370609
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
6 years ago
Bruce Dawson c6ffd7af7d Include ARM64 bits into win_toolchain/package_from_installed.py
These changes ensure that the packaged Windows SDK bits support targeting
ARM64 Windows. This change must not land until crrev.com/c/1339101 lands,
since it is needed to the varying number of vs_runtime_dll_dirs.

Bug: 893460
Change-Id: Ie32563067c6fb6078acfaccd6d3d572d1dd44888
Reviewed-on: https://chromium-review.googlesource.com/c/1330185
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
6 years ago
Henrique Ferreiro c5a26a769e [win-cross] Support using a zip file for the Windows SDK
Renames DEPOT_TOOLS_WIN_TOOLCHAIN_HTTP_BASE_URL to
DEPOT_TOOLS_WIN_TOOLCHAIN_BASE_LOCATION. This environment variable can
now point to a local directory where the Windows SDK zip file is stored.
This allows non-Googlers to cross-compile Chromium for Windows.

Bug: 852347
Change-Id: I00650e84247f35b4b8cfba204e0f2afd0882b69b
Reviewed-on: https://chromium-review.googlesource.com/1098256
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
7 years ago
Bruce Dawson 3fb5aa7bc5 Remove dbghelp.dll hack from packaging script
The script that packages the Visual Studio toolchain had special code to
handle grabbing dbghelp.dll from the VS install and copying it to the
Debuggers directories. This was necessary around VS 2017 RTM to handle
/debug:fastlink but it now causes failures due to mismatched DLLs
leading to a missing ordinal. The hack shouldn't be needed anymore
because we no longer depend on /debug:fastlink (lld!) and because the
Debuggers package is assumed to have been updated to handle VS 2017 by
now.

Bug: 846313
Change-Id: I2b5fd87aaa785ce24a0905d70e7e586ff4838b1c
Reviewed-on: https://chromium-review.googlesource.com/1086990
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
7 years ago
Kimmo Kinnunen d3f2c8e783 win_toolchain: Add another source dir for dbghelp.dll when packaging the toolchain
It appears that newer VS 2017 installers install CppUnitFramework in different
way. My version VS 15.6.6. The problem is that normal users do not seem to be
able to download the VS 15.3.2 that Chromium suggests and this script was
probably originally tested with.

I installed:
vs_professional__428566190.xxx.exe
  --add Microsoft.VisualStudio.Workload.NativeDesktop
  --add Microsoft.VisualStudio.Component.Debugger.JustInTime
  --add Microsoft.VisualStudio.Component.Graphics.Tools
  --add Microsoft.VisualStudio.Component.Graphics.Win81
  --add Microsoft.VisualStudio.Component.NuGet
  --add Microsoft.VisualStudio.Component.Static.Analysis.Tools
  --add Microsoft.VisualStudio.Component.VC.ATL
  --add Microsoft.VisualStudio.Component.VC.DiagnosticTools
  --add Microsoft.VisualStudio.Component.VC.TestAdapterForGoogleTest
  --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64
  --add Microsoft.VisualStudio.Component.Windows10SDK.15063.Desktop
  --add Microsoft.VisualStudio.Component.Windows10SDK.15063.UWP
  --add Microsoft.VisualStudio.Component.Windows10SDK.15063.UWP.Native

 Go to apps -> Windows sdk 15063
  Modify  -> Change
   Select Debugging tools for Windows

Ended up with VS 15.6.6 and Windows SDK 10.0.15063.673

Packaged with:
  python  %HOME%\depot_tools\win_toolchain\package_from_installed.py -w 10.0.15063.0 2017

Bug: 834656
Change-Id: I5e6bf6a5f94a28d5ba81a3c8cd79e37028b9429b
Reviewed-on: https://chromium-review.googlesource.com/1018470
Reviewed-by: Scott Graham <scottmg@chromium.org>
Commit-Queue: Scott Graham <scottmg@chromium.org>
7 years ago