Huge thanks to piman@ for working on a patch. I chose a different design but he
gave me motivation and ideas. Sorry for not accepting his patch earlier, this
was mostly due to broken gclient implementation itself.
gclient can now run an unlimited number of parallel checkouts and always keep
the checkout coherency correct.
--jobs=1 is single threaded as before, albeit with a different code path.
Issues:
- Using --jobs with a value other than 1 will result in a mangled output.
- Exceptions thrown in a thread will be have the wrong stack trace.
TEST=gclient sync -j 99 in a ssh:// chromiumos checkout is dramatically faster.
---
Here's the perf on linux on i7-860 for a chromium checkout with warm cache. Cold
cache will result is significantly reduced improvements so this is best case
improvements. The sync was no-op all the time except where noted. All execution
where with "time gclient sync " + args. Didn't include 'sys' column since it was
statistically insignifiant and highly correlated with 'user'.
runs with -f runs with -m without -f nor -m
args real user real user real user
-j 12 20.59s 18.00s 5.64s 7.95s 5.86s 8.10s
#1 1m05.26s 20.02s 5.20s 7.94s 5.10s 8.09s
22.79s 18.17s
-j 1 #2 1m47.00s 16.72s 9.69s 5.72s 12.35s 5.96s
1m31.28s 17.06s 9.54s 5.85s 10.51s 6.20s
1m31.79s 16.39s
before #3 1m30.94s 16.74s 9.77s 5.83s 10.45s 5.77s
1m30.17s 17.30s 10.36s 5.68s 10.16s 5.88s
hook #4 8.52s 7.93s
8.73s 8.13s
#1 This particular run synched to r56023, a webkit roll updating layout tests.
It's still faster than a no-op sync without parallel checkout.
#2 Maybe there was a sync or computer hickup, I didn't realize.
#3 This is depot_tools@56020
#4 Since -f implies runhooks, I ran the hook 'python src/build/gyp_chromium'
manually to compare. Hooks are still run in a single thread. I didn't rest
'gclient runhooks'.
I tried to go a ssh:// checkout of chromium os tree but it timed out everytime I
tried to sync so I couldn't get data points. I expect an order of magnitude of
improvement or more.
Review URL: http://codereview.chromium.org/3135014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@56079 0039d316-1c4b-4281-b951-d872f2087c98
I've never known why there were added in the first place
Review URL: http://codereview.chromium.org/3155008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55896 0039d316-1c4b-4281-b951-d872f2087c98
Instead of keeping the reference information in the parent, note for each Dependency if it should be processed with self.should_process. I had to hack a bit with the hooks to also enforce recursion_limit() to keep the old behavior, otherwise From() could cause hooks to run that weren't run before.
BUG=50015
TEST=hooks are run properly. Tested with webkit and pagespeed and buildbot master, fixes running the same checkout multiple times.
Review URL: http://codereview.chromium.org/3124017
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55895 0039d316-1c4b-4281-b951-d872f2087c98
ExecutionQueue takes care of reordering execution depending on each Dependency
requirements. There is no need to sort anymore.
ExecutionQueue also manages the Progress instance and is multithread safe, in preparation to parallel checkout.
Review URL: http://codereview.chromium.org/3112002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55882 0039d316-1c4b-4281-b951-d872f2087c98
gclient revinfo alone will now not return the actual checked out revisions but simply what's in the .gclient&DEPS.
BUG=none
TEST=add a --snapshot test to make sure it is not broken
Review URL: http://codereview.chromium.org/3130007
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55760 0039d316-1c4b-4281-b951-d872f2087c98
It raises the time to run the presubmit to ~30 seconds but it's still worth.
TEST=run more often
Review URL: http://codereview.chromium.org/3146010
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55759 0039d316-1c4b-4281-b951-d872f2087c98
Add Dependency.processed and .hooks_ran to ease debugging.
Accept when more than one reference exist for FromImpl(), it may happen as it is the case for pagespeed.
Not much behavior change.
Review URL: http://codereview.chromium.org/3126003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55575 0039d316-1c4b-4281-b951-d872f2087c98
It would modify self.parsed_url even when it's called for a From() keyword
Review URL: http://codereview.chromium.org/3171001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55574 0039d316-1c4b-4281-b951-d872f2087c98
Also rename an argument to better fit reality.
No functional change
Review URL: http://codereview.chromium.org/3108001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55559 0039d316-1c4b-4281-b951-d872f2087c98
BUG=36135
TEST=gclient sync on a svn lock'ed directory will work automatically
Review URL: http://codereview.chromium.org/3028053
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55438 0039d316-1c4b-4281-b951-d872f2087c98
That change allows for merging multiple revisions into a local working copy.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3041034
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@54399 0039d316-1c4b-4281-b951-d872f2087c98
BUG=none
TEST=Make change in local sandbox and confirm proper things happen on "git try help", "git try xyzzy", and "git try -b linux."
Review URL: http://codereview.chromium.org/3075009
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@54374 0039d316-1c4b-4281-b951-d872f2087c98
This was the behavior before the refactor.
TEST=sync parfait and hooks should be run when necessary.
TBR=bradnelson
Review URL: http://codereview.chromium.org/2819062
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53430 0039d316-1c4b-4281-b951-d872f2087c98
This issue is exhibited by pagespeed.
Also a solution is always a direct dependency by definition.
TEST=none
TBR=bradnelson
Review URL: http://codereview.chromium.org/3036013
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53397 0039d316-1c4b-4281-b951-d872f2087c98
No behavior change.
TEST=all test still pass
TBR=bradnelson
Review URL: http://codereview.chromium.org/3005026
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53392 0039d316-1c4b-4281-b951-d872f2087c98
PageSpeed is an example of such ordering dependency.
TBR=bradnelson
Review URL: http://codereview.chromium.org/2809072
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53377 0039d316-1c4b-4281-b951-d872f2087c98
Bring some OOP and sanity to gclient.py.
- Changed the algorithm from breadth-first to depth-first.
- Added infinite recursion support.
- Fixed From() of From() dependency.
- Fixed cross solution custom deps aliasing.
- Removed support code for old .gclient_entries format.
- Removed IsGitCheckout() in favor for a direct check for .git presence.
TEST=all test pass
The main difference with r51760 is that new smoke tests have been added since which replicates how webkit checkouts with gclient.
Review URL: http://codereview.chromium.org/2917009
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53205 0039d316-1c4b-4281-b951-d872f2087c98
To reduce server load, only fetch the ref we need.
This is a redo of: http://codereview.chromium.org/2856038
First upload was the original CL 2856038. Subsequent uploads
are my changes.
TEST=ran presubmit tests
Review URL: http://codereview.chromium.org/2842060
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53072 0039d316-1c4b-4281-b951-d872f2087c98
This looks like it broke syncing from git repos:
File "/home/derat/local/depot_tools/gclient.py", line 1208, in <module>
sys.exit(Main(sys.argv[1:]))
File "/home/derat/local/depot_tools/gclient.py", line 1198, in Main
return command(parser, argv[1:])
File "/home/derat/local/depot_tools/gclient.py", line 1035, in CMDsync
return client.RunOnDeps('update', args)
File "/home/derat/local/depot_tools/gclient.py", line 621, in RunOnDeps
scm.RunCommand(command, self._options, args, file_list)
File "/home/derat/local/depot_tools/gclient_scm.py", line 115, in RunCommand
return getattr(self, command)(options, args, file_list)
File "/home/derat/local/depot_tools/gclient_scm.py", line 264, in update
print_error=False)
File "/home/derat/local/depot_tools/scm.py", line 84, in Capture
return gclient_utils.CheckCall(c, in_directory, print_error)
File "/home/derat/local/depot_tools/gclient_utils.py", line 58, in CheckCall
raise CheckCallError(command, cwd, process.returncode, std_out, std_err)
gclient_utils.CheckCallError: (['git', 'fetch', 'origin', 'cc1c63d'],
'/usr/local/google/home/derat/chrome/src/third_party/cros', 128, '', "fatal:
Couldn't find remote ref cc1c63d\nfatal: The remote end hung up unexpectedly\n")
BUG=none
TEST=works again after the revert
Review URL: http://codereview.chromium.org/2888023
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53056 0039d316-1c4b-4281-b951-d872f2087c98
This is the same as the last cl, but hopefully with the right svn
server.
Review URL: http://codereview.chromium.org/3023008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53043 0039d316-1c4b-4281-b951-d872f2087c98
To reduce server load, only fetch the ref we need.
TEST=ran presubmit tests
Review URL: http://codereview.chromium.org/2856038
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52941 0039d316-1c4b-4281-b951-d872f2087c98
Rather than updating git-cl whenever gclient is invoked, update it
whenever git-cl is invoked.
In the chromium-os workflow, folks often run git-cl but rarely run gclient.
This results in folks running an older git-cl which may not
have the latest features/fixes.
TEST=
Tested that git-cl is updated when it is stale, git-cl is cloned when it doesn't exist and is not updated when DEPOT_TOOLS_UPDATE=0
Review URL: http://codereview.chromium.org/2962013
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52236 0039d316-1c4b-4281-b951-d872f2087c98
Chromium OS continuous builds are failing because of the following error.
Traceback (most recent call last):
File "/b/depot_tools/gclient.py", line 1212, in <module>
sys.exit(Main(sys.argv[1:]))
File "/b/depot_tools/gclient.py", line 1202, in Main
return command(parser, argv[1:])
File "/b/depot_tools/gclient.py", line 1122, in CMDrevinfo
client.PrintRevInfo()
File "/b/depot_tools/gclient.py", line 776, in PrintRevInfo
url = entries.pop(name)
KeyError: 'chromiumos'
This change will fix the error.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/2892007
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52180 0039d316-1c4b-4281-b951-d872f2087c98
Emulate running gclient from a checkout containing .gclient and a DEPS, with the
solution having an url set to None.
TEST=new smoke test
Review URL: http://codereview.chromium.org/2897010
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52119 0039d316-1c4b-4281-b951-d872f2087c98
This is being abused by WebKit.
TEST=new smoke test
Review URL: http://codereview.chromium.org/2923005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52096 0039d316-1c4b-4281-b951-d872f2087c98
This case covers the webkit gclient usage.
TEST=improved smoke test
Review URL: http://codereview.chromium.org/2968005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52092 0039d316-1c4b-4281-b951-d872f2087c98
maruel@chromium.org
scottz@google.com
tfarina@chromium.org
jochen@chromium.org
rdsmith@google.com
sky@chromium.org
bradnelson@google.com
msb@chromium.org
derat@chromium.org
mazda@chromium.org