Some SVN-only trybots get errors when running gclient revert with this
change. I assume it's because the flag is not present for the gclient
revert command, and that the code checking if options.with_branch_heads
is set is common to both revert and sync.
BUG=none
TEST=reverted locally and gclient revert worked again.
Review URL: https://chromiumcodereview.appspot.com/12616006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@186967 0039d316-1c4b-4281-b951-d872f2087c98
The "branch-heads" are refspecs created from upstream svn branches. This flag
is mostly needed for official builders, and adds about 1/2GB of data to the
checkout, so not currently enabling by default.
Review URL: https://codereview.chromium.org/12546017
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@186598 0039d316-1c4b-4281-b951-d872f2087c98
Support 'git grep <match> <branch>' type searches in gclient grep.
These searches had a slightly different output so needed more
tweeking to the regex. I cleaned it up a bit too.
R=maruel@chromium.org
BUG=
Review URL: https://chromiumcodereview.appspot.com/11316213
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@170039 0039d316-1c4b-4281-b951-d872f2087c98
- This will be used by navabi to have his gclient recurse
command print the current repo.
R=maruel@chromium.org
BUG=
Review URL: https://chromiumcodereview.appspot.com/11308115
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@169812 0039d316-1c4b-4281-b951-d872f2087c98
This allows specificying that a target OS list should override the base
list instead of adding to it. This is useful for iOS builds, where the
desired behavior is to pull only the iOS deps, not the union of the Mac
and iOS deps.
Review URL: https://chromiumcodereview.appspot.com/11363036
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@165671 0039d316-1c4b-4281-b951-d872f2087c98
This will let DEPS files specify a higher local bound for recursion.
Note there is an edge case where a dependency can be initially pulled
in with a lower recursion level and then gets ignored subsequently
when a lower level DEPS overrides it's inclusion level and then
includes the same dependency. I do not solve this case, as
I am intending to use this syntax for top level deps files.
BUG=155780
Review URL: https://chromiumcodereview.appspot.com/11146032
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@162464 0039d316-1c4b-4281-b951-d872f2087c98
- On high core, fast machines, jobs=8 is does not offer good
parallelization. Switch to use number of local cpus.
R=maruel@chromium.org
BUG=
Review URL: https://chromiumcodereview.appspot.com/11140019
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@162072 0039d316-1c4b-4281-b951-d872f2087c98
Let's say there's 2 solutions, solA and solB, both fetching the dependency depA
with different urls, solA has 'depA': 'http://foo/bar' and solB has 'depA':
From('depB', 'bar').
This case used to not work, because LateOverride() was not called in
verify_validity(), so it was not comparing the resolved urls but instead what is
textually specified in both DEPS files.
Also because LateOverride() was not called, verify_validity() was comparing the
original urls even if custom_deps is specified in the solution(s).
Finally, allow one to mismatch the other IFF one is None.
R=szager@chromium.org
BUG=
Review URL: https://chromiumcodereview.appspot.com/11088023
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@161228 0039d316-1c4b-4281-b951-d872f2087c98
This simplifies some of the git-submodule stuff. For example, deps2hooks.bat
can be changed from this:
@echo off
setlocal
set TMP_GCLIENT_FILE=.tmp.gclient
echo solutions=[{"name":".","url":None,"deps_file":".DEPS.git"}] > %TMP_GCLIENT_FILE%
FOR /F "delims=" %%a in ('call gclient hookinfo --gclientfile=%TMP_GCLIENT_FILE%') do set HOOKINFO=%%a
set HOOKINFO="call %HOOKINFO:;= & call%"
cd ..
cmd /c %HOOKINFO%
del /F src\%TMP_GCLIENT_FILE%
... to this:
@echo off
setlocal
call gclient runhooks --spec="solutions=[{'name':'src','url':None,'deps_file':'.DEPS.git'}]"
Review URL: https://chromiumcodereview.appspot.com/10835045
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@149223 0039d316-1c4b-4281-b951-d872f2087c98
Cygwin used to have a bug where the number of lock was very limited, causing it
to crash when --jobs was used with a value different than 1. This bug seems to
have been fixed a few months ago as I can't reproduce the anymore after
extensive usage.
R=cmp@chromium.org
BUG=
TEST=
Review URL: https://chromiumcodereview.appspot.com/10689194
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@146798 0039d316-1c4b-4281-b951-d872f2087c98
into stdout and creating the GitDiffFilterer for git repository
BUG=125894
TEST=
Review URL: http://codereview.chromium.org/10317002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@135324 0039d316-1c4b-4281-b951-d872f2087c98
This ability is useful for versions of Chromium (i.e. Android) which want to
cross-compile to another platform that has a fixed set of custom dependencies.
Review URL: http://codereview.chromium.org/10127004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@134280 0039d316-1c4b-4281-b951-d872f2087c98
Previous attempts inside the buildbot config missed some places.
This just does it directly in gclient.
Review carefully, this is a depot_tools change.
BUG=None
TEST=None
R=maruel@chromium.org,dschuff@chromium.org
Review URL: http://codereview.chromium.org/10116033
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@132997 0039d316-1c4b-4281-b951-d872f2087c98
This permits running git grep over the dependencies. With --ignore, gclient
recurse continues even if nothing is found in a dependency. Without this flag,
git grep returns 1 when nothing is found, causing gclient to stop processing the
remaining dependencies.
TBR=nsylvain@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/9600010
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@124914 0039d316-1c4b-4281-b951-d872f2087c98
Support --jobs in 'fetch' and 'recurse' commands.
BUG=115840
TEST=
Review URL: http://codereview.chromium.org/9560010
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@124657 0039d316-1c4b-4281-b951-d872f2087c98
The fetch command is simply an alias for:
gclient recurse -s git git fetch origin
Rework 'recurse' command to use a WorkQueue so that --jobs is supported.
BUG=115840
TEST=
Review URL: http://codereview.chromium.org/9463044
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@124334 0039d316-1c4b-4281-b951-d872f2087c98
An example use case is github over ssh at a pinned revision.
R=petermayo@chromium.org
BUG=115202
TEST=
Review URL: http://codereview.chromium.org/9443025
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@123325 0039d316-1c4b-4281-b951-d872f2087c98
This is required to avoid the need to clobber the bots when moving a directory
to deps/. Currently, the directory in question is likely to remain in the
working copy, despite having been removed, due to the presence of untracked
files. This causes the checkout from deps/ to fail.
With this change, when both --reset and --delete_unversioned_trees are
specified, the the directory in question will be removed from the working copy,
thereby allowing the copy in deps/ to be checked out correctly.
Note that untracked directories which are explicitly ignored (ie in .gitignore
or svn:ignore) will not be removed.
Note that this was previously landed in http://codereview.chromium.org/9348054
but reverted due to problems with symlinks in the chromeos build.
BUG=112887, chromium-os:20759
Review URL: http://codereview.chromium.org/9404014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@122300 0039d316-1c4b-4281-b951-d872f2087c98
This is required to avoid the need to clobber the bots when moving a directory
to deps/. Currently, the directory in question is likely to remain in the
working copy, despite having been removed, due to the presence of untracked
files. This causes the checkout from deps/ to fail.
With this change, when both --force and --delete_unversioned_trees are
specified, the the directory in question will be removed from the working copy,
thereby allowing the copy in deps/ to be checked out correctly.
Note that untracked directories which are explicitly ignored (ie in .gitignore
or svn:ignore) will not be removed.
BUG=112887,chromium-os:20759
Review URL: http://codereview.chromium.org/9348054
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@121986 0039d316-1c4b-4281-b951-d872f2087c98
R=maruel@chromium.org
TEST=Configure a checkout using the NewGitWorkflow with a safesync_url and everything works (though possibly with a really long git svn fetch time).
BUG=106015
Review URL: http://codereview.chromium.org/8382030
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@115011 0039d316-1c4b-4281-b951-d872f2087c98
isatty() wasn't defined on Windows, causing it to crash.
R=dpranke@chromium.org
BUG=
TEST=manually gclient sync on windows
Review URL: http://codereview.chromium.org/8371006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@106865 0039d316-1c4b-4281-b951-d872f2087c98
For now only affect gclient help to figure out if there is any issue with it.
Because of the way it's done, there is no way to disable coloring, which could
be annoying when redirected. This needs to be taken in consideration.
R=dpranke@chromium.org
TEST=manually tested on linux, cygwin and windows, both run manually and
redirected to a text file.
BUG=
Review URL: http://codereview.chromium.org/8365001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@106708 0039d316-1c4b-4281-b951-d872f2087c98
It's not supported at the moment and it's misleading.
Support for it should be added eventually.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8357014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@106351 0039d316-1c4b-4281-b951-d872f2087c98
Otherwise it breaks naked --revision usage. One example issue is linux_chromeos
applying the revision to cros_deps instead of src.
TBR=dpranke@chromium.org
BUG=100335
TEST=
Review URL: http://codereview.chromium.org/8298002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@105522 0039d316-1c4b-4281-b951-d872f2087c98
It will help catch issues like the one we are getting on linux_chromeos where
the ordering of the solution is now by name, causing the default solution to be
used for --revision being the wrong one.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8276020
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@105387 0039d316-1c4b-4281-b951-d872f2087c98
It wasn't broken. The tree collapsed around the same time for unrelated reason.
TBR=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8227036
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@105090 0039d316-1c4b-4281-b951-d872f2087c98
I'm not sure if it's causing the tree blow up but I'm not taking any chance.
TBR=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8228017
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104956 0039d316-1c4b-4281-b951-d872f2087c98
This fixes GClientSmokeBoth.testMultiSolutionsJobs flakiness by having
consistent ordering.
Now no out of thread modification is ever done, which result in much saner code.
R=dpranke@chromium.org
BUG=60725
TEST=tested manually with gclient sync --jobs 100 on chrome and with
gclient_smotetest.py
Review URL: http://codereview.chromium.org/8174014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104920 0039d316-1c4b-4281-b951-d872f2087c98
This means parsing the DEPS file all the time preemptively in case it's needed
for a From() reference. This could break certain checkout, we'll see.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8143022
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104817 0039d316-1c4b-4281-b951-d872f2087c98
Also, add more locking to properties.
Now most mutations are contained. Will now figure out a way to remove the
ParseDepsFile() from an unrelated thread.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8142030
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104778 0039d316-1c4b-4281-b951-d872f2087c98
This makes coherency checks more consistent for DEPS vs .gclient.
It's still not thread safe but the mutation code paths are easier to follow.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8135019
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104772 0039d316-1c4b-4281-b951-d872f2087c98
... On my quest to make that stuff thread safe.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8135008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104033 0039d316-1c4b-4281-b951-d872f2087c98
While trying to fix all the multithreading issues...
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8135007
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104026 0039d316-1c4b-4281-b951-d872f2087c98
Helps figuring out what is mutable and what is not.
The goal is to shorten the class Dependency to reduce places where mutations can occur.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8122005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@103782 0039d316-1c4b-4281-b951-d872f2087c98
Merge _FindDependencies() back into __init__() now that __init__() is somewhat smaller.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8059011
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@103053 0039d316-1c4b-4281-b951-d872f2087c98
First, it reduces the class size, which is too large at the moment.
Second, it makes clear what is 'static' for the livetime of the instance life.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8038015
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@102766 0039d316-1c4b-4281-b951-d872f2087c98
The dependencies will continue to be managed. A new flag is present in the .gclient file to control this behavior.
Review URL: http://codereview.chromium.org/7918027
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@102002 0039d316-1c4b-4281-b951-d872f2087c98
Make Dependency.file_list a tuple so it can't be modified.
Move the guesswork of file_list at the right place.
Remove tree() since it's unnecessary.
R=dpranke@chromium.org
BUG=
TEST=all smoke tests still pass
Review URL: http://codereview.chromium.org/7922011
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@101951 0039d316-1c4b-4281-b951-d872f2087c98
I had forgot one access to _requirements. Now that tests are run, this change is
safer to commit.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/7909012
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@101856 0039d316-1c4b-4281-b951-d872f2087c98
I need to add real thread locking to some Dependency methods. That was long
needed anyway.
TBR=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/7891053
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@101100 0039d316-1c4b-4281-b951-d872f2087c98
I've fixed the From() issue that was causing it to hangs. We'll see if anything
pops up again.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/7885028
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@101084 0039d316-1c4b-4281-b951-d872f2087c98
This would result in a hang during gclient sync.
Includes regression test.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/7885008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@101070 0039d316-1c4b-4281-b951-d872f2087c98
It should be fine now that requirements with /@ works fine.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/7782034
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@100162 0039d316-1c4b-4281-b951-d872f2087c98
It breaks upstream webkit and nacl. I'll make then both use --jobs 1 before
relanding.
TBR=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/7833051
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@99761 0039d316-1c4b-4281-b951-d872f2087c98
Verify correctness with a unit test written explicitly to verify all corner cases.
BUG=60725
TEST=new unit test
Review URL: http://codereview.chromium.org/6598087
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@99234 0039d316-1c4b-4281-b951-d872f2087c98
'Cause linux2 is so last decade.
TBR=evan@chromium.org
BUG=85845
TEST=
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@88788 0039d316-1c4b-4281-b951-d872f2087c98
url is pointing to a git repo, the name of the directory created on disk will not
contain the .git suffix.
Review URL: http://codereview.chromium.org/7104009
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@87492 0039d316-1c4b-4281-b951-d872f2087c98
When specifying a revision (with -r) propagates the date of the revision to its children.
Review URL: http://codereview.chromium.org/6873110
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@83313 0039d316-1c4b-4281-b951-d872f2087c98
Make sure every site that catches gclient_utils.Error also catch
subprocess2.CalledProcessError.
BUG=
TEST=
Review URL: http://codereview.chromium.org/6691034
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@80339 0039d316-1c4b-4281-b951-d872f2087c98
Reapply r79006 with fixes for ubuntu 10.4.
The only change is in fix_default_encoding() to trap an exception
locale.getlocale() may throw.
R=dpranke@chromium.org
BUG=none
TEST=unit test
Review URL: http://codereview.chromium.org/6721029
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@79144 0039d316-1c4b-4281-b951-d872f2087c98
It is causing exception for some users on ubuntu 10.4 with:
"category LC_ALL is not supported"
TBR=dpranke
Review URL: http://codereview.chromium.org/6717025
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@79012 0039d316-1c4b-4281-b951-d872f2087c98
With this code in action, I can successfully print arabic and chinese on linux
and cygwin. It fails to print chinese on my Mac but prints arabic. It prints '?'
on Windows console but it *doesn't throw* which is the biggest improvement here.
It was particularly a problem on windows because WindowsError's description text
is in the current ANSI code page so it failed to print if the text was not pure
ASCII, like error message when using Windows' French UI.
R=dpranke@chromium.org
BUG=none
TEST=manual unit test ./tests/fix_encoding_test.py and it shouldn't throw
Review URL: http://codereview.chromium.org/6676090
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@79006 0039d316-1c4b-4281-b951-d872f2087c98
Don't block execution.
Python 2.4 has been untested for a while and may break unexpectedly.
TEST=manual testing
BUG=none
Review URL: http://codereview.chromium.org/5110009
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@66870 0039d316-1c4b-4281-b951-d872f2087c98
Make -D -f combination more effective by deleting directories even if there is
modified files.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/4803001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@65845 0039d316-1c4b-4281-b951-d872f2087c98
Most of them are style issues or variable aliasing.
TEST=Can almost enable pylint warnings
BUG=none
Review URL: http://codereview.chromium.org/4360002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@64908 0039d316-1c4b-4281-b951-d872f2087c98
Previously, solution's properties would be ignored if there was no DEPS file in
source control. Change the behavior to evaluate the properties anyway.
TEST=new smoke test
BUG=none
Review URL: http://codereview.chromium.org/4154004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@64232 0039d316-1c4b-4281-b951-d872f2087c98
Prints a meaningful message instead of throwing an exception.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3828010
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@62920 0039d316-1c4b-4281-b951-d872f2087c98
This occurs when gclient compares the cached deps in .gclient_entries
to the new DEPS and tries to construct an SCM for any cached entries
which no longer exist. In this situation gclient doesn't account for
the case where previous entries may have been ignored by setting their
url to None via the custom_deps section in .gclient.
Contributed by Jay Soffian.
Review URL: http://codereview.chromium.org/3769002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@62540 0039d316-1c4b-4281-b951-d872f2087c98
sys.stdout.isatty() is True even on buildbot.
Otherwise, logs can't be untangled when using --jobs.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3596006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@61552 0039d316-1c4b-4281-b951-d872f2087c98
This makes the code saner to understand.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3420014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@60050 0039d316-1c4b-4281-b951-d872f2087c98
In the next change, we can now remove all the options.stdout bookeeping since
it's not unnecessary.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3398008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59795 0039d316-1c4b-4281-b951-d872f2087c98
The end goal is to remove options.stdout, to remove a lot of bookkeeping.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3418014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59792 0039d316-1c4b-4281-b951-d872f2087c98
It hoses the chromium slaves on build.webkit.org.
It was fun while it lasted.
TBR=bradnelson
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3350018
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59146 0039d316-1c4b-4281-b951-d872f2087c98
It seems sys.stdout.isatty() is not a good signal on the build slaves.
TBR=bradnelson
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3342028
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59131 0039d316-1c4b-4281-b951-d872f2087c98
This way engineers can still enjoy the perf benefit while someone fixes the
buildbot stdio parsing issue.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3334017
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59127 0039d316-1c4b-4281-b951-d872f2087c98
This is because buildbot can't parse gclient output with parallel checkout,
causing got_revision to not be set, causing some scripts to fail.
TBR=bradnelson
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3325023
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59121 0039d316-1c4b-4281-b951-d872f2087c98
Simplify ExecutionQueue.run() by moving code into utility functions.
Reduce the amount of code in WorkerThread.run() to improve reliability.
Don't trap exceptions in single-threaded usage!
BUG=54084
TEST=none
Review URL: http://codereview.chromium.org/3336015
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58974 0039d316-1c4b-4281-b951-d872f2087c98
An automated work around for python lack of stdout flushing.
This design simplifies life once parallel checkout is enabled.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3342024
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58819 0039d316-1c4b-4281-b951-d872f2087c98
pylint is useful to catch errors before they go live so fix little annoyances.
BUG=none
TEST=none
Review URL: http://codereview.chromium.org/3372001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58817 0039d316-1c4b-4281-b951-d872f2087c98
This way the function is much more similar to subprocess.call(). Further changes
will be done, to be able to convert all the function, to be able to parallelize
output without corrupting it.
Used pylint to verify call graph correctness, causing some other unrelated
changes.
TEST=unit tests
BUG=none
Review URL: http://codereview.chromium.org/3117039
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@57369 0039d316-1c4b-4281-b951-d872f2087c98
It will eventually replace 'print_messages' and be used to annotate printed
lines when using --jobs.
Review URL: http://codereview.chromium.org/3174020
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@57092 0039d316-1c4b-4281-b951-d872f2087c98
Huge thanks to piman@ for working on a patch. I chose a different design but he
gave me motivation and ideas. Sorry for not accepting his patch earlier, this
was mostly due to broken gclient implementation itself.
gclient can now run an unlimited number of parallel checkouts and always keep
the checkout coherency correct.
--jobs=1 is single threaded as before, albeit with a different code path.
Issues:
- Using --jobs with a value other than 1 will result in a mangled output.
- Exceptions thrown in a thread will be have the wrong stack trace.
TEST=gclient sync -j 99 in a ssh:// chromiumos checkout is dramatically faster.
---
Here's the perf on linux on i7-860 for a chromium checkout with warm cache. Cold
cache will result is significantly reduced improvements so this is best case
improvements. The sync was no-op all the time except where noted. All execution
where with "time gclient sync " + args. Didn't include 'sys' column since it was
statistically insignifiant and highly correlated with 'user'.
runs with -f runs with -m without -f nor -m
args real user real user real user
-j 12 20.59s 18.00s 5.64s 7.95s 5.86s 8.10s
#1 1m05.26s 20.02s 5.20s 7.94s 5.10s 8.09s
22.79s 18.17s
-j 1 #2 1m47.00s 16.72s 9.69s 5.72s 12.35s 5.96s
1m31.28s 17.06s 9.54s 5.85s 10.51s 6.20s
1m31.79s 16.39s
before #3 1m30.94s 16.74s 9.77s 5.83s 10.45s 5.77s
1m30.17s 17.30s 10.36s 5.68s 10.16s 5.88s
hook #4 8.52s 7.93s
8.73s 8.13s
#1 This particular run synched to r56023, a webkit roll updating layout tests.
It's still faster than a no-op sync without parallel checkout.
#2 Maybe there was a sync or computer hickup, I didn't realize.
#3 This is depot_tools@56020
#4 Since -f implies runhooks, I ran the hook 'python src/build/gyp_chromium'
manually to compare. Hooks are still run in a single thread. I didn't rest
'gclient runhooks'.
I tried to go a ssh:// checkout of chromium os tree but it timed out everytime I
tried to sync so I couldn't get data points. I expect an order of magnitude of
improvement or more.
Review URL: http://codereview.chromium.org/3135014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@56079 0039d316-1c4b-4281-b951-d872f2087c98
I've never known why there were added in the first place
Review URL: http://codereview.chromium.org/3155008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55896 0039d316-1c4b-4281-b951-d872f2087c98
Instead of keeping the reference information in the parent, note for each Dependency if it should be processed with self.should_process. I had to hack a bit with the hooks to also enforce recursion_limit() to keep the old behavior, otherwise From() could cause hooks to run that weren't run before.
BUG=50015
TEST=hooks are run properly. Tested with webkit and pagespeed and buildbot master, fixes running the same checkout multiple times.
Review URL: http://codereview.chromium.org/3124017
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55895 0039d316-1c4b-4281-b951-d872f2087c98
ExecutionQueue takes care of reordering execution depending on each Dependency
requirements. There is no need to sort anymore.
ExecutionQueue also manages the Progress instance and is multithread safe, in preparation to parallel checkout.
Review URL: http://codereview.chromium.org/3112002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55882 0039d316-1c4b-4281-b951-d872f2087c98
gclient revinfo alone will now not return the actual checked out revisions but simply what's in the .gclient&DEPS.
BUG=none
TEST=add a --snapshot test to make sure it is not broken
Review URL: http://codereview.chromium.org/3130007
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55760 0039d316-1c4b-4281-b951-d872f2087c98
Add Dependency.processed and .hooks_ran to ease debugging.
Accept when more than one reference exist for FromImpl(), it may happen as it is the case for pagespeed.
Not much behavior change.
Review URL: http://codereview.chromium.org/3126003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55575 0039d316-1c4b-4281-b951-d872f2087c98
It would modify self.parsed_url even when it's called for a From() keyword
Review URL: http://codereview.chromium.org/3171001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55574 0039d316-1c4b-4281-b951-d872f2087c98
Also rename an argument to better fit reality.
No functional change
Review URL: http://codereview.chromium.org/3108001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55559 0039d316-1c4b-4281-b951-d872f2087c98
This was the behavior before the refactor.
TEST=sync parfait and hooks should be run when necessary.
TBR=bradnelson
Review URL: http://codereview.chromium.org/2819062
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53430 0039d316-1c4b-4281-b951-d872f2087c98
This issue is exhibited by pagespeed.
Also a solution is always a direct dependency by definition.
TEST=none
TBR=bradnelson
Review URL: http://codereview.chromium.org/3036013
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53397 0039d316-1c4b-4281-b951-d872f2087c98
No behavior change.
TEST=all test still pass
TBR=bradnelson
Review URL: http://codereview.chromium.org/3005026
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53392 0039d316-1c4b-4281-b951-d872f2087c98
PageSpeed is an example of such ordering dependency.
TBR=bradnelson
Review URL: http://codereview.chromium.org/2809072
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53377 0039d316-1c4b-4281-b951-d872f2087c98
Bring some OOP and sanity to gclient.py.
- Changed the algorithm from breadth-first to depth-first.
- Added infinite recursion support.
- Fixed From() of From() dependency.
- Fixed cross solution custom deps aliasing.
- Removed support code for old .gclient_entries format.
- Removed IsGitCheckout() in favor for a direct check for .git presence.
TEST=all test pass
The main difference with r51760 is that new smoke tests have been added since which replicates how webkit checkouts with gclient.
Review URL: http://codereview.chromium.org/2917009
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53205 0039d316-1c4b-4281-b951-d872f2087c98
Chromium OS continuous builds are failing because of the following error.
Traceback (most recent call last):
File "/b/depot_tools/gclient.py", line 1212, in <module>
sys.exit(Main(sys.argv[1:]))
File "/b/depot_tools/gclient.py", line 1202, in Main
return command(parser, argv[1:])
File "/b/depot_tools/gclient.py", line 1122, in CMDrevinfo
client.PrintRevInfo()
File "/b/depot_tools/gclient.py", line 776, in PrintRevInfo
url = entries.pop(name)
KeyError: 'chromiumos'
This change will fix the error.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/2892007
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52180 0039d316-1c4b-4281-b951-d872f2087c98
This is being abused by WebKit.
TEST=new smoke test
Review URL: http://codereview.chromium.org/2923005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52096 0039d316-1c4b-4281-b951-d872f2087c98
This case covers the webkit gclient usage.
TEST=improved smoke test
Review URL: http://codereview.chromium.org/2968005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52092 0039d316-1c4b-4281-b951-d872f2087c98
Add the fix for FileImpl() plus unit tests.
TEST=more tests
Review URL: http://codereview.chromium.org/2808048
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52005 0039d316-1c4b-4281-b951-d872f2087c98
I didn't run the unit test after doing the last minute change and it was indeed broken.
TBR=msb
Review URL: http://codereview.chromium.org/2823040
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@51761 0039d316-1c4b-4281-b951-d872f2087c98
kjellander@chromium.org
mmoss@chromium.org
bcwhite@chromium.org
ilevy@chromium.org
sivachandra@chromium.org
stuartmorgan@chromium.org
maruel@chromium.org
szager@chromium.org
dbeam@chromium.org
haitao.feng@intel.com
peter@chromium.org
bradnelson@google.com
szager@google.com
davidbarr@chromium.org
steveblock@chromium.org
thakis@chromium.org
cmp@chromium.org
phajdan.jr@chromium.org
bauerb@chromium.org
nsylvain@google.com
floitsch@google.com
thomasvl@chromium.org
joi@chromium.org
jochen@chromium.org
mazda@chromium.org