While using --jobs, empty lines are adding a lot of noise.
An example output would look like:
>gclient sync --jobs=10
1>
3>
4>
6>
5>
7>
5>
5>________ running 'git clone ...'
Review URL: http://codereview.chromium.org/7101012
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@87542 0039d316-1c4b-4281-b951-d872f2087c98
When specifying a revision (with -r) propagates the date of the revision to its children.
Review URL: http://codereview.chromium.org/6873110
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@83313 0039d316-1c4b-4281-b951-d872f2087c98
The reason of this change (beside deleting 30 lines) is to fix pylint on Mac.
For an unknown reason, it was dying here trying to process the minidom import.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/6799021
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@80770 0039d316-1c4b-4281-b951-d872f2087c98
It should be fine now that I've fixed python2.5 compatilibity issues in subprocess2.py.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/6759074
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@80216 0039d316-1c4b-4281-b951-d872f2087c98
This change moves most of the code into presubmit_support and gclient_utils and just leaves the formatting differences for the messages between the two tools.
Review URL: http://codereview.chromium.org/6719004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@79002 0039d316-1c4b-4281-b951-d872f2087c98
pass pylint cleanly.
There should be no functional changes in this patch, although there's at least one minor bug that lint caught.
Review URL: http://codereview.chromium.org/6685023
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@78638 0039d316-1c4b-4281-b951-d872f2087c98
It would fail on win32 when these modules are not present.
BUG=75902
TEST=none
R=mhm@chromium.org
Review URL: http://codereview.chromium.org/6683028
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@77947 0039d316-1c4b-4281-b951-d872f2087c98
Keep an alias to RemoveDirectory, will be removed in a later change.
TEST=new unit test to test this function
BUG=none
Review URL: http://codereview.chromium.org/6628032
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@77463 0039d316-1c4b-4281-b951-d872f2087c98
When an HTTP exception information is logged, no information is sent by
default. Add an helper function to manually extract interesting information
to append it to the exception type.
Also disable another pylint warning.
TEST=manual test
BUG=none
Review URL: http://codereview.chromium.org/5303002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@67306 0039d316-1c4b-4281-b951-d872f2087c98
Python standard library is not thread safe. Work around it by leaking a bit.
The problem is particularly visible on python 2.4.
TEST=none
BUG=60623
Review URL: http://codereview.chromium.org/4465001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@65048 0039d316-1c4b-4281-b951-d872f2087c98
Most of them are style issues or variable aliasing.
TEST=Can almost enable pylint warnings
BUG=none
Review URL: http://codereview.chromium.org/4360002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@64908 0039d316-1c4b-4281-b951-d872f2087c98
This makes the code saner to understand.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3420014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@60050 0039d316-1c4b-4281-b951-d872f2087c98
In the next change, we can now remove all the options.stdout bookeeping since
it's not unnecessary.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3398008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59795 0039d316-1c4b-4281-b951-d872f2087c98
The end goal is to remove options.stdout, to remove a lot of bookkeeping.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3418014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59792 0039d316-1c4b-4281-b951-d872f2087c98
BUG=54084
TEST=updated smoke test and added a new one + stdout untangling
Review URL: http://codereview.chromium.org/3357020
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59002 0039d316-1c4b-4281-b951-d872f2087c98
Simplify ExecutionQueue.run() by moving code into utility functions.
Reduce the amount of code in WorkerThread.run() to improve reliability.
Don't trap exceptions in single-threaded usage!
BUG=54084
TEST=none
Review URL: http://codereview.chromium.org/3336015
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58974 0039d316-1c4b-4281-b951-d872f2087c98
Add back support for python 2.4.
Looks like a few slaves haven't been updated yet.
TBR=bradnelson
BUG=none
TEST=none
Review URL: http://codereview.chromium.org/3338017
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58820 0039d316-1c4b-4281-b951-d872f2087c98
An automated work around for python lack of stdout flushing.
This design simplifies life once parallel checkout is enabled.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3342024
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58819 0039d316-1c4b-4281-b951-d872f2087c98
Change CheckCallError to inherit from Error. This will simplify try/except changes.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3324007
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58695 0039d316-1c4b-4281-b951-d872f2087c98
Move logging call to the base function.
BUG=none
TEST=none
Review URL: http://codereview.chromium.org/3300008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58523 0039d316-1c4b-4281-b951-d872f2087c98
This way the function is much more similar to subprocess.call(). Further changes
will be done, to be able to convert all the function, to be able to parallelize
output without corrupting it.
Used pylint to verify call graph correctness, causing some other unrelated
changes.
TEST=unit tests
BUG=none
Review URL: http://codereview.chromium.org/3117039
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@57369 0039d316-1c4b-4281-b951-d872f2087c98
It will eventually replace 'print_messages' and be used to annotate printed
lines when using --jobs.
Review URL: http://codereview.chromium.org/3174020
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@57092 0039d316-1c4b-4281-b951-d872f2087c98
Original Description:
Remove code duplication and improve style.
Create a Popen function to reduce code duplication.
Use RemoveDirectory where relevant.
Make drover slightly more posix friendly.
Review URL: http://codereview.chromium.org/3199011
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@57083 0039d316-1c4b-4281-b951-d872f2087c98
Create a Popen function to reduce code duplication.
Use RemoveDirectory where relevant.
Make drover slightly more posix friendly.
Review URL: http://codereview.chromium.org/3126020
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@56893 0039d316-1c4b-4281-b951-d872f2087c98
Speed up flushing of output on platforms that may not
flush as often as we expect (like Windows). Wait at
least 10s between flushes to avoid overloading busy
readers.
BUG=none
TEST=gclient flushes on Windows more often, regular
gclient operations continue to work as expected
Review URL: http://codereview.chromium.org/3140013
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@56217 0039d316-1c4b-4281-b951-d872f2087c98
Needed to use the special 3 args raise notation.
Review URL: http://codereview.chromium.org/3164014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@56209 0039d316-1c4b-4281-b951-d872f2087c98
Huge thanks to piman@ for working on a patch. I chose a different design but he
gave me motivation and ideas. Sorry for not accepting his patch earlier, this
was mostly due to broken gclient implementation itself.
gclient can now run an unlimited number of parallel checkouts and always keep
the checkout coherency correct.
--jobs=1 is single threaded as before, albeit with a different code path.
Issues:
- Using --jobs with a value other than 1 will result in a mangled output.
- Exceptions thrown in a thread will be have the wrong stack trace.
TEST=gclient sync -j 99 in a ssh:// chromiumos checkout is dramatically faster.
---
Here's the perf on linux on i7-860 for a chromium checkout with warm cache. Cold
cache will result is significantly reduced improvements so this is best case
improvements. The sync was no-op all the time except where noted. All execution
where with "time gclient sync " + args. Didn't include 'sys' column since it was
statistically insignifiant and highly correlated with 'user'.
runs with -f runs with -m without -f nor -m
args real user real user real user
-j 12 20.59s 18.00s 5.64s 7.95s 5.86s 8.10s
#1 1m05.26s 20.02s 5.20s 7.94s 5.10s 8.09s
22.79s 18.17s
-j 1 #2 1m47.00s 16.72s 9.69s 5.72s 12.35s 5.96s
1m31.28s 17.06s 9.54s 5.85s 10.51s 6.20s
1m31.79s 16.39s
before #3 1m30.94s 16.74s 9.77s 5.83s 10.45s 5.77s
1m30.17s 17.30s 10.36s 5.68s 10.16s 5.88s
hook #4 8.52s 7.93s
8.73s 8.13s
#1 This particular run synched to r56023, a webkit roll updating layout tests.
It's still faster than a no-op sync without parallel checkout.
#2 Maybe there was a sync or computer hickup, I didn't realize.
#3 This is depot_tools@56020
#4 Since -f implies runhooks, I ran the hook 'python src/build/gyp_chromium'
manually to compare. Hooks are still run in a single thread. I didn't rest
'gclient runhooks'.
I tried to go a ssh:// checkout of chromium os tree but it timed out everytime I
tried to sync so I couldn't get data points. I expect an order of magnitude of
improvement or more.
Review URL: http://codereview.chromium.org/3135014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@56079 0039d316-1c4b-4281-b951-d872f2087c98
maruel@chromium.org
nsylvain@google.com
floitsch@google.com
dpranke@chromium.org
thakis@chromium.org
jochen@chromium.org
chase@chromium.org