I fixed the overlapping argument and fixed up the tests so that the actual sha1 hash is now passed (since the tool actually check now that the downloaded file hash matches)
BUG=
R=hinoka@google.com
Review URL: https://codereview.chromium.org/1285423002 .
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@296346 0039d316-1c4b-4281-b951-d872f2087c98
Reason for revert:
This conflicts with (and duplicates?) the -z option below and has broken one of the chrome android downstream bots.
Safer to revert than trying to fix
Original issue's description:
> Add support for tar.gz archive files to download from download_from_google_storage
>
> Also, add support for directly setting the public read flag on uploaded files.
>
> The support for tar.gz allows the uploaded bundle to be a single tar.gz file instead of a lot of individual files using the directory support already in. The benefit here is that it is much easier to update the dependency. Simply clean out the existing files, copy in the new ones, create a tar.gz file, with the same name as the directory + 'tar.gz'. If the directory name and file name does not match up we will not clean up existing artifacts on download (i.e., there can be left over files after extracting).
>
> I am doing this because I am moving a bunch of the dart dependencies to gcs, and a lot of our dependencies is much easier to manage with this in. If you don't like this, I can simply wrap the download script in another python script and do the logic there, but this may be handy for other people as well.
>
> Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=295872
TBR=hinoka@google.com,hinoka@chromium.org,azarchs@chromium.org,ricow@google.com
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1209033006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@295883 0039d316-1c4b-4281-b951-d872f2087c98
Also, add support for directly setting the public read flag on uploaded files.
The support for tar.gz allows the uploaded bundle to be a single tar.gz file instead of a lot of individual files using the directory support already in. The benefit here is that it is much easier to update the dependency. Simply clean out the existing files, copy in the new ones, create a tar.gz file, with the same name as the directory + 'tar.gz'. If the directory name and file name does not match up we will not clean up existing artifacts on download (i.e., there can be left over files after extracting).
I am doing this because I am moving a bunch of the dart dependencies to gcs, and a lot of our dependencies is much easier to manage with this in. If you don't like this, I can simply wrap the download script in another python script and do the logic there, but this may be handy for other people as well.
Review URL: https://codereview.chromium.org/807463005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@295872 0039d316-1c4b-4281-b951-d872f2087c98
Handle KeyboardInterrupt gracefully rather the printing a
backtrace. Most users of these tools don't expect a
backtrace when then hit Ctrl-C.
Also, fix a few other inconsistencies found in the python
startup code of these different scripts:
- always call main function 'main' (rather than 'Main')
- always return 0 from main function
- if main takes args never include argv[0]
Review URL: https://codereview.chromium.org/955993006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@294250 0039d316-1c4b-4281-b951-d872f2087c98
Currently check_bucket_permissions() in download_from_google_storage.py
performs a gsutil ls gs://bucket to determine whether the user has access
to the bucket or not.
This can be an EXTREMELY expensive operation (~minutes) if the bucket in
question has a lot of objects in the root (real case: chrome-telemetry).
It is worth noting that check_bucket_permissions() is not called just for
uploads but also for downloads, hence this is slowing down all invocations
of gclient sync on users and bots machine.
Removing the check_bucket_permissions() and let gsutil fail with an
Unauthorized error message if ACLs are not met.
BUG=458059
Review URL: https://codereview.chromium.org/923473002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@294083 0039d316-1c4b-4281-b951-d872f2087c98
This pins gsutil to a vanilla 4.7 instead of the weird custom 3.4 we have in depot_tools
BUG=
R=pgervais@chromium.org
Review URL: https://codereview.chromium.org/797663003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@293413 0039d316-1c4b-4281-b951-d872f2087c98
r293250 broke it. I didn't realize that check_bucket_permissions
was shared code. It now only returns a single value.
TBR=vadimsh@chromium.org
Review URL: https://codereview.chromium.org/782643002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@293256 0039d316-1c4b-4281-b951-d872f2087c98
Code path:
1. plugins.sso_auth is imported, which adds the AuthHandler class to the global state.
2. HasConfiguredCredentials() in gslib/utils.py is called by gsutil, and will return true if "prodaccess" exists on the system, which tells the system that we don't want a no-op auth handler.
3. When a command is called, all the auth handlers are cycled through and sso_auth.SSOAuth is called, which calls a stubby command to emit a gaiamint'ed oauth2 access token, which is then used as the Authorization Header
if --bypass_prodaccess is passed in, then:
1. HasConfiguredCredentials() will bypass the check for prodaccess, as if it didn't exist.
2. plugins.sso_auth does not get imported.
Which will essentially cause gsutil to behave as if this patch never existed.
So the expected behavior is:
=.boto file does not exist, prodaccess exists, but unauthenticated=
Failure: No handler was ready to authenticate. 3 handlers were checked. ['OAuth2Auth', 'HmacAuthV1Handler', 'SSOAuth'] Check your credentials.
=.boto file exists, prodaccess exists, but unauthenticated=
sso_auth will raise NotReadyToAuthenticate, and the .boto file will be used instead
=.boto file exists, prodaccess exists, authenticated=
sso_auth will be run _after_ the default gsutil authenticator, which causes the sso_auth to be used over whatever the default authentication is.
bypass_prodaccess is passed in by default to upload_to_google_storage because we expect people who use upload_to_google_storage to not need prodaccess and have their own boto file already. Also the sso_auth plugin will only request a readonlyi token, which will not work for uploading.
BUG=258152
Review URL: https://codereview.chromium.org/86123002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@240266 0039d316-1c4b-4281-b951-d872f2087c98
Adds a command line flag --platform to control which platforms the given download occurs on.
Automatically marks downloads executable on non-Windows if the header x-goog-meta-executable is set. Automatically set this flag on upload when the executable bit is set.
BUG=233100
Review URL: https://codereview.chromium.org/42273002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@231387 0039d316-1c4b-4281-b951-d872f2087c98
continuation of: https://codereview.chromium.org/11664024
Moved it from chrome/trunk/src/build to depot_tools/
BUG=153360
TEST=two unittests included in tests/
For end-to-end testing, check out a large directory. Run
find . -name .svn -prune -o -size +1000k -type f -print0 | upload_to_google_storage.py -b chrome-artifacts -0 -
(replacing chrome-artifacts with an upload-able bucket)
to test upload
run "find . -name .svn -prune -o -size +1000k -type f -print0 | xargs -0 rm" to remove the files uploaded. Check that the large binary files have been removed
run "download_from_google_storage.py -r -d -b chrome-artifacts ." to download the files again.
Review URL: https://chromiumcodereview.appspot.com/12042069
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@187951 0039d316-1c4b-4281-b951-d872f2087c98
ricow@chromium.org
boliu@chromium.org
ricow@google.com
pasko@chromium.org
sbc@chromium.org
primiano@chromium.org
hinoka@chromium.org
ojan@chromium.org
hinoka@chromium.org
brettw@chromium.org
ukai@chromium.org
hinoka@google.com