superbeeeeeee/gcs-rsync repository overview

⁠gcs-rsync

repository: https://github.com/cboudereau/gcs-rsync⁠

gcs-rsync (Google Cloud Storage rsync) is faster than gsutil rsync (see benchmarks⁠).

no hard limit to 32K objects or specific conf to compute state.

⁠Side notes

• The image uses an env var GOOGLE_APPLICATION_CREDENTIALS bound to /creds.json that is why a read-only volume is necessary.
• gcs-rsync supports mirror and sync feature. /!\ Mirror deletes all extra dest files.
• For more info, use the --help flag.

⁠Sync local folder to gcs

docker run --rm -it -v ${GOOGLE_APPLICATION_CREDENTIALS}:/creds.json:ro -v <YourFolderToUpload>:/source:ro superbeeeeeee/gcs-rsync /source gs://<YourBucket>/<YourPrefix>

⁠Sync gcs to local folder

docker run --rm -it -v ${GOOGLE_APPLICATION_CREDENTIALS}:/creds.json:ro -v <YourFolderToDownloadTo>:/dest superbeeeeeee/gcs-rsync gs://<YourBucket>/<YourPrefix> /dest

⁠Help

docker run --rm -it superbeeeeeee/gcs-rsync --help