zappi/s3-sync

By zappi

Updated 5 months ago

Synchronises a container's local directory with an S3 bucket.

Image
Databases & Storage

2.3K

S3 Sync

release-docker-imagetest-docker-image

This Docker container syncs a local directory to an AWS S3 bucket, allowing for easy backups and synchronization of data.

If the specified local directory is empty, it performs an initial sync from the specified S3 bucket. Then, it syncs that directory with the specified S3 bucket. If the local directory was not empty to begin with, it skips the initial sync.

This container is suitable for creating a recent backup copy of data in S3, which can then be easily retrieved when launching new nodes.

Note: This container is designed for syncing files to S3 from a single node and is not recommended as a permanent backup solution.

By default, the download location inside the container is set to /sync. However, this can be changed via the SYNCDIR environment variable.

Commands

The image supports the following commands:

  • download: (default) downloads the files and exits
  • upload: uploads the files and exits
  • sync: uses inotify to upload a directory to S3 when files change (see SYNCDIR)
  • periodic_upload: sets up a cron job to upload files to S3 periodically (see CRON_TIME and INITIAL_DOWNLOAD)
  • periodic_download: sets up a cron job to download files from S3 periodically (see CRON_TIME and INITIAL_DOWNLOAD)

Environment variables

Environment variablesRequiredDescription
AWS_ACCESS_KEY_IDYes(or functional IAM profile)
AWS_SECRET_ACCESS_KEYYes(or functional IAM profile)
AWS_DEFAULT_REGIONYes(or functional IAM profile)
S3PATHYesthe S3 synchronize location (ex: s3://mybucket/myprefix)
SYNCDIRYesthe local synchronize location (must be writable by the s3sync user)
AWS_S3_SSENouse S3 Server Side Encryption; it can be false for no encryption, aes256 or true for Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3) and kms for Server-Side Encryption with AWS KMS-Managed Keys (SSE-KMS) (defaults to false). For more information refer to https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html (Note: Server-Side Encryption with Customer-Provided Keys (SSE-C) is not currently supported)
AWS_S3_SSE_KMS_KEY_IDNoThe AWS KMS key ID that should be used to server-side encrypt the object in S3 (only available if use in conjunction with AWS_S3_SSE)
CRON_TIMENoa valid cron expression (ex: CRON_TIME='0 */6 * * *' runs every 6 hours; defaults to hourly)
INITIAL_DOWNLOADNowhether to download files initially (defaults to true); this will only download the files if the directory is empty. Set this to force to skip this check
SYNCEXTRANoadd extra options to aws-cli sync command

Usage

Download files and exit
docker run --rm \
  -e S3PATH='s3://mybucket/myprefix' \
  zappi/s3sync
Upload files and exit
docker run --rm \
  -e S3PATH='s3://mybucket/myprefix' \
  zappi/s3sync upload
Upload files periodically (every 6 hours)
docker run -d \
  -e S3PATH='s3://mybucket/myprefix' \
  -e CRON_TIME='0 */6 * * *' \
  zappi/s3sync cron
Watch local directory
docker run -d \
  -e S3PATH='s3://mybucket/myprefix' \
  zappi/s3sync sync
Watch the specified local directory (host mount)
docker run -d \
  -e S3PATH='s3://mybucket/myprefix' \
  -e SYNCDIR='/mydir' \
  -v $(pwd):/mydir \
  zappi/s3sync sync
External AWS credentials
docker run -d \
  -e S3PATH='s3://mybucket/myprefix' \
  -v ~/.aws:/root/.aws:ro \
  zappi/s3sync

Notes:

  • The --rm flag in the "Download files and exit" command removes the container after it exits, avoiding a build-up of stopped containers.
  • In "Upload files and exit" command, the upload argument tells the container to upload files before exiting.
  • In "Watch the specified local directory (host mount)" command, $(pwd) is used to specify the current directory as the one to sync. Change this to the appropriate directory for your use case.

Credits

This image is based heavily on the work of @vladgh's s3sync base image.

Docker Pull Command

docker pull zappi/s3-sync