Sign inSign up

lstellway/sync-s3

By lstellway

•Updated over 4 years ago

Docker image to synchronize S3 endpoints

Image
0

2.9K

lstellway/sync-s3 repository overview

⁠Sync Data to S3

Docker container image used to backup data to S3 using AWS cli⁠.

⁠Environment Variables

This script utilizes the AWS command-line interface program.
As a result, this program supports related environment variables⁠.
Refer to the documentation⁠ to learn how to configure AWS cli (including authentication).

  • AWS_ACCESS_KEY_ID_FILE
    • Path to file containing S3 access key
  • AWS_SECRET_ACCESS_KEY_FILE
    • Path to file containing S3 access secret
  • FORCE_INITIAL_RESTORE (Default: "false")
    • Whether or not to force a restore even if the ALPHA location is not empty
  • BACKUP_INTERVAL (Default 42)
    • Time (in seconds) in between synchronization attempts
  • S3_ENDPOINT_URL
    • S3 endpoint URL
  • S3_SYNC_RESTORE_FLAGS
  • S3_SYNC_BACKUP_FLAGS

⁠Usage

The entrypoint command takes two arguments:

  1. "Alpha" - The data location to backup
  2. "Beta" - The destination to backup data to

Pull the docker image⁠

docker pull lstellway/sync-s3

Run a container

# This commanad will backup to an S3 bucket from the local `/data` directory
docker run --rm -it lstellway/sync-s3 /data s3://bucket/path/to/destination

# s3:// endpoints can be used for both "Alpha" and "Beta"
docker run --rm -it lstellway/sync-s3 s3://bucket1/path/to/destination s3://bucket2/path/to/destination

Example Docker Compose

---
version: "3"
services:
    sync-s3-test:
        command: ["/data", "s3://bucket/path/to/destination"]
        container_name: sync-s3-test
        environment:
            AWS_ACCESS_KEY_ID: xxxxxxxxxx
            AWS_SECRET_ACCESS_KEY: xxxxxxxxxxxxxxxxxxxx
            AWS_DEFAULT_REGION: nyc3
            S3_ENDPOINT_URL: https://nyc3.digitaloceanspaces.com
            FORCE_INITIAL_RESTORE: "true"
        # Stat file every 10 seconds.
        # This configuration allows for a ~15 minute startup time to download files:
        # 10s interval x 90 retries = 900s / 60 = 15min
        healthcheck:
            test: ["stat", "/tmp/running"]
            interval: 10s
            timeout: 5s
            retries: 90
            start_period: 5s
        image: lstellway/sync-s3
        volumes:
            - sync-s3-data:/data
volumes:
    sync-s3-data:
        name: sync-s3-data

Immediately Invoke a Backup

docker kill --signal=USR1 <container-name-or-id>

Immediately Invoke a Restore

docker kill --signal=USR2 <container-name-or-id>

⁠Resources

⁠Credits

While working on my own solution for backing up data to S3 from containers, I came across the elementar/docker-s3-volume⁠ project. While my solution was headed in the same direction, the project was a very useful reference which I was able to borrow some ideas from.

Tag summary

Content type

Image

Digest

Size

30.1 MB

Last updated

over 4 years ago

docker pull lstellway/sync-s3