zmazay/s3sync-service
Realtime S3 synchronisation service/daemon with k8s integration
100K+
zmazay/s3sync-service repository overview
s3sync-service
The s3sync-service is a lightweight tool designed with k8s in mind and aimed to syncing data to S3 storage service for multiple sites (path + bucket combination). Each site can have its own set of credentials and be in different region, which makes the s3sync-service really flexible.
Check out the quickstart note or head over to the documentation where you will find more examples on running the application locally or on k8s.
Quickstart
- Create directory with configuration file, eg. -
/path/to/config/config.yml. - Run docker container with providing AWS credentials via environment variables (IAM role should also do the trick), alternatively credentials could be provided in the config file, mount directory containing the config file and all of the backup directories listed in the config file:
docker run --rm -ti \
-e "AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE" \
-e "AWS_SECRET_ACCESS_KEY=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY" \
-e "AWS_DEFAULT_REGION=us-east-1" \
-v "/path/to/config:/opt/s3sync-service" \
-v "/backup/path:/backup" \
zmazay/s3sync-service \
./s3sync-service -config /opt/s3sync-service/config.yml
Docker Compose can also be used:
version: '3.3'
services:
s3sync-service:
environment:
- AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE
- AWS_SECRET_ACCESS_KEY=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
- AWS_DEFAULT_REGION=us-east-1
volumes:
- '/backup/path:/backup'
- '/path/to/config.yml:/app/config.yml'
image: zmazay/s3sync-service
Configuration
Example configuration, check this for more details:
upload_workers: 10
sites:
- local_path: /local/path1
bucket: backup-bucket-path1
bucket_region: us-east-1
storage_class: STANDARD_IA
access_key: AKIAI44QH8DHBEXAMPLE
secret_access_key: je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
exclusions:
- .[Dd][Ss]_[Ss]tore
- .[Aa]pple[Dd]ouble
- local_path: /local/path2
bucket: backup-bucket-path2
bucket_path: path2
exclusions:
- "[Tt]humbs.db"
- local_path: /local/path3
bucket: backup-bucket-path3
bucket_path: path3
exclusions:
- "[Tt]humbs.db"
Contributing
If you feel like contributing to the project - there are various ways of doing so.
Support
You can buy me a coffee if you feel like supporting my motivation in working on this project. :)
Tag summary
Content type
Image
Digest
sha256:3abec0180…
Size
23.8 MB
Last updated
2 months ago
docker pull zmazay/s3sync-service