Public | Automated Build

Last pushed: 3 months ago
Short Description
Docker volume container which restores and backs-up data to various types of storage
Full Description

Over 900 people downloaded this image within days of its creation, so I guess I have to say something :)

What this image does

  1. When you execute this image, it will first pull the latest backup from the underlying persistent storage.
  2. Next, it will monitor the data folder's file system for changes (create, modify, add, remove) - recursively.
  3. If a change is detected, it will wait for a 'quiet period' of N seconds, before initiating an incremental backup to the underlying store. Quiet period == no changes.

Features

  • Secure: Encrypted and signed archives, and transport
  • Bandwidth and space efficient
  • Standard file format: tar + rdiff
  • Choice of remote protocol: scp/ssh, ftp, rsync, HSI, WebDAV, Tahoe-LAFS, and Amazon S3
  • Choice of backend: S3 Amazon Web Services, Google Cloud Storage, Rackspace Cloud, Dropbox, copy.com, ftp, ftps, gdocs, gio, rsync, mega, swift...

I'm using it under CoreOS, so this is a part of my unit file that shows how you could use it:

ExecStartPre=-/usr/bin/rm -rf ${DATA_DIR} 
ExecStartPre=-/usr/bin/mkdir ${DATA_DIR}

ExecStart=/bin/bash -c  ' \     
  /usr/bin/docker run \     
    -v  ${DATA_DIR}:/var/backup \  
    --name=${CONTAINER_NAME} \  
    --rm \ 
    -e  AWS_ACCESS_KEY_ID=<ID> -e AWS_SECRET_ACCESS_KEY=<SECRET> \
    --privileged  \
    yaronr\backup-volume-container s3://s3.amazonaws.com/<bucket_name>/backup 60'

60 is the quiet period. It's important to keep this value high enough, else you will be creating a lot of backups.

Backups are incremental, and can be encrypted.
Underlying persistent stores include:

This image uses Duplicity for backup and restore. Go to the Duplicity docs to see more.

The way I use it, is to map the container's data dir to a host dir, and map host data/subdirs to relevant containers.
You could also map it directly to your various docker containers without the host.

I prefer having the host as an intermediary, because it increases robustness and is a cleaner approach in my view.

I built this image for my own use, but the source code is available and I will be very to receive pull requests that enhance / improve this solution, as well as comments whether or not you find this useful and how you use it.

Docker Pull Command
Owner
yaronr
Source Repository

Comments (9)
netquest
a year ago

So...if this is using duplicity and you're doing log backups that use non date format (mean standard logrotate ones like myapp.log.3.gz) duplicity can't make incremental backups of this ones and the local and remote indexes getting bigger every time you do a backup? or this is solved?

axot
2 years ago

I want preform a mysqldump backup, how can i do this?

kopax
2 years ago

Is it possible to use it with supervisor instead of coreos services do you know ?

yaronr
2 years ago

@raitom seriously?

google: "duplicity documentation"
First link:
http://duplicity.nongnu.org/docs.html

raitom
2 years ago

@yaronr : Where can I find the duplicity documentation?

yaronr
2 years ago

@insanity54 just read the duplicity documentation.

insanity54
2 years ago

How do you use this with Dropbox?

yaronr
3 years ago

suda
What I do is, as part of cloud-init, I write a file to /opt/conf/environment, which contains certain variables that are going to remain true throughout the life of this host, such as: ${DATA_DIR}.

And then, in the unit file, I do:
EnvironmentFile=/etc/environment
EnvironmentFile=/opt/conf/environment

${CONTAINER_NAME} is just a matter of tidying up your unit files. Set it to a value and re-use it in all other places. For example:

Environment="IMAGE_NAME=yaronr/openjdk-7-jre"
Environment="CONTAINER_NAME=cloudcontrol"

etc etc.

suda
3 years ago

This is a great container man!
One CoreOS specific questions though:
In your unit file you mention ${DATA_DIR} and ${CONTAINER_NAME}. where those meant to be just placeholder for example or do you set those variables somewhere?