It's a general purpose to solve backup matter on Rancher.
The goal, it's to have ability to use docker command to perform dump (when needed) before to start external backup with duplicity.
To do the job in easiest way, we use the power of Rancher API to discover the service witch must be dumped before to start the backup.
We use some settings files on
/app/config to explain how discover the service witch must be dumped and how to do that.
Next, all the contains of
BACKUP_PATH (default is /backup) is backuped on remote backend with duplicity. So you can map your data volume on this container to backup it in the same time.
You are welcome to contribute on github to extend the supported service.
Standard databases compatibilities
No extra need, use dump tools utilities to do remote dump.
MySQL: the docker image must have
mysqlon name. Use
mysqldumpto perform the dump.
MariaDB: the docker image must have
mariadbon name. Use
mysqldumpto perform the dump.
PostgreSQL: the docker image must have
postgreson name. Use
pgdumpto perform the dump.
MongoDB: the docker image must have
mongoon name. Use
mongodumpto perform the dump.
Elasticsearch: the docker image must have
elasticsearchon name. Use
elasticdumpto perform the dump.
Distributed NoSQL databases
Need to have shared volume (like glusterfs, S3, Ceph, etc.) between each database nodes and the backup service.
To to dump, we use tools utilities to ask each nodes perform a local dump (on shared volume) and we mount this shared volume on backup service to perform the remote backup.
For example, if you have 3 Cassandra nodes on 3 hosts, you must to have sharded storage on each hosts (
/mnt/cassandra) witch is mounted on each nodes (
Then, you need to mount the shared storage on backup service (
When we detect Cassandra service, we send command to Cassandra to ask it to perform a dump of each nodes on
/dump, ans next we perform a backup with duplicity of
Disable dump on specific service
If you should to not dump a particular service witch is supported, you can add label on service
The following options permit to set the backup policy :
CRON_SCHEDULE: when you should start backup (incremental if full is not needed). For example, to start backup each day set
0 0 0 * * *
BACKEND: this is the target URL to externalize the backup. For example, to use FTP as external backup set
ftp://firstname.lastname@example.org add environment variable
FTP_PASSWORD. For Amazon S3, set
s3://host[:port]/bucket_name[/prefix]. Read the ducplicity man for all supported backend. There are no default value.
TARGET_PATH: The path were store backup on local and remote. The default value is
BK_FULL_FREQ: The frequency when you should make a full backup. For example, if you should make a full backup each 7 days, set
7D. The default value is
BK_KEEP_FULL: How many full backup you should to keep. For example, to keep 3 full backup set
3. The default value is
BK_KEEP_FULL_CHAIN: The number of intermediate incremental backup you should keep with the full backup. For example, if you should keep only the incremental backend after the last full backup set
1. The default value is set to
VOLUME_SIZE: The volume size to store the backup (in MB). The default value is
To set the Rancher API connection prefer to add special label that generate access on the flow:
Or you can define them manually :
CATTLE_URL: the API URL with your project ID
CATTLE_ACCESS_KEY: the API key
CATTLE_SECRET_KEY: the API secret key
How to extend this
You need to dump another service before to save it (note yet supported) ? Just clone this repository and add the file in
Then, add your new entry (sample with MySQL):
mysql: regex: "mysql" image: "mysql:latest" commands: - "mysqldump -h %ip% -u %env_MYSQL_USER% %env_MYSQL_DATABASE% > %target_dir%/%env_MYSQL_DATABASE%.dump" environment: - MYSQL_PWD:%env_MYSQL_PASSWORD%
regex: It's the regex to discover service witch must be dumped. This regex is applied to image docker used in service.
image: It's the docker image to use to run the dump (generaly the latest tag). If you not add image entry, it use the service docker image.
commands: It's the list of commands to launch on container to perform the dump
environment: It's the list of environment variables you need to perform the dump
There are few macro you can use in command and in environment section:
%ip%: the IP to join the container to perform a remote dump
%env_SERVICE_ENV%: Take the value of service environment called
%target_dir%: It's the path where store the dump (