Public | Automated Build

Last pushed: a year ago
Short Description
Backup SETS hosts to an S3 bucket per a backup configuration file.
Full Description

Overview

A Docker container that provides backups to an S3 bucket

Backup a Host

See CMD line in Dockerfile for a description of each ENV used by the backup service.

The key input is a json-formatted dictionary file that defines what needs to be backed up, and the backup schedule.

The dictionary file is passed to the container via the CONFIGS_PATH environment variable.

The dictionary file MUST exist in an S3 bucket, and MUST be specified by an S3 URL, formatted as:
s3://my-bucket/my-path/my-key

A sample configuration dictionary is as follows:

{
  "period-min": "60",
  "period-min-comment": "use the period-min field to specify the periodicity of backup runs in minutes (e.g. period-min=60 triggers hourly backups)",
  "files": {
    "server.xml": {
       "src":  "/var/local/myapp/WEB-INF/server.xml",  
       "dest": "s3://tts-sets-team/myapp/WEB-INF/server.xml"
    }
  },
  "directories": {
    "attachments": {
       "src":     "/var/local/myapp/attachments",  
       "dest":    "s3://tts-sets-team/myapp/attachments",
       "comment": "we should not delete attachments in the backup folder if they are are no longer present in the source. this allows us to recover attachments that are accidentally deleted from the app"
    },
    "configs": {
       "src":     "/var/local/myapp/configs",  
       "dest":    "s3://tts-sets-team/myapp/configs",
       "options": "--delete",
       "comment": "the --delete option is set because src and dest should be kept in perfect sync"
    },
    "archives": {
       "src":     "/var/local/myapp/archives",  
       "dest":    "s3://tts-sets-team/myapp/archives",
       "options": "--delete --exclude \"*\" --include \"*.gz\"",
       "comment": "backup the archives directory, but include ONLY files with a .gz extension"
    }
  } 
}

Files are backed up first, in their dictionary order.

Directories are backup up next, in their dictionary order.

Directory backups and restores are implemented using the "aws s3 sync" cli utility. The "options" attribute in the dictionary above is per the Options portion of the sync command, i.e.

aws s3 sync <LocalPath> <S3Path> <Options>

The backup service needs access to the src directories being backed up, typically via volume arguments to docker run.

The following command will create a container named 'backups', where the backup instructions and periodicity are per backups.json.

sudo docker run -d -e "CONFIGS_PATH=s3://tts-sets-team/myapp/backups.json" -e "HOSTNAME=myapp-dev" -v /var/local/myapp:/var/local/myapp:ro --name backups -it nordstromsets/backups:latest

The container can also be used to take a "snap" backup. The command below will launch a task container that takes a backup immediately (per backup instructions is backups.json) and then returns.

sudo docker run -e -v /var/local/myapp:/var/local/myapp:ro -it nordstromsets/backups:latest python -m src.backup s3://tts-sets-team/myapp/backups.json myapp-dev -s

Restore a Host

The container can also be used to restore the directories and files that have been backed up.

Here again, the service needs access to the src directories to restore into, typically via volume arguments to docker run/exec.

New Host

To restore on a new Host that is not yet running the Backup container, ssh to the host and restore as:

sudo docker run -t nordstromsets/backup:latest python -m src.restore s3://tts-sets-team/myapp/backups.json myapp-dev

Existing Host - Restore All

To restore on a Host that is already running the Backup container, ssh to a host and restore as:

sudo docker run exec -it backup python -m src.restore s3://tts-sets-team/myapp/backups.json myapp-dev

Restore Selected

You can instead restore selected files or directories by specifying the keys of each (per the configs dictionary).

For example, to restore JUST the attachments directory with key "attachments", and the server.xml file with key "server.xml" defined in backups.json, ssh to a host running the backup container and restore as:

sudo docker run exec -t backup python -m src.restore s3://tts-sets-team/myapp/backups.json myapp-dev --dirs attachments --files server.xml

To specify multiple files or directories separate each key with a comma, e.g.

... --dirs attachments,configs ...

Logs

The backups job logs to the /var/local/backups/backup.log file, per conf/logger.conf

Logs can be forwarded to a log aggregator such that backup errors and exceptions can be trapped and alerted appropriately.

Docker Pull Command
Owner
nordstromsets
Source Repository

Comments (0)