jdrydn/github-to-s3 repository overview

⁠github-to-S3

• A Docker container to dump GitHub repo tarballs directly to S3.
• Wraps around rclone/rclone⁠ to support backing up multiple repos in series.
• Designed to be dropped into an ECS Fargate scheduled CRON task, passing all configuration through IAM roles & environment variables.
• Single-process execution, and the container terminates when finished.

$ docker pull jdrydn/github-to-s3:latest
$ docker run \
  -e GITHUB_TOKEN=your-github-token \
  -e S3_BUCKET=project-backups \
  jdrydn/github-to-s3:latest \
  jdrydn/Dockerfiles jdrydn/yoem
ACTION: Downloading jdrydn/Dockerfiles to s3:project-backups/20191012T131547Z-jdrydn-Dockerfiles.tar.gz
ACTION: Downloading jdrydn/yoem to s3:project-backups/20191012T131547Z-jdrydn-yoem.tar.gz
SUCCESS!

• You will need to create a personal access token⁠ to reach private repositories with this image. If you're concerned about using your personal access token with an organisation, take a look at machine users⁠.

⁠Configuration

To configure this script or rclone, pass environment variables at runtime:

Pass a list of all the repositories you want to be backed up to the Docker CMD. Files will be named $S3_PREFIX$S3_FILENAME-$REPO.tar.gz.

You can configure other variables too, such as the AWS region, using RCLONE_CONFIG_S3_REGION and other env vars⁠.

You could even use this image to backup GitHub repositories to an S3-compatible source, such as DigitalOcean⁠ or Minio⁠, by using the RCLONE_CONFIG_S3_ variables to set relevant endpoints etc.

⁠Examples

⁠Manually

$ docker run --rm \
  --env-file ./backup.env \
  jdrydn/github-to-s3:latest
  jdrydn/yoem
$ docker run --rm \
  -e GITHUB_TOKEN=your-github-token \
  -e S3_BUCKET=project-backups \
  jdrydn/github-to-s3:latest \
  jdrydn/yoem

⁠ECS Scheduled Task

• Head over to ECS and create a new task definition⁠.
  • Select a Task Role that has S3::putObject permissions to your target bucket.
  • Add a container & set the Image field to the latest tag⁠ or any of the versions listed here⁠: jdrydn/github-to-s3:latest
  • Under Environment Variables, enter your configuration following the variables above. Here you can use SSM⁠ values, so you could keep the GITHUB_TOKEN values in sync with your application, or similar.
• Next, head over to the Scheduled Tasks⁠ in your cluster & create a new scheduled task. Enter a CRON expression⁠ to describe how often you want this to run, select the task definition you made earlier, and hit create.

⁠Manually start a scheduled task

$ aws ecs run-task \
  --cluster default \
  --task-definition database-backup \
  --launch-type FARGATE \
  --network-configuration 'awsvpcConfiguration={subnets=[A,B,C],securityGroups=[D],assignPublicIp=ENABLED}'

• Useful for debugging issues with the scheduled task.
• Remember to put in relevant values for A,B,C & D.

⁠Development

$ docker build -t jdrydn/github-to-s3:dev .
$ cat << EOF >> .env
GITHUB_TOKEN=your-github-token
S3_BUCKET=backup-bucket
S3_PREFIX=codebase-backups-dev/
EOF
$ docker run --rm -it --network=host \
  -v ~/.aws/credentials:/root/.aws/credentials:ro \
  --env-file ./.env \
  jdrydn/github-to-s3:dev \
  jdrydn/Dockerfiles

⁠With Thanks

• rclone/rclone⁠ & @ncw⁠.