Public | Automated Build

Last pushed: 3 months ago
Short Description
Docker image builder for AWS Elastic Container Registry
Full Description

Argus

Extremely simple ruby daemon to build docker images and push them to
AWS Elastic Container Registry.

Features:

  • does one thing well (kinda)
  • queue builds with SQS messages
  • built-in ECR authentication
  • can notify Slack of build details

The primary goal of this project is simplicity. It should be easy to
adapt to your exact image-building needs.

Argus is implemented as a shoryuken
worker. Shoryuken is a super
efficient AWS SQS thread based message processor.

Quick-start

You will need:

  • AWS credentials set, either with the usual environment variables or
    with a credentials file
  • a github repo to build, and access to it, with ssh keys or
    GITHUB_TOKEN
  • a repository created on ECR to push the docker image once built

If you have local docker, you can run a single build with:

argus-build rlister/argus:master

To run the builder daemon, create an SQS queue called argus, then:

gem install argus-builder
argus-send rlister/argus:master  # replace with your github repo
argus-worker

Installation

Install via ruby gems:

gem install argus-builder

or clone from git:

git clone rlister/argus

Usage

argus is implemented as a
Shoryuken worker. It will poll
SQS for build messages, and run:

  • git pull
  • docker build
  • docker push to ECR

Set the following environment variables to configure argus:

export GITHUB_TOKEN=xxx
export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID=xxx
export AWS_SECRET_ACCESS_KEY=xxx
export ARGUS_QUEUE=my_queue
export ARGUS_HOME=/data
export SLACK_WEBHOOK=https://hooks.slack.com/services/xxx

Git repositories will be kept in the ARGUS_HOME directory (default /tmp).

Send a message to your queue:

argus-send org/repo:branch

Start the worker daemon to start consuming the queue:

argus-worker

Build procedure

Argus builds images as follows. Examine and modify lib/argus/git.rb
and lib/argus/docker.rb to change exact behaviour.

  1. git clone a new github repo, or git checkout an existing one
  2. authenticate to elastic container registry using provided credentials
  3. attempt to docker pull existing image from repo to take advantage of cache
  4. docker build the new image
  5. docker push to the registry
  6. notify Slack

Message format

Implementation of your own message sender is simple, using any library
that can push a JSON object to SQS.

Example:

require 'aws-sdk'

msg = {
  org:    org,
  repo:   repo,
  branch: branch,     # optional, defaults to master
  tag:    repo:branch # optional. defaults to repo:branch
}

sqs = Aws::SQS::Client.new
sqs.send_message(
  queue_url:    sqs.get_queue_url(queue_name: myqueue).queue_url,
  message_body: msg.to_json
)

If no tag is given it will be constructed as
registry/repo:branch. The git branch will have / changed to - to
make a legal docker image name.

For example, git repo:

rlister/argus:feature/foo

will produce docker image:

xxx.dkr.ecr.us-east-1.amazonaws.com/argus:feature-foo

Shoryuken options

argus is a simple shoryuken worker. You can run it directly from the
argus repository with any shoryuken options you like:

bundle exec shoyruken -r ./lib/argus/worker.rb -q myqueue -c 1

It is advisable to run a single thread (concurrency of 1) per host, as
argus shares build directory between builds, to take advantage of git
and docker caching.

Writing your own shoryuken worker class is straightforward, see
lib/argus/worker.rb for details.

Docker

Argus runs happily inside a docker container, but needs access to a
docker daemon to trigger builds. For example, using local socket:

docker run \
  --name argus \
  -e GITHUB_TOKEN \
  -e AWS_REGION \
  -e AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY \
  -e ARGUS_QUEUE \
  -e ARGUS_HOME=/data \
  -e SLACK_WEBHOOK \
  -v /data:/data \
  -v /var/run/docker.sock:/var/run/docker.sock \
  rlister/argus:latest

You will probably want to bind-mount the ARGUS_HOME data directory,
to preserve git repos between container restarts. This avoids cloning
the entire repos on the next build.

Development

After checking out the repo, run bin/setup to install
dependencies. You can also run bin/console for an interactive prompt
that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in
version.rb, and then run bundle exec rake release, which will
create a git tag for the version, push git commits and tags, and push
the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at
https://github.com/rlister/argus. This project is intended to be a
safe, welcoming space for collaboration, and contributors are expected
to adhere to the
Contributor Covenant code of
conduct.

License

The gem is available as open source under the terms of the
MIT License.

Docker Pull Command
Owner
rlister
Source Repository

Comments (0)