Public | Automated Build

Last pushed: 10 days ago
Short Description
test build
Full Description

AWSBouncer

A very simple local service that will accept messages to enqueue to SQS
and return quickly, so our web services don't wait to write to SQS.

$ bin/awsbouncer -h
Usage of bin/awsbouncer:
  -aws-region="": aws region
  -buf-size=1000: message channel buffer size
  -dd-host="127.0.0.1": datadog agent host
  -dd-port=8125: datadog agent port
  -dd-prefix="awsbouncer": statsd prefix
  -port=6554: port to listen on

Getting started with Go

Check out the excellent getting started guide in the Bullseye repo.

The following section describe setup for local development. For more information
on other topics, there is also the Dev Documentation

Building

Check out the repo in $GOPATH/src/github.com/thegroundwork/awsbouncer then run make getdeps and make build:

$ mkdir $GOPATH/src/github.com/thegroundwork/
$ cd $GOPATH/src/github.com/thegroundwork/
$ git clone git@github.com:thegroundwork/awsbouncer.git
$ cd $GOPATH/src/github.com/thegroundwork/awsbouncer
$ make getdeps
$ make build

This will result in godep getting installed to help w/ package management, dependencies of the project gettings installed, and then the application getting built to bin/awsbouncer.

Running

You must specify a region, either via the command line or an environment
variable called REGION:

$ bin/awsbouncer -aws-region=us-east-1
# - OR -
$ REGION=us-east-1 bin/awsbouncer

awsbouncer will get AWS credentials automatically from the environment
or ~/.aws/credentials (or an IAM profile). If you have multiple
profiles in your credentials file, use the AWS_PROFILE environment
variable to select among them.

You can also change the listening port (default: 6554) or the datadog
agent address (default: 127.0.0.1:8125) or stats prefix (default:
awsbouncer).

Finally you can configure the internal message buffer size. If the
internal message buffer fills up, calling applications will see errors.
The default is reasonably high, but if you need to test error conditions
or memory pressure is an issue, you can lower it. If you're not seeing
memory issues but need more buffer space, you can raise it.

NB: Do not set the buffer size to 0, it will change the program
behavior.

Using

Deployment / Installation

If you want to add awsbouncer as a deployment dependency, add requires('awsbouncer') to your buildDeb object in your build.gradle file:

buildDeb {
  postInstall file('scripts/after-install.sh')

  // … other dependencies …
  requires('awsbouncer')
}

Next you will need to add the following code to your ansible configuration code:

- name: Ensure awsbouncer is started
  service: name=awsbouncer state=restarted

Python

For SQS publishing, you can use the Python sqr library:

$ pip install --extra-index-url=<dev-pypi> gw-sqr

For Kinesis publishing, you can use the Python kinesis-publisher library:

$ pip install --extra-index-url=<dev-pypi> gw-kinesis-publisher

See the README for details.

Other languages

Any language with an HTTP client can post messages via awsbouncer.
Just make a POST request, with the name of the destination queue as
the path, and the body of the message as the post body. E.g.:

POST /sqs/some-queue HTTP/1.0

{"my":"message","goes":"here"}

No additional processing is done to the message body, it is written to
SQS as-is.

awsbouncer will respond immediately with either a 200 OK indicating
that the message was buffered for sending to SQS, or a 503 Service Unavailable indicating that the internal buffer was full.

Operations Notes

When requests to AWS fail, awsbouncer tries to back up the request to S3. The name of the S3 buckets for each environment is defined in the environment's GWProperties instance under the awsbouncer.s3_backup_bucket key.

As of 10/7/2015, the backup bucket names are:

  • GW Dev: gw-dev-awsbouncer-backup
  • GW Prod: gw-prod-awsbouncer-backup
  • Campaign Dev: campaign-dev-awsbouncer-backup
  • Campaign Prod: campaign-prod-awsbouncer-backup

If awsbouncer fails to save to S3, messages are backed up to /opt/awsbouncer/message_backup.

Once requests are working again, messages that have been backed up can be resent by using the awsbouncer_reaper. This project has already been deployed to each environment and the ASGs can just be resized to start re-sending messages accordingly.

More Information

Docker Pull Command
Owner
thegroundwork
Source Repository

Comments (0)