Public | Automated Build

Last pushed: a year ago
Short Description
An infinitely scalable and cheap queue worker.
Full Description

lambda-queue-worker

This is a template that can be used to create an infinitely scalable SQS queue
worker! It is expecting you to use NodeJS. The index.js currently only logs
outputs.

The dockers created with the files found in this directory allow you to easily
create lambda queue workers. This gives you infinite scalability and lower
costs than having to constantly be running machines that may or may not be
handling your queue at times.

Usage

Please note that you will have to launch both of the Dockers on one instance,
and then keep that instance running to make sure the queue items reach the
lambda functions.

Make sure to set and pass all the environment variables found in .env.

Create the Dockerfile.lambda first, so you have a function deployed
already. Please note that initially on cloning this repo you will only
have an index.js which simply logs out what it receives, you will have to
add your business logic from here.

Next up, the Dockerfile, this facilitates the notifications coming in on the
SQS queue so it makes sure they land at the right lambda function. The instance
that runs this continuous docker service can be very low, as it is just a pass-
through until AWS officially supports SQS as an event source.

You now have a service that can instantly handle the queue and infinitely scale.

fleet Service files

Since we are already building the main Dockerfile, you only have to build the
Dockerfile.lambda.

lambda-queue-worker@.service:

[Unit]
Description=lambda-queue-worker

[Service]
User=core
TimeoutStartSec=0
Restart=always
ExecStartPre=-/usr/bin/docker pull your-registry/your-lambda-function
ExecStartPre=-/usr/bin/docker pull pebbletech/lambda-queue-worker
ExecStartPre=-/usr/bin/docker kill sqs-to-lambda
ExecStartPre=-/usr/bin/docker rm sqs-to-lambda
ExecStartPre=-/bin/sh -c "/usr/bin/docker run --rm --name your-lambda-function --env-file=.env your-registry/your-lambda-function"
EnvironmentFile=/etc/environment
ExecStart=/bin/sh -c "/usr/bin/docker run --rm --name sqs-to-lambda --env-file=.env pebbletech/lambda-queue-worker:latest"
ExecStop=/usr/bin/docker stop sqs-to-lambda

Role

Your instance role will have to allow a few things to continue.

Instance role

{
    "Statement": [
        {
            "Resource": "*",
            "Action": [
                "lambda:CreateAlias",
                "lambda:CreateFunction",
                "lambda:ListFunctions",
                "lambda:UpdateFunctionCode",
                "lambda:UpdateFunctionConfiguration",
                "lambda:UpdateAlias",
                "lambda:PublishVersion",
                "lambda:GetFunction",
                "lambda:InvokeFunction",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets"
            ],
            "Effect": "Allow"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:PassRole"
            ],
            "Resource": "arn:aws:iam::000000000000:role/Role"
        },
        {
            "Resource": "arn:aws:logs:*:*:*",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Effect": "Allow"
        },
        {
            "Resource": "*",
            "Action": [
                "ec2:CreateNetworkInterface",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DetachNetworkInterface",
                "ec2:DeleteNetworkInterface"
            ],
            "Effect": "Allow"
        }
    ],
    "Version": "2012-10-17"
}

Trust Relationship role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Development testing

Install npm install -g git+https://git@github.com/deviavir/node-lambda#vpc-config-support
locally, you can then adjust the event.json with how you expect your event to
arrive at your Lambda function.

You can test your function by running node-lambda run in your project
directory.

Cost

Depending on the amount of time your finished lambda function needs to process
your queue items, you can make the following assumptions:

20.000.000 SQS items
128 MB memory
3 seconds execution time

$3.80 / month
200.000.000 SQS items
128 MB memory
3 seconds execution time

$39.80 / month
250.000.000 SQS items
128 MB memory
3 seconds execution time

$49.80 / month
250.000.000 SQS items
512 MB memory
10 seconds execution time

$63.97 / month

Feel free to calculate for your own use case, Matthew D Fuller created a
cost calculcator: http://blog.matthewdfuller.com/p/aws-lambda-pricing-calculator.html

Docker Pull Command
Owner
pebbletech
Source Repository