trestini/localstack repository overview

⁠LocalStack - A fully functional local AWS cloud stack

LocalStack provides an easy-to-use test/mocking framework for developing Cloud applications.

Currently, the focus is primarily on supporting the AWS cloud stack.

⁠Announcements

• 2018-01-10: Help wanted! Please fill out this survey⁠ to support a research study on the usage of Serverless and Function-as-a-Service (FaaS) services, conducted at Chalmers University of Technology. The survey only takes 5-10 minutes of your time. Many thanks for your participation!!
• 2017-08-27: We need your support! LocalStack is growing fast, we now have thousands of developers using the platform on a regular basis. Last month we have recorded a staggering 100k test runs, with 25k+ DynamoDB tables, 20k+ SQS queues, 15k+ Kinesis streams, 13k+ S3 buckets, and 10k+ Lambda functions created locally - for 0$ costs (more details to be published soon). Bug and feature requests are pouring in, and we now need some support from you to keep the open source version actively maintained. Please check out Open Collective⁠ and become a backer⁠ or supporter⁠ of the project today! Thanks everybody for contributing. ♥
• 2017-07-20: Please note: Starting with version 0.7.0, the Docker image will be pushed and kept up to date under the new name localstack/localstack. (This means that you may have to update your CI configurations.) Please refer to the updated End-User License Agreement (EULA)⁠ for the new versions. The old Docker image (atlassianlabs/localstack) is still available but will not be maintained any longer.

⁠Overview

LocalStack spins up the following core Cloud APIs on your local machine:

• API Gateway at http://localhost:4567⁠
• Kinesis at http://localhost:4568⁠
• DynamoDB at http://localhost:4569⁠
• DynamoDB Streams at http://localhost:4570⁠
• Elasticsearch at http://localhost:4571⁠
• S3 at http://localhost:4572⁠
• Firehose at http://localhost:4573⁠
• Lambda at http://localhost:4574⁠
• SNS at http://localhost:4575⁠
• SQS at http://localhost:4576⁠
• Redshift at http://localhost:4577⁠
• ES (Elasticsearch Service) at http://localhost:4578⁠
• SES at http://localhost:4579⁠
• Route53 at http://localhost:4580⁠
• CloudFormation at http://localhost:4581⁠
• CloudWatch at http://localhost:4582⁠
• SSM at http://localhost:4583⁠
• SecretsManager at http://localhost:4584⁠
• StepFunctions at http://localhost:4585⁠
• CloudWatch Logs at http://localhost:4586⁠
• STS at http://localhost:4592⁠
• IAM at http://localhost:4593⁠
• EC2 at http://localhost:4597⁠

Additionally, LocalStack provides a powerful set of tools to interact with the cloud services, including a fully featured KCL Kinesis client with Python binding, simple setup/teardown integration for nosetests, as well as an Environment abstraction that allows to easily switch between local and remote Cloud execution.

⁠Why LocalStack?

LocalStack builds on existing best-of-breed mocking/testing tools, most notably kinesalite⁠/dynalite⁠ and moto⁠. While these tools are awesome (!), they lack functionality for certain use cases. LocalStack combines the tools, makes them interoperable, and adds important missing functionality on top of them:

• Error injection: LocalStack allows to inject errors frequently occurring in real Cloud environments, for instance ProvisionedThroughputExceededException which is thrown by Kinesis or DynamoDB if the amount of read/write throughput is exceeded.
• Isolated processes: All services in LocalStack run in separate processes. The overhead of additional processes is negligible, and the entire stack can easily be executed on any developer machine and CI server. In moto, components are often hard-wired in RAM (e.g., when forwarding a message on an SNS topic to an SQS queue, the queue endpoint is looked up in a local hash map). In contrast, LocalStack services live in isolation (separate processes available via HTTP), which fosters true decoupling and more closely resembles the real cloud environment.
• Pluggable services: All services in LocalStack are easily pluggable (and replaceable), due to the fact that we are using isolated processes for each service. This allows us to keep the framework up-to-date and select best-of-breed mocks for each individual service.

⁠Requirements

• make
• python (both Python 2.x and 3.x supported)
• pip (python package manager)
• npm (node.js package manager)
• java/javac (Java 8 runtime environment and compiler)
• mvn (Maven, the build system for Java)

⁠Installing

The easiest way to install LocalStack is via pip:

pip install localstack

Once installed, run the infrastructure using the following command:

localstack start

Note: Please do not use sudo or the root user - LocalStack should be installed and started entirely under a local non-root user. If you have problems with permissions in MacOS X Sierra, install with pip install --user localstack

⁠Running in Docker

You can also spin up LocalStack in Docker:

localstack start --docker

(Note that on MacOS you may have to run TMPDIR=/private$TMPDIR localstack start --docker if $TMPDIR contains a symbolic link that cannot be mounted by Docker.)

Or using docker-compose (you need to clone the repository first, currently requires docker-compose version 2.1+):

docker-compose up

(Note that on MacOS you may have to run TMPDIR=/private$TMPDIR docker-compose up if $TMPDIR contains a symbolic link that cannot be mounted by Docker.)

Use on existing docker-compose project. Add in existing services. The project can be found in docker hub, no need to download or clone source:

version: '2.1'
services:
...
  localstack:
    image: localstack/localstack
    ports:
      - "4567-4584:4567-4584"
      - "${PORT_WEB_UI-8080}:${PORT_WEB_UI-8080}"
    environment:
      - SERVICES=${SERVICES- }
      - DEBUG=${DEBUG- }
      - DATA_DIR=${DATA_DIR- }
      - PORT_WEB_UI=${PORT_WEB_UI- }
      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- }
      - KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "${TMPDIR:-/tmp/localstack}:/tmp/localstack"

To facilitate interoperability, configuration variables can be prefixed with LOCALSTACK_ in docker. For instance, setting LOCALSTACK_SERVICES=s3 is equivalent to SERVICES=s3.

⁠Configurations

You can pass the following environment variables to LocalStack:

• SERVICES: Comma-separated list of service names and (optional) ports they should run on. If no port is specified, a default port is used. Service names basically correspond to the service names of the AWS CLI⁠ (kinesis, lambda, sqs, etc), although LocalStack only supports a subset of them. Example value: kinesis,lambda:4569,sqs:4570 to start Kinesis on the default port, Lambda on port 4569, and SQS on port 4570. In addition, the following shorthand values can be specified to run a predefined ensemble of services:

  • serverless: run services often used for Serverless apps (iam, lambda, dynamodb, apigateway, s3, sns)

• DEFAULT_REGION: AWS region to use when talking to the API (defaults to us-east-1).

• HOSTNAME: Name of the host to expose the services internally (defaults to localhost). Use this to customize the framework-internal communication, e.g., if services are started in different containers using docker-compose.

• HOSTNAME_EXTERNAL: Name of the host to expose the services externally (defaults to localhost). This host is used, e.g., when returning queue URLs from the SQS service to the client.

• <SERVICE>_PORT_EXTERNAL: Number of the port to expose a specific service externally (defaults to service ports above) SQS_PORT_EXTERNAL, for example, is used when returning queue URLs from the SQS service to the client.

• USE_SSL: Whether to use https://... URLs with SSL encryption (defaults to false).

• KINESIS_ERROR_PROBABILITY: Decimal value between 0.0 (default) and 1.0 to randomly inject ProvisionedThroughputExceededException errors into Kinesis API responses.

• DYNAMODB_ERROR_PROBABILITY: Decimal value between 0.0 (default) and 1.0 to randomly inject ProvisionedThroughputExceededException errors into DynamoDB API responses.

• LAMBDA_EXECUTOR: Method to use for executing Lambda functions. Possible values are:

  • local: run Lambda functions in a temporary directory on the local machine
  • docker: run each function invocation in a separate Docker container
  • docker-reuse: create one Docker container per function and reuse it across invocations

  For docker and docker-reuse, if LocalStack itself is started inside Docker, then the docker command needs to be available inside the container (usually requires to run the container in privileged mode). Default is docker, fallback to local if Docker is not available.

• LAMBDA_REMOTE_DOCKER determines whether Lambda code is copied or mounted into containers. Possible values are:

  • true (default): your Lambda function definitions will be passed to the container by copying the zip file (potentially slower). It allows for remote execution, where the host and the client are not on the same machine.
  • false: your Lambda function definitions will be passed to the container by mounting a volume (potentially faster). This requires to have the Docker client and the Docker host on the same machine.

• LAMBDA_DOCKER_NETWORK Specifies the docker network for the container running your lambda function.

• DATA_DIR: Local directory for saving persistent data (currently only supported for these services: Kinesis, DynamoDB, Elasticsearch, S3). Set it to /tmp/localstack/data to enable persistence (/tmp/localstack is mounted into the Docker container), leave blank to disable persistence (default).

• PORT_WEB_UI: Port for the Web user interface (dashboard). Default is 8080.

• <SERVICE>_BACKEND: Custom endpoint URL to use for a specific service, where <SERVICE> is the uppercase service name (currently works for: APIGATEWAY, CLOUDFORMATION, DYNAMODB, ELASTICSEARCH, KINESIS, S3, SNS, SQS). This allows to easily integrate third-party services into LocalStack.

• FORCE_NONINTERACTIVE: when running with Docker, disables the --interactive and --tty flags. Useful when running headless.

• DOCKER_FLAGS: Allows to pass custom flags (e.g., volume mounts) to "docker run" when running LocalStack in Docker.

• START_WEB: Flag to control whether the Web API should be started in Docker (values: 0/1; default: 1).

• LAMBDA_FALLBACK_URL: Fallback URL to use when a non-existing Lambda is invoked. Either records invocations in DynamoDB (value dynamodb://<table_name>) or forwards invocations as a POST request (value http(s)://...).

• EXTRA_CORS_ALLOWED_HEADERS: Comma-separated list of header names to be be added to Access-Control-Allow-Headers CORS header

• EXTRA_CORS_EXPOSE_HEADERS: Comma-separated list of header names to be be added to Access-Control-Expose-Headers CORS header

Additionally, the following read-only environment variables are available:

• LOCALSTACK_HOSTNAME: Name of the host where LocalStack services are available. This is needed in order to access the services from within your Lambda functions (e.g., to store an item to DynamoDB or S3 from Lambda). The variable LOCALSTACK_HOSTNAME is available for both, local Lambda execution (LAMBDA_EXECUTOR=local) and execution inside separate Docker containers (LAMBDA_EXECUTOR=docker).

⁠Dynamically updating configuration at runtime

Each of the service APIs listed above⁠ defines a backdoor API under the path /?_config_ which allows to dynamically update configuration variables defined in config.py⁠.

For example, to dynamically set KINESIS_ERROR_PROBABILITY=1 at runtime, use the following command:

curl -v -d '{"variable":"KINESIS_ERROR_PROBABILITY","value":1}' 'http://localhost:4568/?_config_'

⁠Initializing a fresh instance

When a container is started for the first time, it will execute files with extensions .sh that are found in /docker-entrypoint-initaws.d. Files will be executed in alphabetical order. You can easily create aws resources on localstack using awslocal (or aws) cli tool in the initialization scripts.

⁠A note about using custom SSL certificates (for USE_SSL=1)

If you need to use your own SSL Certificate and keep it persistent and not use the random automatic generated Certificate, you can place into the localstack temporary directory :

/tmp/localstack/

the three named files below :

server.test.pem
server.test.pem.crt
server.test.pem.key

• the file server.test.pem must contains your key file content, your certificat and chain certificate files contents (do a cat in this order)
• the file server.test.pem.crt must contains your certificate and chains files contents (do a 'cat' in this order)
• the file server.test.pem.key must contains your key file content

⁠Using USE_SSL and own persistent certificate with docker-compose

Typically with docker-compose you can add into docker-compose.yml this volume to the localstack services :

volumes:
      - "${PWD}/ls_tmp:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

local directory ls_tmp must contains the three files (server.test.pem, server.test.pem.crt, server.test.pem.key)

⁠Accessing the infrastructure via CLI or code

You can point your aws CLI to use the local infrastructure, for example:

aws --endpoint-url=http://localhost:4568 kinesis list-streams
{
    "StreamNames": []
}

NEW: Check out awslocal⁠, a thin CLI wrapper that runs commands directly against LocalStack (no need to specify --endpoint-url anymore). Install it via pip install awscli-local, and then use it as follows:

awslocal kinesis list-streams
{
    "StreamNames": []
}

UPDATE: Use the environment variable $LOCALSTACK_HOSTNAME to determine the target host inside your Lambda function. See Configurations⁠ section for more details.

⁠Client Libraries

• Python: https://github.com/localstack/localstack-python-client⁠
  • alternatively, you can also use boto3 and use the endpoint_url parameter when creating a connection
• (more coming soon...)

⁠Integration with nosetests

If you want to use LocalStack in your integration tests (e.g., nosetests), simply fire up the infrastructure in your test setup method and then clean up everything in your teardown method:

from localstack.services import infra

def setup():
    infra.start_infra(asynchronous=True)

def teardown():
    infra.stop_infra()

def my_app_test():
    # here goes your test logic

See the example test file tests/integration/test_integration.py for more details.

⁠Integration with Serverless

You can use the serverless-localstack⁠ plugin to easily run Serverless⁠ applications on LocalStack. For more information, please check out the plugin repository here: https://github.com/localstack/serverless-localstack⁠

⁠Using local code with Lambda

In order to mount a local folder, ensure that LAMBDA_REMOTE_DOCKER is set to false then set the S3 bucket name to __local__ and the S3 key to your local path:

    awslocal lambda create-function --function-name myLambda \
      --code S3Bucket="__local__",S3Key="/my/local/lambda/folder" \
      --handler index.myHandler \
      --runtime nodejs8.10 \
      --role whatever

⁠Integration with Java/JUnit

In order to use LocalStack with Java, the project ships with a simple JUnit runner and a JUnit 5 extension. Take a look at the example JUnit test in ext/java. When you run the test, all dependencies are automatically downloaded and installed to a temporary directory in your system.

...
import cloud.localstack.LocalstackTestRunner;
import cloud.localstack.TestUtils;

@RunWith(LocalstackTestRunner.class)
public class MyCloudAppTest {

  @Test
  public void testLocalS3API() {
    AmazonS3 s3 = TestUtils.getClientS3()
    List<Bucket> buckets = s3.listBuckets();
    ...
  }

}

Or with JUnit 5 :

@ExtendWith(LocalstackExtension.class)
public class MyCloudAppTest {
   ...
}

Additionally, there is a version of the LocalStack Test Runner which runs in a docker container instead of installing LocalStack on the current machine. The only dependency is to have docker installed locally. The test runner will automatically pull the image and start the container for the duration of the test. The container can be configured by using the @LocalstackDockerProperties annotation.

@RunWith(LocalstackDockerTestRunner.class)
@LocalstackDockerProperties(services = { "sqs", "kinesis:77077" })
public class MyDockerCloudAppTest {

  @Test
  public void testKinesis() {
    AmazonKinesis kinesis = DockerTestUtils.getClientKinesis();

    ListStreamsResult streams = kinesis.listStreams();
    ...

Or with JUnit 5 :

@ExtendWith(LocalstackDockerExtension.class)
@LocalstackDockerProperties(services = { "sqs", "kinesis:77077" })
public class MyDockerCloudAppTest {
   ...
}

The LocalStack JUnit test runner is published as an artifact in Maven Central. Simply add the following dependency to your pom.xml file:

<dependency>
    <groupId>cloud.localstack</groupId>
    <artifactId>localstack-utils</artifactId>
    <version>0.1.22</version>
</dependency>

You can configure the Docker behaviour using the @LocalstackDockerProperties annotation with the following parameters:

NB : When specifying the port in the services property, you cannot use randomizePorts = true

⁠Troubleshooting

• If you're using AWS Java libraries with Kinesis, please, refer to CBOR protocol issues with the Java SDK guide⁠ how to disable CBOR protocol which is not supported by kinesalite.

• Accessing local S3 from Java: To avoid domain name resolution issues, you need to enable path style access on your client:

s3.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build());
// There is also an option to do this if you're using any of the client builder classes:
AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard();
builder.withPathStyleAccessEnabled(true);
...

• Mounting the temp. directory: Note that on MacOS you may have to run TMPDIR=/private$TMPDIR docker-compose up if $TMPDIR contains a symbolic link that cannot be mounted by Docker. (See details here: https://bitbucket.org/atlassian/localstack/issues/40/getting-mounts-failed-on-docker-compose-up⁠)

• If you run into file permission issues on pip install under Mac OS (e.g., Permission denied: '/Library/Python/2.7/site-packages/six.py'), then you may have to re-install pip via Homebrew (see this discussion thread⁠). Alternatively, try installing with the --user flag: pip install --user localstack

• If you are deploying within OpenShift, please be aware: the pod must run as root, and the user must have capabilities added to the running pod, in order to allow Elasticsearch to be run as the non-root localstack user.

• The environment variable no_proxy is rewritten by LocalStack. (Internal requests will go straight via localhost, bypassing any proxy configuration).

• For troubleshooting LocalStack start issues, you can check debug logs by running DEBUG=1 localstack start

• In case you get errors related to node/nodejs, you may find (this issue comment: https://github.com/localstack/localstack/issues/227#issuecomment-319938530⁠) helpful.

• If you are using AWS Java libraries and need to disable SSL certificate checking, add -Dcom.amazonaws.sdk.disableCertChecking to the java invocation.

⁠Developing

If you pull the repo in order to extend/modify LocalStack, run this command to install all the dependencies:

make install

This will install the required pip dependencies in a local Python virtualenv directory .venv (your global python packages will remain untouched), as well as some node modules in ./localstack/node_modules/. Depending on your system, some pip/npm modules may require additional native libs installed.

The Makefile contains a target to conveniently run the local infrastructure for development:

make infra

Check out the developer guide⁠ which contains a few instructions on how to get started with developing (and debugging) features for LocalStack.

⁠Testing

The project contains a set of unit and integration tests that can be kicked off via a make target:

make test

⁠Web Dashboard

The projects also comes with a simple Web dashboard that allows to view the deployed AWS components and the relationship between them.

localstack web

⁠Other UI Clients

• Commandeer desktop app⁠
• DynamoDB Admin Web UI⁠

⁠Change Log

• v0.9.6: Add API Gateway SQS proxy; fix command to push Docker image; fix Docker bridge IP configuration; fix