arvslat/lambda-env

By arvslat

•Updated over 7 years ago

Serverless Development and Deployment Environment for Python 3.6 and 3.7

Image

1.3K

Overview Tags

arvslat/lambda-env repository overview

⁠AWS Lambda Development and Deployment Environment for Intermediate Projects using Python 3.6 and 3.7

⁠Getting Started

The arvslat/lambda-env:python3.6 builds on top of lambci/lambda:build-python3.6 to allow rapid development and deployment of complex AWS Lambda functions which require package optimization and custom deployment steps. You'll most likely encounter different types of issues and blockers trying to build and develop your AWS Lambda functions once your application starts to involve more complex requirements and heavy libraries. This environment allows users to work inside an isolated container environment with several shortcuts and scripts as long as the required folder structure and steps are followed.

⁠Features

Project Initialization
Project Configuration
Environment Configuration Management (Development, Staging, Production)
Packaging and Deployment of Lambda Functions
Optimization of Packages
Testing and Debugging Lambda Functions Locally (i.e. IPython.embed)
Correction of Uploaded Lambda Package
Invocation of Deployed Lambda Functions
Debugging of CloudFormation Events (i.e. Lambda Deployment)

⁠Notes

Your current working directory is /var/task and the lambda.cli tool assumes that you are running it inside that directory
arvslat/lambda-env:python3.6 is an upgraded version of lambci/lambda:build-python3.6 that includes lambda.cli and other tools and scripts you'll need to develop, maintain, and deploy your AWS Lambda Functions
You can start with an empty project directory or you can continue working on an existing project or application generated with lambda.cli init
arvslat/lambda-env makes use of AWS CLI and AWS SAM internally to manage your resources and functions

⁠Steps (High Level)

Run Container and mount your directory
Activate Environment using source lambda-environment
Initialize Project using lambda.cli init
Configure AWS Credentials using aws configure
Configure Project Configuration and Environments using lambda.cli configure
Build Project and Install Dependencies using lambda.cli build
Package Project using lambda.cli package
Test Project using lambda.cli test
Deploy Project using lambda.cli deploy
Invoke Deployed Function using lambda.cli invoke
Debug Deployment using lambda.cli debug-deployment

⁠Details (Usage)

To get started with a new project or application, create an empty directory, mount that directory inside the container, and initialize with lambda.cli init

$ cd /Projects/sample-application-01
$ docker run --rm -it -v "$PWD":/var/task arvslat/lambda-env:python3.6 bash
bash-4.2# source lambda-environment
(venv) [07:47:31] /var/task # lambda.cli init

... (and so on) ...

⁠Folder Structure (after initialization)

.
├── app
│   └── index.py
├── bin
├── config
│   ├── env.development.yml
│   ├── env.production.yml
│   ├── env.staging.yml
│   ├── requirements.txt
│   └── sam.template.yml
├── README.md
├── scripts
│   ├── install.sh
│   └── optimize.sh
├── tests
│   └── events
├── tmp
└── venv
    ├── bin
    ├── include
    ├── lib
    ├── lib64 -> lib
    └── pyvenv.cfg

⁠Run Environment

$ docker run --rm -it -v "$PWD":/var/task arvslat/lambda-env:python3.6 bash

# Example: Memory = 4GB
$ docker run -m=4g --rm -it -v "$PWD":/var/task arvslat/lambda-env:python3.6 bash

⁠Manage Lambda Development and Deployment Package (Python 3.6 / 3.7)

⁠Using existing shortcuts and scripts inside the container [Version 1]

bash-4.2# source lambda-environment
(venv) [00:00:00] /var/task # lambda-install-packages
(venv) [00:00:00] /var/task # lambda-check-package-size
(venv) [00:00:00] /var/task # lambda-optimize-package-size
(venv) [00:00:00] /var/task # lambda-check-package-size
(venv) [00:00:00] /var/task # lambda-generate-zip
(venv) [00:00:00] /var/task # lambda-check-package-size
(venv) [00:00:00] /var/task # aws configure
(venv) [00:00:00] /var/task # sam package --template-file config/sam.template.yml  --output-template-file tmp/sam.package.yml   --s3-bucket <bucket>
(venv) [00:00:00] /var/task # aws s3 cp /tmp/app.zip `lambda-get-s3-package-location`
(venv) [00:00:00] /var/task # sam deploy --template-file tmp/sam.package.yml --stack-name <stack name> --capabilities CAPABILITY_IAM

⁠Using the `lambda.cli` [Version 2 - Preferred]

bash-4.2# source lambda-environment
(venv) [00:00:00] /var/task # aws configure
(venv) [00:00:00] /var/task # lambda.cli init
(venv) [00:00:00] /var/task # lambda.cli configure
(venv) [00:00:00] /var/task # lambda.cli build
(venv) [00:00:00] /var/task # lambda.cli test --event event
(venv) [00:00:00] /var/task # lambda.cli package
(venv) [00:00:00] /var/task # lambda.cli deploy
(venv) [00:00:00] /var/task # lambda.cli check-deployment
(venv) [00:00:00] /var/task # lambda.cli invoke

⁠Sample Output of `lambda.cli init`

CURRENT_WORKING_DIRECTORY = /var/task

RUNNING INSIDE VENV = True

COMMAND = init

CONTEXT, PARAMETERS, and DEFAULT VALUES

{
    "app_name": "hello_world",
    "command": "init",
    "environment": "development",
    "test_event": "event",
    "version": false
}

EXECUTE COMMAND [init]

[+] Creating directory [bin]
[+] Creating directory [tests]
[+] Creating directory [tests/events]
[+] Creating directory [config]
[+] Creating directory [app]
[+] Creating directory [tmp]
[+] Creating directory [scripts]
[+] Creating empty file [README.md]
[+] Creating empty file [config/requirements.txt]
[+] Creating empty file [scripts/install.sh]
[+] Creating empty file [scripts/optimize.sh]
[+] Creating empty file [tmp/.gitkeep]
[+] Creating empty file [bin/.gitkeep]
[+] Creating empty file [tests/events/.gitkeep]
[+] Loading contents of [/root/lambda_cli/templates/env.development.yml]: START
[+] Loading contents of [/root/lambda_cli/templates/env.development.yml]: END
[+] Writing contents to [config/env.development.yml]: START
[+] Writing contents to [config/env.development.yml]: DONE
[+] Loading contents of [/root/lambda_cli/templates/event.json]: START
[+] Loading contents of [/root/lambda_cli/templates/event.json]: END
[+] Writing contents to [tests/events/event.json]: START
[+] Writing contents to [tests/events/event.json]: DONE
[+] Loading contents of [/root/lambda_cli/templates/env.staging.yml]: START
[+] Loading contents of [/root/lambda_cli/templates/env.staging.yml]: END
[+] Writing contents to [config/env.staging.yml]: START
[+] Writing contents to [config/env.staging.yml]: DONE
[+] Loading contents of [/root/lambda_cli/templates/env.production.yml]: START
[+] Loading contents of [/root/lambda_cli/templates/env.production.yml]: END
[+] Writing contents to [config/env.production.yml]: START
[+] Writing contents to [config/env.production.yml]: DONE
[+] Loading contents of [/root/lambda_cli/templates/sam.template.yml]: START
[+] Loading contents of [/root/lambda_cli/templates/sam.template.yml]: END
[+] Writing contents to [config/sam.template.yml]: START
[+] Writing contents to [config/sam.template.yml]: DONE
[+] Loading contents of [/root/lambda_cli/templates/.gitignore]: START
[+] Loading contents of [/root/lambda_cli/templates/.gitignore]: END
[+] Writing contents to [.gitignore]: START
[+] Writing contents to [.gitignore]: DONE
[+] Loading contents of [/root/lambda_cli/templates/index.py]: START
[+] Loading contents of [/root/lambda_cli/templates/index.py]: END
[+] Writing contents to [app/index.py]: START
[+] Writing contents to [app/index.py]: DONE

TREE of /var/task

[+] Executing [tree -L 2]

.
├── app
│   └── index.py
├── bin
├── config
│   ├── env.development.yml
│   ├── env.production.yml
│   ├── env.staging.yml
│   ├── requirements.txt
│   └── sam.template.yml
├── README.md
├── scripts
│   ├── install.sh
│   └── optimize.sh
├── tests
│   └── events
├── tmp
└── venv
    ├── bin
    ├── include
    ├── lib
    ├── lib64 -> lib
    └── pyvenv.cfg

⁠Manage Config: `env.development.yml`

NOTE: Update this file with your custom configuration details. Run lambda.cli configure to check if your configuration is valid.

resources:
  function:
    label: 'TargetFunction'
    name: '<Unique Lambda Function Name per Region>'

app:
  stack_name: '<Unique CloudFormation Stack Name per Region>'
  region: '<us-east-1 / us-east-2 / ...>'
  s3_bucket: '<S3 Bucket name located in the same region of the Lambda Function>'

⁠Sample output of `lambda.cli configure`

(venv) [07:48:18] /var/task # lambda.cli configure
CURRENT_WORKING_DIRECTORY = /var/task

RUNNING INSIDE VENV = True

COMMAND = configure

CONTEXT, PARAMETERS, and DEFAULT VALUES

{
    "app_name": "hello_world",
    "command": "configure",
    "environment": "development",
    "test_event": "event",
    "version": false
}

EXECUTE COMMAND [configure]

CONFIGURATION CHECKS [config/env.development.yml]

[+] resources.function.label = TargetFunction
[+] resources.function.name = TargetFunctionDevelopmentZZ
[+] app.stack_name = lambda-env-stack-dev-01
[+] app.region = us-east-1
[+] app.s3_bucket = sample-sam-bucket-12345

GENERATE SAM CONFIG FILE

[+] Loading contents of [config/sam.template.yml]: START
[+] Loading contents of [config/sam.template.yml]: END
[+] Writing contents to [tmp/sam.development.yml]: START
[+] Writing contents to [tmp/sam.development.yml]: DONE
[+] Generated [tmp/sam.development.yml]

COMPLETED EXECUTION [configure]

⁠Sample Execution of `lambda.cli build`

CURRENT_WORKING_DIRECTORY = /var/task

RUNNING INSIDE VENV = True

COMMAND = build

CONTEXT, PARAMETERS, and DEFAULT VALUES

{
    "app_name": "hello_world",
    "command": "build",
    "environment": "development",
    "test_event": "event",
    "version": false
}

EXECUTE COMMAND [build]

CONFIGURATION CHECKS [config/env.development.yml]

[+] resources.function.label = TargetFunction
[+] resources.function.name = TargetFunctionDevelopmentZZ
[+] app.stack_name = lambda-env-stack-dev-01
[+] app.region = us-east-1
[+] app.s3_bucket = sample-sam-bucket-12345

GENERATE SAM CONFIG FILE

[+] Loading contents of [config/sam.template.yml]: START
[+] Loading contents of [config/sam.template.yml]: END
[+] Writing contents to [tmp/sam.development.yml]: START
[+] Writing contents to [tmp/sam.development.yml]: DONE
[+] Generated [tmp/sam.development.yml]

[+] Executing [lambda-install-packages]

---------------------------------------------------
[+] 'scripts/install.sh' exists. Executing specified installation command(s)
---------------------------------------------------
docutils==0.14
jmespath==0.9.3
python-dateutil==2.7.3
six==1.11.0
---------------------------------------------------

[+] Completed [lambda-install-packages]

[+] Executing [lambda-optimize-package-size]

---------------------------------------------------
[+] find "$VIRTUAL_ENV/lib/python3.6/site-packages/" -name "*.so" | xargs strip
[-] Nothing to strip
[+] "$VIRTUAL_ENV/lib/python3.6/site-packages" -name "test" | xargs rm -rf
[+] "$VIRTUAL_ENV/lib/python3.6/site-packages" -name "tests" | xargs rm -rf
---------------------------------------------------
[+] 'scripts/optimize.sh' exists. Executing additional optimization command(s)
---------------------------------------------------

[+] Completed [lambda-optimize-package-size]

COMPLETED EXECUTION [build]

⁠Sample Output of `lambda.cli package`

NOTE: Run aws configure first

(venv) [07:53:25] /var/task # lambda.cli package
CURRENT_WORKING_DIRECTORY = /var/task

RUNNING INSIDE VENV = True

COMMAND = package

CONTEXT, PARAMETERS, and DEFAULT VALUES

{
    "app_name": "hello_world",
    "command": "package",
    "environment": "development",
    "test_event": "event",
    "version": false
}

EXECUTE COMMAND [package]

CONFIGURATION CHECKS [config/env.development.yml]

[+] resources.function.label = TargetFunction
[+] resources.function.name = TargetFunctionDevelopmentZZ
[+] app.stack_name = lambda-env-stack-dev-01
[+] app.region = us-east-1
[+] app.s3_bucket = sample-sam-bucket-12345

GENERATE SAM CONFIG FILE

[+] Loading contents of [config/sam.template.yml]: START
[+] Loading contents of [config/sam.template.yml]: END
[+] Writing contents to [tmp/sam.development.yml]: START
[+] Writing contents to [tmp/sam.development.yml]: DONE
[+] Generated [tmp/sam.development.yml]

[+] Executing [sam package --template-file tmp/sam.development.yml --output-template-file tmp/sam.development.packaged.yml --s3-bucket sample-sam-bucket-12345]


Uploading to d8b9d3c745f8e37af4512ffe3e4c91d4  236 / 236.0  (100.00%)
Successfully packaged artifacts and wrote output template to file tmp/sam.development.packaged.yml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file /var/task/tmp/sam.development.packaged.yml --stack-name <YOUR STACK NAME>

[+] Completed [sam package --template-file tmp/sam.development.yml --output-template-file tmp/sam.development.packaged.yml --s3-bucket sample-sam-bucket-12345]

[+] Executing [lambda-generate-zip]

---------------------------------------------------
[-] '/tmp/app.zip' does not exist
---------------------------------------------------
[+] pushd /var/task/venv/lib/python3.6/site-packages/
/var/task/venv/lib/python3.6/site-packages /var/task
[+] zip -r -9 -q /tmp/app.zip * --exclude venv
[+] popd
/var/task
[+] cd /var/task/app
[+] zip -9 $FILE * --exclude "venv/" *.zip
  adding: index.py (deflated 13%)
---------------------------------------------------
[+] '/tmp/app.zip' generated
---------------------------------------------------

[+] Completed [lambda-generate-zip]

[+] Executing [lambda-check-package-size]

---------------------------------------------------
[+] Uncompressed (./venv/lib/python3.6/site-packages): ~ 12M
---------------------------------------------------
[+] Compressed (/tmp/app.zip): ~ 3.6M
---------------------------------------------------

[+] Completed [lambda-check-package-size]

[+] Executing [aws s3 cp /tmp/app.zip s3://sample-sam-bucket-12345/d8b9d3c745f8e37af4512ffe3e4c91d4]

Completed 256.0 KiB/3.5 MiB (143.1 KiB/s) with 1 file(s) remaining
Completed 512.0 KiB/3.5 MiB (72.7 KiB/s) with 1 file(s) remaining 
Completed 768.0 KiB/3.5 MiB (82.1 KiB/s) with 1 file(s) remaining 
Completed 1.0 MiB/3.5 MiB (89.1 KiB/s) with 1 file(s) remaining   
Completed 1.2 MiB/3.5 MiB (94.3 KiB/s) with 1 file(s) remaining   
Completed 1.5 MiB/3.5 MiB (98.9 KiB/s) with 1 file(s) remaining   
Completed 1.8 MiB/3.5 MiB (101.4 KiB/s) with 1 file(s) remaining  
Completed 2.0 MiB/3.5 MiB (103.1 KiB/s) with 1 file(s) remaining  
Completed 2.2 MiB/3.5 MiB (105.2 KiB/s) with 1 file(s) remaining  
Completed 2.5 MiB/3.5 MiB (106.2 KiB/s) with 1 file(s) remaining  
Completed 2.8 MiB/3.5 MiB (107.2 KiB/s) with 1 file(s) remaining  
Completed 3.0 MiB/3.5 MiB (107.6 KiB/s) with 1 file(s) remaining  
Completed 3.2 MiB/3.5 MiB (108.4 KiB/s) with 1 file(s) remaining  
Completed 3.5 MiB/3.5 MiB (109.0 KiB/s) with 1 file(s) remaining  
Completed 3.5 MiB/3.5 MiB (99.1 KiB/s) with 1 file(s) remaining   
upload: ../../tmp/app.zip to s3://sample-sam-bucket-12345/d8b9d3c745f8e37af4512ffe3e4c91d4

[+] Completed [aws s3 cp /tmp/app.zip s3://sample-sam-bucket-12345/d8b9d3c745f8e37af4512ffe3e4c91d4]

COMPLETED EXECUTION [package]

⁠Lambda CLI Common Commands

Lambda CLI 1.0 Usage

$ lambda.cli init

[Development]
$ aws configure
$ lambda.cli configure
$ lambda.cli build
$ lambda.cli test --event event01
$ lambda.cli package
$ lambda.cli deploy
$ lambda.cli check-deployment
$ lambda.cli invoke --event event01

[Staging]
$ lambda.cli configure --staging
$ lambda.cli build --staging
$ lambda.cli invoke --staging
$ lambda.cli package --staging
$ lambda.cli deploy --staging
$ lambda.cli deploy --staging

[Production]
$ lambda.cli configure --production
$ lambda.cli build --production
$ lambda.cli package --production
$ lambda.cli deploy --production
$ lambda.cli invoke --production --event event01

⁠Lambda CLI Notes

/root/lambda_cli
├── __init__.py
├── build.py
├── check_deployment.py
├── command_line.py
├── config_utils.py
├── configure.py
├── deploy.py
├── display_help.py
├── initialize.py
├── invoke.py
├── lambda.cli
├── package.py
├── templates
│   ├── env.development.yml
│   ├── env.production.yml
│   ├── env.staging.yml
│   ├── event.json
│   ├── index.py
│   └── sam.template.yml
└── test.py

NOTE: /usr/bin/lambda.cli is symlinked to /root/lambda_cli/lambda.cli

⁠Template: `sam.template.yml`

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  Description

Globals:
  Function:
    Timeout: 30
    MemorySize: 1024

Resources:
  {{ resources.function.label }}:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ../app/
      Handler: index.handler
      Runtime: python3.6
      FunctionName: {{ resources.function.name }}

⁠Template: `env.development.yml`

resources:
  function:
    label: 'TargetFunction'
    name: ''

app:
  stack_name: ''
  region: ''
  s3_bucket: ''

⁠Template: `index.py`

def handler(event, context):
    print('> This is a sample function generated by LAMBDA CLI')
    print(f"> EVENT = {event}")

    return {}

⁠Sample Application and Configuration Files

⁠`app/index.py`


import pandas as pd
import os 

from fbprophet import Prophet

def absolute_path(filename):
    dir_path = os.path.dirname(os.path.realpath(__file__))
    output = os.path.join(dir_path, filename)

    return output

def handler(event, context):
    s = pd.Series([1, 3, 5, 7, 6, 8])
    print(s)

    df = pd.read_csv(absolute_path('data.csv'))

    print(df.head())

    m = Prophet()
    m.fit(df)

    future = m.make_future_dataframe(periods=365)
    print(future.tail())

    forecast = m.predict(future.tail(26))
    print(forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']].tail())

if __name__ == "__main__":
    handler(event={}, context={})

⁠`app/data.csv`

Sample data.csv can be obtained here⁠

"ds","y"
"2007-12-10",9.59076113897809
"2007-12-11",8.51959031601596
"2007-12-12",8.18367658262066
"2007-12-13",8.07246736935477
"2007-12-14",7.8935720735049

. . .

⁠`config/sam.template.yml`

AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  TargetFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ../app
      Handler: index.handler
      Runtime: python3.6
      MemorySize: 2048
      Timeout: 30

⁠`scripts/install.sh`

#!/usr/bin/env bash

pip install pandas==0.23.4 --no-cache --verbose
pip install pyparsing==2.3.0 --no-cache --verbose
pip install pystan==2.17 --no-cache --verbose
pip install fbprophet==0.3.post2 --no-cache --verbose

⁠`scripts/optimize.sh`

#!/usr/bin/env bash

pip uninstall -y matplotlib
rm -rf "$VIRTUAL_ENV/lib/python3.6/site-packages/pystan/stan/src"
rm -rf "$VIRTUAL_ENV/lib/python3.6/site-packages/pystan/stan/lib/stan_math/lib"

⁠Script Details

⁠lambda-environment

The lambda-environment script initializes the virtual environment and activates the virtual environment prepared. The virtual environment directory is created in the directory mounted in the container.

⁠lambda-install-packages

The lambda-install-packages script looks for the /var/tasks/scripts/install.sh provided by the user and executes that script if it exists. After the installation has completed, pip freeze is executed to allow the user to check the installed Python packages. This script requires the lambda-environment script executed first

This is an example of /var/tasks/scripts/install.sh script that involves relatively trickier installations (i.e. Facebook Prophet) which may not work right away using a requirements.txt installation approach.

#!/usr/bin/env bash

pip install pandas==0.23.4 --no-cache --verbose
pip install pyparsing==2.3.0 --no-cache --verbose
pip install pystan==2.17 --no-cache --verbose
pip install fbprophet==0.3.post2 --no-cache --verbose

⁠lambda-optimize-package-size

The lambda-optimize-package-size script performs two things: (1) default optimization and (2) custom optimization specified by the user. This script looks for /var/task/scripts/optimize.sh and executes it if it exists. This script is usually performed before the application is packaged and proves to be useful when dealing with packages (i.e. machine learning and data engineering packages) and files that increase the file size of the zip file for the AWS Lambda Function.

This is an example /var/task/scripts/optimize.sh script that reduces the size of the uncompressed package from 300-500MB+ to ~ 130MB for AWS Lambda functions that use pystan and fbprophet.

#!/usr/bin/env bash

pip uninstall -y matplotlib
rm -rf "$VIRTUAL_ENV/lib/python3.6/site-packages/pystan/stan/src"
rm -rf "$VIRTUAL_ENV/lib/python3.6/site-packages/pystan/stan/lib/stan_math/lib"

⁠lambda-generate-zip

The lambda-generate-zip script generates a zip file from the packages installed in the virtual environment (/var/task/venv/lib/python3.6/site-packages/) and the files in the working directory (/var/task). The generated zip file can be found in the /tmp directory: /tmp/app.zip. This is usually executed after the lambda-optimize-package-size script to reduce the size of the package significantly

⁠`lambda-check-package-size`

The lambda-check-package-size simply calculates the total size of the uncompressed and compressed package to allow the user to perform additional optimization work in case the total size exceeds the AWS Lambda limit.

⁠lambda-get-s3-package-location

The lambda-get-s3-package-location script simply extracts the Amazon S3 Zip file location from the SAM package YAML file. The uploaded ZIP file will be replaced by the generated zip file after using the lambda-generate-zip script. The AWS S3 CLI upload will be used to replace the zip file in the target Amazon S3 bucket

⁠Additional Notes

⁠Developer Utilities Included

vim
tmux
tree
IPython (embed)

Tag summary

Recent tags

Content type

Image

Digest

Size

737.5 MB

Last updated

over 7 years ago

docker pull arvslat/lambda-env