Public Repository

Last pushed: a day ago
Short Description
Short description is empty for this repo.
Full Description

air-tasks

DooD support and AWS ECR Credential Helper

How to develop:

By running DAGS:

  1. Clone this repo
  2. Install requirements
  3. Modify docker/docker-compose-CeleryExecutor.yml and uncomment dag folder mounts
     #- ../dags/:/usr/local/airflow/dags
    
  4. Deploy Local
  5. Go to localhost
  6. Activate dag and trigger run

By testing plugins / new operators / DockerFile / DAGS:

  1. Do above steps 1-3
  2. Build the image with your own tag (good idea to use the branch name)
     docker build -f docker/Dockerfile -t wongwill86/air-tasks:<your tag> .
    
  3. Modify docker/docker-compose-CeleryExecutor.yml to use your image
     <every service that has this>:
         image: wongwill86/air-tasks:<your tag>
    
  4. Run Tests

Debug tools:

localhost:80 - Airflow Webserver

localhost:81 - Celery Flower (Monitor Workers)

localhost:82 - Swarm Visualizer (Visualize Stack Deployment)

localhost:83 - RabbitMQ Management Plugin (Queue Info)

Setup:

  1. Install docker
     wget -qO- https://get.docker.com/ | sh
    
  2. Install docker compose
     pip install docker-compose
    

    Deploy

    Local

    docker-compose -f docker/docker-compose-CeleryExecutor.yml up -d
    

Swarm

echo '<blank or username here>' | docker secret create basic_auth_username -
echo '<blank or password here>' | docker secret create basic_auth_password -
docker stack deploy -c docker/docker-compose-CeleryExecutor.yml <stack name>

Testing

export PYTHONDONTWRITEBYTECODE=1 
export IMAGE_NAME=wongwill86/air-tasks:<your branch/tag>
docker-compose -f docker/docker-compose.test.yml -p ci build
docker-compose -f docker/docker-compose.test.yml -p ci run --rm sut

To watch/test. (Warning: if nothing runs, make sure all tests pass first)

docker-compose -f docker/docker-compose.test.yml -p ci run --rm sut ptw -- --pylama

AWS

Use Cloudformation to create a new stack.
Use this template

NOTES:

Chunkflow: make sure AWS_ACCESS_KEY_ID, etc... are set in environment variables!

Docker Pull Command
Owner
wongwill86

Comments (0)