satapps/ard-workflow-s2 repository overview

⁠ARD Workflow Container for Sentinel-2 L2A Datasets

⁠Docker Hub images

Pre-built Docker images for production use can be pulled from our Docker Hub repo⁠.

⁠Dockerfile for development

The provided Dockerfile⁠ creates a Docker image with necessary packages for running an ARD workflow for Sentinel-2 L2A datasets, set up by means of Miniconda v4.7.10. Jupyter Notebook⁠ is included for interactive development and started once the Docker image is run.

⁠Docker Compose

A Docker Compose⁠ example file is provided to set up an interactive ARD workflow instance for development purposes.

⁠Environment variables for Docker Compose

Environment variables should be set in a .env file for Docker Compose. You might use .env.example⁠ as a starting point. The .gitignore⁠ file contains an entry for .env in order to avoid it from being accidentally added to this repository, so the .env file is suitable for storing sensitive information.

⁠Building and running a development platform

Set up an ARD workflow instance by issuing:

docker-compose up -d

Once the above completes, the job queue is ready to be filled in with work items by issuing:

docker exec -it redis-master /bin/bash
redis-cli -h redis-master
rpush jobS2 '{"in_scene": "S2A_MSIL2A_20190812T235741_N0213_R030_T56LRR_20190813T014708", "s3_bucket": "pds-satapps", "s3_dir": "fiji/Sentinel_2/"}'
...
lrange jobS2 0 -1

For mass insertion⁠ you can use e.g.:

docker exec -it redis-master /bin/bash
cat <<EOF | redis-cli -h redis-master --pipe
rpush jobS2 '{"in_scene": "S2A_MSIL2A_20190812T235741_N0213_R030_T56LRR_20190813T014708", "s3_bucket": "pds-satapps", "s3_dir": "fiji/Sentinel_2/"}'
...
EOF

At any time afterwards, the queue can be processed interactively by running the worker Jupyter Notebook.

⁠Jupyter Notebook

Jupyter Notebook can be accessed at the URL: http://{Serve's IP Address}:8877.
The access token is secretpassword, which is set by means of the CMD statement within the Dockerfile⁠.

⁠Amending the workflow

The actual workflow can be developed within the ard-workflows⁠ submodule at workflows directory.

⁠TODO

• Define the PLATFORM and QUEUE_NAME environment variables, so these can be set to SENTINEL_2 and jobS2 respectively, making the worker code agnostic of the satellite/platform to work on
• Define the LEASE_SECS and TIMEOUT environment variables, so these can be set according to what is appropriate for the satellite/platform to work on; alternatively read defaults from a configuration file that can be provided as an env_file in Docker Compose or as a ConfigMap in Kubernetes
• Generate a single Docker image: Jupyter Notebook could be optionally installed upon deployment, based on an environment variable, e.g. JUPYTER_NOTEBOOK set to YES; the main drawback of doing so (compared to building separate Docker images) is that dependencies might fail to support the installation of Jupyter Notebook
• Evaluate the use of RQ⁠, Celery⁠, or pyres⁠ for implementing a more resilient work queue