scl3/task_base
Base image for tasks and Earth Engine tasks
1.3K
Base docker/python setup for running tasks, for use in calculating the Human Impact Index and Species Conservation Landscapes.
This repository contains the framework for use by all HII and SCL repos:
It is not meant to be run or instantiated directly. An example for how to use this repo can be found at
https://github.com/SpeciesConservationLandscapes/task_hii_popdens
Based on this example, to create a new task:
task_hii_
or task_scl_
task.py
and any other Python files needed.env
file that contains at minimum a SERVICE_ACCOUNT_KEY
entryTask
: base class defining taskdate
and other key properties - use for basic pipeline tasks not involving Earth
EngineEETask
: base Earth Engine task - sufficient for all non-species-specific EE tasksHIITask
: use for Human Impact Index-specific EE tasksSCLTask
: use for species-specific EE tasksTo run locally, copy into your root either:
a) [recommended] a .env file containing stringified GCP service account authentication details, or
b) a .config
dir containing your Earth Engine credentials file
Example commands below use task_hii_popdens
as an example inheriting from this repo.
Note that all tasks require a taskdate
argument, and other arguments may be available or required (not shown here);
for examples see existing task README
s.
Include the -v $PWD/.git:/app/.git
argument to write the SHA of the git commit
used to produce EE assets to their properties.
docker build --pull --no-cache -t scl3/task_hii_popdens .
docker run --rm -it --env-file ./.env -v $PWD/src:/app -v $PWD/.git:/app/.git scl3/task_hii_popdens python task.py
python task.py
from there:docker run -it --env-file ./.env -v $PWD/src:/app -v $PWD/.git:/app/.git scl3/task_hii_popdens bash
docker run -it -v $PWD/.config:/root/.config -v $PWD/src:/app -v $PWD/.git:/app/.git scl3/task_hii_popdens python task.py
Each task will, with appropriate credentials, run locally. To be available for other environments, particularly cloud deployment, create a docker repository and push to it or configure automated builds. The public Docker repos used as part of the HII/SCL project are available at https://cloud.docker.com/u/scl3/repository/list
The Github and Docker repositories referenced herein are used by the GCP pipeline (also open source) available at https://github.com/SpeciesConservationLandscapes/gcp-pipeline-devops With the appropriate GCP configuration, tasks based on this repository can be run serially and in parallel; see README above for details.
Copyright (C) 2022 Wildlife Conservation Society The files in this repository are part of the task framework for calculating Human Impact Index and Species Conservation Landscapes (https://github.com/SpeciesConservationLandscapes) and are released under the GPL license: https://www.gnu.org/licenses/#GPL See LICENSE for details.
docker pull scl3/task_base