civisanalytics/datascience-python repository overview

⁠Data Science Docker Image

If you are reading this README on DockerHub, then the links to files in the GitHub repository will be broken. Please read this documentation from GitHub⁠ instead.

⁠Introduction

This repository defines the "civisanalytics/datascience-python⁠" Docker image. This Docker image provides an environment with data science tools from the Python ecosystem. This image is the execution environment for Python jobs in the Civis data science platform⁠, and it includes the Civis Python API client⁠.

⁠Installation

Either build the Docker image locally

docker build -t datascience-python .

or download the image from DockerHub

docker pull civisanalytics/datascience-python:latest

The latest tag (Docker's default if you don't specify a tag) will give you the most recently-built version of the datascience-python image. You can replace the tag latest with a version number such as 1.0 to retrieve a reproducible environment.

⁠Usage

Inside the datascience-python Docker image, Python packages are installed in the root environment. For a full list of included Python libraries, see the requirements-core.txt⁠ file.

To start a Docker container from the datascience-python image and interact with it from a bash prompt, use

docker run -i -t civisanalytics/datascience-python:latest /bin/bash

You can run a Python command with

docker run civisanalytics/datascience-python:latest python -c "import pandas; print(pandas.__version__)"

The image contains environment variables which allow you to find the current version. There are four environment variables defined:

VERSION
VERSION_MAJOR
VERSION_MINOR
VERSION_MICRO

VERSION contains the full version string, e.g., "1.0.3". VERSION_MAJOR, VERSION_MINOR, and VERSION_MICRO each contain a single integer.

⁠Joblib Temporary Files

The joblib⁠ library enhances multiprocessing capabilities for scientific Python computing. In particular, the scikit-learn library uses joblib for parallelization. This Docker image sets joblib's default location for staging temporary files to the /tmp directory. The normal default is /shm. /shm is a RAM disk which defaults to a 64 MB size in Docker containers, too small for typical scientific computing.

⁠Updating Existing Package Versions

1. Update versions of existing packages in requirements-core.txt
2. Run script generate-requirements-full.sh

⁠Creating Equivalent Local Environments

1. Create a new python environment python -m venv .venv.
2. Activate your new python environment source .venv/bin/activate
3. Install requirements.txt pip install -r requirements-full.txt

⁠Contributing

See CONTRIBUTING⁠ for information about contributing to this project.

If you make any changes, be sure to build a container to verify that it successfully completes:

docker build -t datascience-python:test .

and describe any changes in the change log⁠.

⁠For Maintainers

Publishing to DockerHub is handled by GitHub Actions (see .github/workflows⁠). Both publishing workflows build the image and run the test suite before they push anything.

There is no stored DockerHub password or access token. The workflows authenticate over OIDC: each run exchanges a GitHub identity token for a DockerHub token that expires within minutes. This requires two things a repository admin must set up once -- an OIDC connection on the civisanalytics DockerHub organization, and a DOCKERHUB_OIDC_CONNECTIONID repository variable holding that connection's ID (it is an identifier, not a secret). The publishing jobs also run in a dockerhub-publish environment restricted to master and v*.*.*.

Any PR merged to master is published as the latest tag on DockerHub.

To cut a new version:

1. Bump VERSION, VERSION_MAJOR, VERSION_MINOR, and VERSION_MICRO in the Dockerfile⁠, and move the Unreleased section of the change log⁠ under a [major.minor.micro] heading. Do this before tagging -- the release workflow refuses to publish if the Dockerfile's VERSION doesn't match the tag.
2. Once that is merged, go to the "releases" tab of the repository and click "Draft a new release". GitHub will prompt you to create a new tag, release title, and release description. The tag must use semantic versioning in the form "vX.X.X"; "major.minor.micro". The leading "v" is required -- tags without it won't trigger a build. The title of the release should be the same as the tag. Include a change log in the release description.
3. Publishing the release creates the tag, which builds and pushes three identical containers labelled "major", "major.minor", and "major.minor.micro".

⁠License

BSD-3

See LICENSE.md⁠ for details.