skunkengineering/data-science-nginx repository overview

⁠Skunk Engineering data science tool box

The Skunk Engineering data science tool box is a collection of Docker (https://www.docker.com/⁠) images and Docker Compose (https://docs.docker.com/compose/⁠) files to develop and serve machine learning models using standard data science tools and frameworks such as Jupyter Notebook/Lab (https://jupyter.org/⁠) and MLflow (https://mlflow.org/⁠) on cloud service provider-agnostic containerized (and eventually orchestrated) deployment environments.

⁠Architecture(s)

The reference ML research and development system consists of:

• NGINX proxy server to provide a unified URL;
• Jupyter Lab to research and develop ML models;
• MLflow server to track projects, experiments, and models; and
• MinIO object storage to actually store artifacts.

There exist several configurations of Jupyter Lab:

• vanilla: based on https://hub.docker.com/r/jupyter/scipy-notebook⁠; and
• geo: based on vanilla above w/ the addition of GeoPandas (https://geopandas.org/⁠) and related packages.

The reference ML model serving system consists of:

• NGINX proxy server to provide unified URL for several ML models;
• Express.js-based router(s) to pre-process requests to, and post-process responses from, the ML model; and
• MLflow model server(s) to actually serve the model.

⁠Quickstart

The Docker images for the ML R&D system above are available on Docker Hub at https://hub.docker.com/u/skunkengineering⁠.

The Docker images for the ML model serving component will be made available in the near future.

In the meantime, one can try out the ML R&D system using the following Docker Compose file (change from vanilla to geo in order to launch Jupyer Lab w/ a GeoPandas support):

version: '3'
services:
  nginx-proxy-server:
    # UPGRADE: skunkengineering/data-science-nginx:latest
    image: 'skunkengineering/data-science-nginx:latest'
    # Always restart to be resilient to unresolved hostname in proxy pass
    # FIXME: Nginx won't re-resolve when other services are re-deployed at other IP addresses
    restart: always
    ports:
        - 80:80
  scipy-notebook-mlflow:
    # UPGRADE: skunkengineering/data-science-scipy-notebook-mlflow:vanilla-latest
    image: 'skunkengineering/data-science-scipy-notebook-mlflow:vanilla-latest'
    volumes:
      - '${PWD}:/home/jovyan/data-science-project'
    environment:
      MLFLOW_TRACKING_URI: 'http://mlflow:5000'
      JUPYTER_ENABLE_LAB: 'yes'
      MLFLOW_S3_ENDPOINT_URL: 'http://minio:9000'
      AWS_ACCESS_KEY_ID: 'minio'
      AWS_SECRET_ACCESS_KEY: 'minio123'
  mlflow:
    # UPGRADE: skunkengineering/data-science-mlflow-server:latest
    image: 'skunkengineering/data-science-mlflow-server:latest'
    volumes:
      - compose-scipy-vanilla-mlflow-server-data:/mlflow
    expose:
      - "5000"
    environment:
      MLFLOW_S3_ENDPOINT_URL: 'http://minio:9000'
      AWS_ACCESS_KEY_ID: 'minio'
      AWS_SECRET_ACCESS_KEY: 'minio123'
  # https://raw.githubusercontent.com/minio/minio/master/docs/orchestration/docker-compose/docker-compose.yaml
  minio:
    # UPGRADE: MinIO image RELEASE.2019-10-12T01-39-57Z
    image: 'minio/minio:RELEASE.2019-10-12T01-39-57Z'
    volumes:
      - compose-scipy-vanilla-minio-data:/data
    environment:
      MINIO_ACCESS_KEY: minio
      MINIO_SECRET_KEY: minio123
    command: server data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

volumes:
  compose-scipy-vanilla-mlflow-server-data:
  compose-scipy-vanilla-minio-data:

To get started on your local computer, follow these steps:

1. create a new working directory:

$ mkdir datascience

1. paste the Docker Compose YAML configuration into a new file docker-compose.yaml
2. start the ML R&D system:

$ docker-compose up -d

Upon confirming that the components are up and running w/ docker-compose ps, access the:

• Jupyter Lab at http://localhost/lab⁠ (lookup the access token in the service logs w/ docker-compose logs)
• MLflow server at http://localhost/mlflow⁠
• MinIO server at http://localhost/minio⁠

To save MLflow experiment artifacts, you must create a bucket in the MinIO server. The bucket must be called skunk-data-science-bucket.

You're all set!

Research and develop ML models leveraging Jupyter Lab and MLflow (https://mlflow.org/docs/latest/index.html⁠)

Once you're done, shut down the ML R&D system w/:

$ docker-compose down

The experiments and related artifacts are stored into the two Docker volumes listed in the Docker Compose file, and thus are available across deployments (up and down w/ docker-compose).

You can also readily deploy on a virtual machine (local or remote) using Docker Machine (https://github.com/docker/machine⁠).

For more info or feedback, please e-mail us at [email protected]⁠.