ddmal/postgres-plpython repository overview

⁠rodan-docker

This repository contains Docker images that can be used to set up Rodan⁠ locally for development. These images can also be used in the future with slight modifications for deployment to a production environment.

⁠Objectives

• Simplify the installation process of Rodan on multiple platforms.
• Maintain clear installation documentation.

⁠Installation

First, download and install Docker⁠ for your platform. If Docker is not supported on your platform (e.g. macOS older than 10.10.3, Windows older than 10 Pro, or unsupported hardware), use Docker Toolbox⁠ instead. If you're using Docker Toolbox, see specific instructions below.

After Docker is installed, clone this repository on your computer and run the following command:

docker login

Enter the DDMAL credentials for Docker Hub (contact the lab manager if you don't know them) to save them on your computer. You only need to do this once.

To run Rodan, run the following command in the repository's main directory:

docker-compose up

That's it! Docker will automatically spawn all the containers needed to run Rodan, rodan-client and their dependencies. Point your browser to https://localhost:9002⁠ to view the Rodan web interface. The username is rodan with the password rodan. The Rodan REST API is accessible at https://localhost:8000⁠. If needed, you may observe the log output of the Rodan containers in the terminal window where docker-compose is running. When you are finished, quit docker-compose using Ctrl+C.

Note: if you're using Docker Toolbox instead of Docker, see the "Additional notes for Docker Toolbox users" section below.

⁠How does it work?

A Docker container image⁠ is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings. A container is different from a virtual machine: it is more lightweight and portable because it virtualizes the operating system instead of hardware.

Rodan's Docker setup is comprised of 5 container images:

1. rodan - the Rodan server providing a REST API.
2. rodan-client - a web interface for Rodan.
3. postgres-plpython - the PostgreSQL database used by the Rodan server, with the Python extensions installed.
4. redis - a key-value database used by the Rodan server to manage websocket connections. TODO: Do we actually use this? (matangover)
5. rabbitmq - a message queue used as a backend for celery, a Python library used by Rodan for job management.

rodan, rodan-client and postgres-plpython each have a Dockerfile in this repository. For redis and rabbitmq we use existing public images from the Docker Hub. The docker-compose.yml file specifies the dependencies between the container images and combines them to set a complete Rodan installation.

rodan-client and postgres-plpython are public repositories on GitHub and Docker Hub, but rodan-docker on GitHub and rodan on Docker Hub are private. They must be private because they contain Kakadu, a proprietary software package that is licensed to DDMAL. In the future we might create a separate public version of the rodan image without Kakadu.

⁠Setting up a Development Environment

If you wish to work on Rodan or rodan-client and test your changes locally, you have a few options. The docker-compose setup is modular, so you could choose which parts of it you want to take and which you want to override with your local changes.

⁠Developing a Rodan job

If you wish to create a new type of job for Rodan, follow these steps.

1. Put your job code in a new folder inside the jobs folder under the repository root. For example, jobs/demojob contains the code for a demo job that outputs "Hello, world!".
2. Copy jobs/settings.py.job_development to jobs/settings.py.
3. Open jobs/settings.py in a text editor and replace demojob with the name of your job (line 105).
4. Open docker-compose.job-dev.yml in a text editor and replace both occurrences of demojob with your job's name.
5. Run the following command:

docker-compose -f docker-compose.yml -f docker-compose.job-dev.yml up

Your job is now available in Rodan. Any changes you make in the job code will automatically be reflected inside the container. Steps 1-4 only have to be completed once. Step 5 can be run at any time to launch the Rodan container with your job in it.

If you make any changes to the job schema (for example, adding a new content type or changing the input/output port definitions), you will need to stop docker-compose (Ctrl+C) and re-run it (with the command from step 5).

⁠Working on rodan-client

If you wish to work on rodan-client without modifying Rodan, you may run rodan-client locally and have it connect to the Rodan server run by docker-compose. rodan-client doesn't have many dependencies so it's simple to set up that way.

1. Clone rodan-client (or use the submodule in rodan-docker) and follow the normal setup instructions⁠.
2. Copy configuration.json from rodan-docker/rodan-client/config to your own clone of rodan-client. If you want to use your own configuration, make sure it's connected to the Rodan server launched by docker-compose:

{
  "SERVER_HOST": "localhost",
  "SERVER_PORT": 8000
}

1. Run docker-compose up rodan (instead of just docker-compose up) to run Rodan and its dependencies without running rodan-client.
2. Run gulp as instructed in the Deploy⁠ step of the rodan-client setup instructions.

⁠Working on Rodan

You could set up Rodan and its dependencies locally (like described for rodan-client above), but since Rodan has many dependencies that could be a pain to install, a better option would be to clone the Rodan code locally and mount it into the Docker container that already has all the dependencies installed. To run the Rodan stack with the local development version of Rodan:

1. Make sure Rodan is cloned into rodan-docker/rodan/code/rodan - see Git Submodule Setup below. (You can also use a Rodan clone from a different path, with minor adjustments to the following steps.)

2. Copy rodan-docker/rodan/code/settings.py.development to rodan-docker/rodan/code/rodan/rodan/settings.py. Note that this settings file does not include any job types other than the default. To enable more job types you must install the jobs into rodan-docker/rodan/code/rodan/rodan/jobs and add them to RODAN_JOB_PACKAGES in settings.py.

3. Run docker-compose -f docker-compose.yml -f docker-compose.rodan-dev.yml up

When running docker-compose, Docker will mount your local version of the code into the container, overwriting the version of Rodan that's included in the image.

For more information about volumes in Docker, see Use volumes⁠ in the Docker documentation. See also the docs for the volumes section⁠ of the docker-compose.yml file.

⁠Tips for Interacting with Running Containers

Sometimes it can be useful to inspect files or processes running inside containers. To get a list of running containers, run docker ps. To copy files between the container and the host, use docker cp.

To run any command on a running container, use docker exec. For example, to run bash on the Rodan container launched by docked-compose, run:

docker exec -it rodandocker_rodan_1 bash

Consult the documentation of the Docker command line⁠ for additional information.

⁠Git Submodule Setup

This repository uses the Git submodules mechanism to include all the libraries needed to set up Rodan. Git submodules are essentially pointers to other Git repositories, and are specified in the .gitmodules file.

Cloning the submodules is necessary only if you wish to rebuild the Docker images. When using graphical Git clients such as GitHub Desktop or SourceTree, the submodules are usually automatically cloned for you. When cloning on the command line, run the following command after git clone to clone the submodules as well:

git submodule update --init

When you update an upstream repository, e.g. Rodan or rodan-client, the changes will not be reflected in this repository (and hence, in the Docker images) until you pull the changes in. In order to pull a new version of Rodan, for example, run the following commands:

# Pull the submodule changes. The branch name must be specified explicitly.
cd rodan/code/rodan
git pull origin master
# Commit and push the changes to the parent repository (docker-rodan).
cd ../..
git add rodan/code/rodan
git commit -m "Update the rodan submodule"
git push

⁠Rebuilding the Docker images

If you change Rodan or rodan-client, or wish to change something in the way they are installed, you may build new versions of the Docker images, use them locally and eventually publish them. To build a new version of the rodan image, for example, run the following commands from the repository's main directory:

docker build -t ddmal/rodan rodan

The build operation will take into account any changes you have made to rodan/Dockerfile or to any files it depends on. Depending on your changes, the build might take a long time to complete.

After an image is built, you may create a container using that image using docker run rodan. Refer to Docker's Get Started⁠ documentation for a more complete guide.

When you're happy with an image and want to publish it to the Docker Hub, run docker push, e.g. docker push ddmal/rodan.

⁠Automated Build

Docker Cloud can automatically build new Docker images when new commits are pushed to a Git repository. Unfortunately, we had problems connecting the rodan-docker GitHub repository to Docker Cloud due to authentication issues, so we set up a private repository on Bitbucket instead.

Every time a new commit is pushed to the Bitbucket repository on the master branch, Docker Cloud will build the Rodan Docker images automatically. To push to the Bitbucket repo, first add it as a remote to your local clone:

git remote add bitbucket https://[email protected]/ddmaldocker/rodan-docker.git

(The password for ddmaldocker on Bitbucket is the same as the password for ddmal on Docker Hub.)

After you're finished committing your changes to the master branch, push to the GitHub remote (origin) as usual, and then also push to to the bitbucket remote:

git push bitbucket master

This will trigger an automated build of the rodan, rodan-client and postgres-plpython images on the Docker Cloud servers. You may monitor the build results on Docker Cloud⁠ using the ddmal account credentials.

⁠Additional configuration for Docker Toolbox users

Docker Toolbox doesn't expose container ports on localhost, it exposes them on the Docker host's IP address instead, which you can get using the following command:

 docker-machine ip default

In order to get the Rodan-Client to find Rodan without running Rodan-Client outside of docker, you must add an entry to your hosts file redirecting localhost to the Docker host's IP address. The hosts file can be found at /etc/hosts on macOS and c:\Windows\System32\Drivers\etc\hosts on Windows. Add the following entry, replacing the IP address with the output from the docker-machine ip default command:

192.168.xxx.xxx     localhost

When you want rodan-client to be run from outside the docker, remember to comment out the line in the host file.