fertiglab/fertigcompute repository overview

⁠Fertig Lab Compute Environments

Managing packages in Python and R can be a hassle when working across different projects and in different environments (Local Computer, AWS, MARCC, JHPCE, etc...). The goal of this tool is to remove this hassle and allow you to jump right in to the work you actually care about. Several pre-configured environments in Python and R are provided here with easy to follow set up instructions. These environments only have a single dependency, Docker⁠. Docker is a tool that lets us package up our Python and R environments so that no matter where they are being run, the exact same environment will be provided. This consistency allows us to reproduce both results and errors in our code. It also means you never have to deal with issues regarding package installation or errors that show up on one computer but not another.

There is an environment provided for Python 3.6, Python 3.7, R 3.5, and R 3.6. You can access these environments either through a Jupyter notebook or RStudio. It is recommended to use GitHub for recording all of your scripts/workflows and AWS for storing all your data. There is a projects directory on our AWS account, so if you make a directory in there with the same name as the project on GitHub, everything becomes easy to manage.

⁠Installing Dependencies

There is only one dependency, Docker⁠. Install Docker by following the instructions for your operating system (Mac⁠, Ubuntu⁠, Windows⁠). If you are running on an AWS server, most likely the OS is "Ubuntu". If you want to install Docker for the Windows Subsystem for Linux, follow these instructions⁠.

To check if Docker has been installed correctly and is running, type into your terminal:

docker run hello-world

You should see,

Hello from Docker!
This message shows that your installation appears to be working correctly.

⁠Connecting to a Server

Everything described in the following sections can be done either on a server or your own computer. One of the advantages of using this Docker container is that your analysis will run the same no matter where you are running it. If you want to run these commands on a server, first make sure that Docker is installed on the server, and second, make sure that you connect to the server with:

ssh -L 8888:127.0.0.1:8888 -L 8787:127.0.0.1:8787 user@host

This connects the 8888 (Jupyter) and 8787 (RStudio) ports between the server and your computer, allowing you to access the environments using a browser on your computer.

⁠Launching the Compute Environment

Once you are on the computer or server you want to use, and have successfully installed Docker, you are ready to start your analysis. Launch the Docker container with:

docker run \
    -p 8888:8888 \
    -p 8787:8787 \
    -v PATH/TO/YOUR/DIRECTORY:/home/fertigcompute \
    fertiglab/fertigcompute:latest

Make sure to replace PATH/TO/YOUR/DIRECTORY with the path to the directory you want to work in. Some reasonable defaults for each operating system are:

Linux (i.e. a server):

docker run \
    -p 8888:8888 \
    -p 8787:8787 \
    -v $HOME:/home/fertigcompute \
    fertiglab/fertigcompute:latest

Mac:

docker run \
    -p 8888:8888 \
    -p 8787:8787 \
    -v $HOME/Documents:/home/fertigcompute \
    fertiglab/fertigcompute:latest

Windows:

docker run \
    -p 8888:8888 \
    -p 8787:8787 \
    -v "%userprofile%\documents":/home/fertigcompute \
    fertiglab/fertigcompute:latest

⁠Using the Compute Environment

Once you launch the Docker container, you should see a message like:

RStudio started at 127.0.0.1:8787
Username: rstudio
Password: rstudio
Jupyter URL: http://127.0.0.1:8888/?token=fb10206f06291571498f50f64606979453a1669e326c58c9

In this example, to access the RStudio session, go to 127.0.0.1:8787 in your favorite browser. To access the Jupyter Notebook session, go to http://127.0.0.1:8888/?token=fb10206f06291571498f50f64606979453a1669e326c58c9 in your favorite browser.

If you have an error connecting to either RStudio and Jupyter once it's running, it is most likely due to an issue with how the ports were set up. When you are launching the Docker container, make sure to provide the -p 8787:8787 option for RStudio and -p 8888:8888 for Jupyter. If you are connecting to a server, make sure to provide the -L 8787:127.0.0.1:8787 option for RStudio and -L 8888:127.0.0.1:8888 for Jupyter.

⁠Docker Commands

The following commands are useful for managing your running Docker containers.

This will list all running containers
$ docker ps

This will stop a container with the specified ID
$ docker kill CONTAINER_ID

This will stop all running containers
$ docker kill $(docker ps -q)

In practice, it is often enough to just run docker kill $(docker ps -q) before launching a new container. If you ever see an error that says port already allocated, it is likely that there is already a fertigcompute Docker container running. In this case, you must kill this container before starting a new one. As long as your are mounting a local directory with the -v option, you never have to worry about losing data by shutting down a container.

⁠Configuring AWS Access

Often, you want to be able to access our AWS resources at some point in your analysis. You can always open up a terminal in the the Jupyter notebook GUI and run aws configure but it is also possible to just start the container with your AWS keys already present. This is done with the variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

docker run \
    -p 8888:8888 \
    -p 8787:8787 \
    -v $HOME:/home/fertigcompute \
    -e AWS_ACCESS_KEY_ID=XXXX \
    -e AWS_SECRET_ACCESS_KEY=XXXX \
    fertiglab/fertigcompute:latest

⁠Mounting Your S3 Home Directory

Instead of mounting a locally accessible directory, you can mount your S3 home directory by passing the S3_HOME_DIR argument. You need to set the AWS key variables for this to work. In general, it is not recommended to use this approach. One use case is when you are running on a server that might be shut down at unexpected times (i.e. an AWS Spot Instance). By mounting your S3 home directory, you can lose the server without losing your work.

docker run \
    -p 8888:8888 \
    -p 8787:8787 \
    -e S3_HOME_DIR=fertig-lab-bucket:/users/tom \
    -e AWS_ACCESS_KEY_ID=XXXX \
    -e AWS_SECRET_ACCESS_KEY=XXXX \
    --privileged \
    fertiglab/fertigcompute:latest