Sign inSign up

agladstein/data_science_popgen_notebook

By agladstein

•Updated over 6 years ago

Data science applications for population genetics

Image
0

100K+

agladstein/data_science_popgen_notebook repository overview

⁠Dockerfile:
FROM jupyter/datascience-notebook:latest

#maintainer "Ariella Gladstein <[email protected]>"
#organization "University of North Carolina at Chapel Hill"
#department "Genetics"
#date "01 September 2019"
#application "Data science applications for population genetics"

# Switch to root user
USER root

RUN export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true && \
    apt-get update && apt-get install -y \
        curl \
        ssh

# Install tskit
RUN conda install -c conda-forge --quiet --yes tskit=0.2.2

# Install msprime
RUN conda install --quiet --yes 'msprime'=0.7.3

# Install scikit-allel
RUN conda install -c conda-forge --quiet --yes 'scikit-allel'

# Install R packages
RUN conda install --quiet --yes 'simplegeneric'
RUN wget https://cran.r-project.org/src/contrib/POPdemog_1.0.3.tar.gz
RUN R CMD INSTALL POPdemog_1.0.3.tar.gz

# Install pylibseq
RUN conda config --add channels bioconda
RUN conda install -c bioconda pylibseq=0.2.2

# Install Pweave
RUN conda install pweave -c conda-forge --quiet --yes

# instal fileStats
ADD src/data/filet_stats /tmp/filet_stats
RUN ls -l /tmp && \
    cd /tmp/filet_stats && \
    which python && \
    python setup.py install

# install dadi
RUN git clone https://bitbucket.org/gutenkunstlab/dadi.git && \
    cd dadi && \
    python setup.py build_ext --inplace && \
    PYTHONPATH="${PYTHONPATH}:/home/jovyan/dadi" && \
    export PYTHONPATH

# Switch back from root
USER $NB_UID

# build info
#RUN echo "Timestamp:" `date --utc` | tee /image-build-info.txt
⁠How to use the container
⁠How to run with Docker
⁠Pull the image

(You can actually skip this step. When you run the container with the command below, if the image isn't present, Docker will automatically pull it.)

docker pull agladstein/data_science_popgen_notebook:<tagname>

where <tagname> is a tag (e.g. 1.0.1). If you leave off :<tagname>, Docker will use the latest version.

⁠Enter the container with bash
docker run -it -v <host_path>:/home/jovyan/work --rm -p 8888:8888 agladstein/data_science_popgen_notebook:<tagname> /bin/bash

where <host_path>, is the path you'd like to mount to container (e.g. my home /home/aglad). Note, that in this case the port 8888 must be accessible on your host machine.

Once in the container, run any of the desired applications.

⁠or, immediately start a Jupyter Lab container
docker run -it -v <host_path>:/home/jovyan/work --rm -p 8888:8888 agladstein/data_science_popgen_notebook:<tagname> jupyter-lab
⁠How to run with Singularity

Most HPC's have Singularity (https://www.sylabs.io/⁠), and if they don't, they should be willing to install it (Singularity is intended for HPC's).

⁠Pull the image
singularity pull docker://agladstein/data_science_popgen_notebook:<tagname>

This creates a local image in the singularity file format.

⁠Enter the container
singularity shell data_science_popgen_notebook_<tagname>.simg

Singularity will automatically see your host file system (so you do not need to mount as with Docker). However, you may need to bind other file systems (e.g. -B /scratch -B /tmp).

Once in the container, run any of the desired applications.

⁠Special instructions for running Jupyter via the container on an HPC

If you want to run a Jupyter Notebook from a container there a few steps. No guarantee this will work on your HPC.

  1. Get an interactive session:

For example, with SLURM

srun -t 8:00:00 -p interact -N 1 -n 1 --x11=first --pty /bin/bash
  1. Once in the interactive session, run a Singularity shell command. Modify this example as necessary for your use case:
singularity shell -B <other_path>  -B <host_path>:/home/jovyan/ data_science_popgen_notebook_<tagname>.simg

where <other_path> could be something like /scratch that you want your container to have access to.

  1. Then once at the Singularity shell, do these commands:
ipnport=$(shuf -i8000-9999 -n1)
ipnip=$(hostname -i)
export XDG_RUNTIME_DIR=""
jupyter-notebook --no-browser --port=$ipnport --ip=$ipnip

You'll see a bunch of output and a line of output like--

[I 19:46:12.500 NotebookApp] The Jupyter Notebook is running at:
[I 19:46:12.500 NotebookApp] http://172.26.113.253:8455/?token=0ad1ec...

Make a note of the ipnport (8455) and ipnip (172.26.113.253).

  1. On your local machine, bring up a terminal and do
ssh -N -L $ipnport:$ipnip:$ipnport <username>@<hpc>

e.g. <username>@<hpc> = [email protected]

Make sure to substitute the appropriate values for $ipnport and $ipnip from step 3 in the ssh command.

  1. On your local machine, bring up a web browser and in the address bar type

localhost:$ipnport

Make sure to substitute the appropriate value for $ipnport from step 3.

  1. Back at the Singularity shell (in Longleaf) copy and paste the displayed token string into your local web browser in the jupyter password field.
⁠Special instructions for dadi

To run dadi, once in the container, must add dadi to the Python path

PYTHONPATH="${PYTHONPATH}:/home/jovyan/dadi"
export PYTHONPATH
⁠Included applications:
⁠Languages
  • Python
  • R
  • Julia
⁠Data science
  • Jupyter lab
  • Jupyter notebook
  • Jupyter hub
  • Pweave
  • rpy2
  • pandas
  • matplotlib
  • scipy
  • seaborn
  • scikit-learn
  • scikit-image
  • dask
  • bokeh
  • and others...
⁠Population genetics
⁠Base stacks

Tag summary

Content type

Image

Digest

Size

2.1 GB

Last updated

over 6 years ago

docker pull agladstein/data_science_popgen_notebook:1.0.8