louisdorard/oml repository overview

This directory contains Jupyter notebooks (the .ipynb files), which themselves contain Python code. The best way to use these notebooks (and the Python libraries/packages they depend on) is to get a "conda" environment set up.

Contents:

• I. Introduction to notebooks⁠
• II. Setting up the conda environment⁠
• III. Accessing notebooks from Jupyter⁠
• Appendix A: How the conda environment and docker image were created⁠
• Appendix B: Further reading⁠
• Appendix C: Work In Progress⁠

⁠I. Introduction to notebooks

Notebooks are very popular in the Data Science and Machine Learning communities as they provide great ways to learn and experiment. They are interactive web pages that include blocks of code that you can run and edit. For this to work, they need to be served by a notebook server called Jupyter. For a quick introduction to Jupyter notebooks, check out this video: https://youtu.be/pwr-pR0tu5Y⁠

The easiest way to install Python and Jupyter is via conda⁠ (I recommend to use Python 2.7 to maximize compatibility with various packages). You can choose from 3 options:

1. Using a docker image that already contains Miniconda
2. Installing Miniconda (command line)
3. Installing Anaconda (graphical interface)

⁠Remarks

• Options 1 and 2 require some basic command line knowledge (you can learn more about the command line in this free course: Codeacademy — Learn the Command Line⁠).
• If you're not familiar with docker and you're leaning towards option 2 or 3, but aren't sure which one to choose, check out this discussion on Miniconda vs Anaconda⁠.
• Consult the system requirements for your preferred option and make sure to meet them.
• The Python code in the notebooks included here is pretty simple, but if you want to learn Python from scratch or to improve your Python skills, check out the free resources below:
  • Codeacademy — Learn Python⁠
  • Robert Johansson — Scientific Python Lectures⁠ (and in particular "Introduction to Python programming")

⁠II. Setting up the conda environment

Conda environments allow us to make sure we are all working with the same version of Python and Python packages. Follow the indications below to create an environment that will allow you to use my code.

⁠Option 1: docker

Docker "containers" act as lightweight virtual machines. Similarly to VMs, they are built from images. The docker image we’ll use is referenced by "louisdorard/oml" (check out the Dockerfile in this repository for more information on how it was built). For a quick introduction to docker, check out this video: https://youtu.be/9hzFzcIX10g⁠

⁠Install docker

Learn more about docker and how to install it on your OS at https://www.docker.com/⁠

⁠Run docker container

Launch the following command (from the directory where this README file is):

docker run -d -p 8888:8888 -v $PWD:/home/jovyan/work --name oml louisdorard/oml start-notebook.sh --NotebookApp.token=''

That's it! Everything's already set up in the docker container. You can jump straight to the section of this document on "Accessing notebooks from Jupyter".

If you're curious, here's some more information on the command line above:

• it creates a docker container called 'oml' -- which you can think of as a lightweight virtual machine -- from an image found on docker hub and referenced by 'louisdorard/oml'
• it maps the current directory in your OS to /home/jovyan/work in the container
• it maps the 8888 port so that when accessing it in your OS, everything gets redirected to the container on the same port
• when the container is created the command start-notebook.sh --NotebookApp.token='' is executed inside of it, which starts the Jupyter notebook (the "token" option is to make it easier to connect to that notebook and is not recommended in production settings).

⁠Option 2: local install via Miniconda and the command line

2.1 On Linux:

wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -O /tmp/miniconda.sh
/bin/bash /tmp/miniconda.sh -b -p ~/anaconda/
export PATH="~/anaconda/bin:$PATH"
echo "export PATH=\"\$PATH:~/anaconda/bin\"" >> ~/.bashrc

On macOS and Windows, follow these instructions⁠. Choose Python 2.7, and the 64-bit installer if in doubt.

Make sure that conda is installed:

conda --version

2.2 Install Jupyter:

conda install jupyter

2.2 Create an environment which contains all the libraries used in the notebooks. The environment configuration file is oml.yml. Execute this command from the directory where this file is (i.e. the same where this README file is):

conda env create --name oml --file oml.yml

(Side note: this environment also contains libraries that I use in my Operational Machine Learning workshop, in particular...

• scikit-learn, the most popular open source machine learning library
• skll, a library to compare predictive models easily
• bigmler, a command-line tool for BigML
• xgboost, an efficient multi-threaded implementation of a very powerful predictive modeling technique
• hyperopt, a library to optimize (hyper)parameters used to train predictive models.)

2.2 Make this environment available to Jupyter:

source activate oml
python -m ipykernel install --user --name oml --display-name "Python (oml)"

Launching Jupyter is as simple as:

jupyter notebook

Note: it's best to run this command from the directory where your .ipynb notebook files are.

⁠Option 3: local install via Anaconda and the GUI

Follow the instructions⁠. If asked for a version of Python, choose 2.7.

Once this is done, launch Anaconda Navigator:

• Go to the 'Environments' tab and choose 'Import'; enter 'oml' as the name of the environment and point to oml.yml
• Go back to the 'Home' tab and click on the Jupyter icon

⁠III. Accessing notebooks from Jupyter

Launching Jupyter should open a new window in your web browser (http://localhost:8888⁠)

The home page served by Jupyter lists the contents of the folder from which the server was launched. Make sure that the notebooks you want to open are within that folder (or a sub-folder) so you can navigate to them.

If you're new to Jupyter notebooks, start with opening Simple intro to Jupyter.ipynb. You can then try Machine Learning with BigML API - Interactive Code Tutorial.ipynb and move on to AmazonML-Python.ipynb to see how Amazon ML compares to BigML.

When opening a notebook, make sure that the kernel displayed on the right hand side of the page is "Python (oml)". If Jupyter notifies you that the kernel was not found, choose "Python [conda env:oml]".

⁠Appendix A: How the conda environment and docker image were created

⁠Conda environment

Update your jupyter/base-notebook⁠ image with docker pull jupyter/base-notebook. Start a new container with docker run -d -p 8888:8888 -v $PWD:/home/jovyan/work --name jupyter jupyter/base-notebook. Start a bash session in that container as root with docker exec -i -t --user root jupyter bash and from there:

apt-get update
apt-get install -yq gcc # required for bigmler later on

Start a bash session as jovyan with docker exec -i -t jupyter bash and from there:

conda update conda --yes
conda create --name oml python=2.7 scikit-learn=0.17.1 pandas nltk boto ipykernel matplotlib tensorflow=1.0.0 --yes
source activate oml
conda install --yes -c aterrel xgboost=0.4.0.c4fa2f
conda install --yes -c conda-forge keras=2.0.2
pip install bigmler==3.8.7 skll==1.2.1 bash_kernel pymongo indicoio
pip install -i https://pypi.anaconda.org/pypi/simple hyperopt==0.0.2
conda env export -n oml

⁠Docker image

From this directory:

docker build -t="louisdorard/oml" .

Automated builds were also set up on docker hub⁠.

⁠Appendix B: Further reading

• What is the difference between pip and conda?⁠
• Conda: Myths and Misconceptions⁠
• TODO: docker quickstart

⁠Appendix C: Work In Progress

If you notice anything that needs fixing, please open an issue⁠.

TODO test Amazon ML notebook

TODO test Option 2 on Windows