The Jupyter Datascience Notebook augmented with Spark and selected Python remote sensing packages
1.5K
This image is the Jupyter Datascience notebook with a stack of open source Remote Sensing packages.
It contains the Jupyter Notebook Data Science Stack plus an additional stack of useful remote sensing, machine learning and data analysis packages including pylidar, gdal, rasterio, rios, rsgislib, python-fmask, rasterstats, py6s, arcsi, pysolar, astropy, geoplot, h5py, folium, siphon, netCDF4, pandas, geopandas, cartopy, xarray, dask, matplotlib, plotly, scipy, sympy, numba, scikit-learn, scikit-image, ipyparallel, psycopg2, sshtunnel and keras/tensorflow.
It also contains the opendatacube package and dependencies including ssh and sshfs that allow tunneled access to the AGDC database and Lustre file systems (including the full Australian Landsat archive) on the NCI if you have a login using the commands below. You'll need to run the container using the --privileged option to mount /g using the commands below. Get in touch via the Slack group for more details on the database config you'll need to pass to datacube.Datacube if you get stuck.
ssh -N -f -o ServerAliveInterval=60 -L 6432:130.56.244.105:6432 [email protected]
sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3 -o compression=yes -o Ciphers=arcfour -o allow_other -o follow_symlinks [email protected]:/g /g
If you wish to use the installed python Earth Engine environment (earthengine-api) you'll need to authenticate your credentials once you've signed up. The first time you run import ee you'll be directed to authenticate in a terminal session (you can open a new terminal from the notebook directory).
It also has pyspark + Spark 2.0.2 + Hadoop 2.7 for use in local mode or to connect to a cluster of Spark workers for large-scale data processing.
The following command starts a container with the Notebook server listening for HTTP connections on port 8888 without authentication configured.
docker run -it -p 8888:8888 petescarth/datascience-rs
Take note of the authentication token included in the notebook startup log messages. Include it in the URL you visit to access the Notebook server or enter it in the Notebook login form.
See the details for the Jupyter Datascience notebook for additional start options including mounting local folders and authentication.
To activate the Python 2 environment use:
source activate python2
You can return to the default environment with this command:
source deactivate
There are a lot of packages with various dependencies in this build. You will probaby find something that is broken. If you do, let me know and I'll do my best to fix it.
# PyLidar container Dockerfile
# Based on https://github.com/jupyter/docker-stacks/tree/master/datascience-notebook
FROM jupyter/datascience-notebook:latest
MAINTAINER Peter Scarth <[email protected]>
LABEL Description="Jupyter Opensource Remote Sensing Tookit" Vendor="vegcover.com" Version="0.3"
USER root
# Spark dependencies and install taken from
# https://github.com/jupyter/docker-stacks/tree/master/pyspark-notebook
ENV APACHE_SPARK_VERSION 2.0.2
ENV HADOOP_VERSION 2.7
# Temporarily add jessie backports to get openjdk 8, but then remove that source
RUN echo 'deb http://cdn-fastly.deb.debian.org/debian jessie-backports main' > /etc/apt/sources.list.d/jessie-backports.list && \
apt-get -y update && \
apt-get install --no-install-recommends -t jessie-backports -y openjdk-8-jre-headless ca-certificates-java && \
rm /etc/apt/sources.list.d/jessie-backports.list && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
RUN cd /tmp && \
wget -q http://d3kbcqa49mib13.cloudfront.net/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz && \
echo "e6349dd38ded84831e3ff7d391ae7f2525c359fb452b0fc32ee2ab637673552a *spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" | sha256sum -c - && \
tar xzf spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz -C /usr/local && \
rm spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz
RUN cd /usr/local && ln -s spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION} spark
# Mesos dependencies
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF && \
DISTRO=debian && \
CODENAME=jessie && \
echo "deb http://repos.mesosphere.io/${DISTRO} ${CODENAME} main" > /etc/apt/sources.list.d/mesosphere.list && \
apt-get -y update && \
apt-get --no-install-recommends -y --force-yes install mesos=0.25.0-0.2.70.debian81 && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
# Spark and Mesos config
ENV SPARK_HOME /usr/local/spark
ENV PYTHONPATH $SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.3-src.zip
ENV MESOS_NATIVE_LIBRARY /usr/local/lib/libmesos.so
ENV SPARK_OPTS --driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info
# Update the install and install SSl libs
RUN apt-get update && apt-get install -y --no-install-recommends openssh-client sshfs libgdal1-dev libproj-dev libssl-dev libffi-dev git mercurial imagemagick unzip htop nload screen nano && apt-get clean && rm -rf /var/lib/apt/lists/*
# Install crytography and Earth Engine API
RUN pip2 install cryptography earthengine-api
RUN pip3 install cryptography earthengine-api
# Set up the mount for datacube to connect to the NCI
RUN mkdir /g && chown $NB_USER:users -R /g && chmod -R 755 /g
RUN echo 'user_allow_other' >> /etc/fuse.conf
# Switch back to default user
USER jovyan
# Add conda-forge, rios and change R priority
RUN /opt/conda/bin/conda config --prepend channels r
RUN /opt/conda/bin/conda config --prepend channels conda-forge
RUN /opt/conda/bin/conda config --prepend channels rios
#RUN /opt/conda/bin/conda update conda -y
# Install TensorFlow, GDAL, RasterStats, PyLidar, RSGISLib and statistical utilities
RUN /opt/conda/bin/conda install gdal rasterstats netCDF4 pylidar python-fmask rsgislib datacube arcsi pysolar py6s sixs=1.1 h5py numpy scipy sympy numba matplotlib xerces-c geopandas folium geoplot siphon astropy plotly psycopg2 sshtunnel ipyparallel xarray dask jsonschema pyyaml sqlalchemy cachetools cartopy holoviews postgresql bottleneck python-dateutil pathlib click distributed pyparsing keras tensorflow tpot r-base r-spatial r-maps r-mapproj r-leaflet r-raster r-maptools -y
# Note geoplot is not available in python 2.7.x
RUN /opt/conda/bin/conda install -n python2 gdal rasterstats netCDF4 pylidar python-fmask datacube multiprocessing rsgislib arcsi pysolar py6s sixs=1.1 h5py numpy scipy sympy numba matplotlib xerces-c geopandas folium siphon astropy plotly psycopg2 sshtunnel ipyparallel xarray dask jsonschema pyyaml sqlalchemy cachetools cartopy holoviews postgresql bottleneck python-dateutil pathlib click distributed pyparsing keras tensorflow tpot -y
RUN /opt/conda/bin/conda clean -t
# Fix for some regressions - we need xerces-c= 3.1.4-3 and r-base updated to use conda-forge
RUN /opt/conda/bin/conda install xerces-c=3.1.4 matplotlib=2.0.0 r-base readline r-stringi -y
RUN /opt/conda/bin/conda install -n python2 xerces-c=3.1.4 matplotlib=2.0.0 -y
RUN /opt/conda/bin/conda clean -t
Content type
Image
Digest
Size
5.4 GB
Last updated
over 9 years ago
docker pull petescarth/datascience-rs