oneoffcoder/spark-jupyter repository overview

⁠Purpose

This docker container is meant to be used for learning purpose for programming PySpark. It has the following components.

• Hadoop v3.2.1
• Spark v2.4.4
• Conda 3 with Python v3.7

After running the container, you may visit the following pages.

• HDFS⁠
• YARN⁠
• Spark⁠
• Spark History⁠
• Jupyter Lab⁠

As can be seen, Jupyter Lab is running on port 8888. An example notebook is mounted at /root/ipynb. To get the PySpark code to run, you will have to upload the data.csv file to HDFS first. View the example notebook⁠.

⁠Running

To run the docker container, type in the following.

docker run -it \
    -p 9870:9870 \
    -p 8088:8088 \
    -p 8080:8080 \
    -p 18080:18080 \
    -p 9000:9000 \
    -p 8888:8888 \
    -p 9864:9864 \
    -v $HOME/git/docker-containers/spark-jupyter/ubuntu/root/ipynb:/root/ipynb \
    oneoffcoder/spark-jupyter

⁠GitHub

The source code is on GitHub⁠.