lamastex/hsjupyter repository overview

⁠hsJupyter

Docker image containing Apache Hadoop, Apache Spark and Jupyter with Toree kernels for Spark support in Jupyter notebooks. Like hsBase it is preconfigured for using hdfs and Yarn in pseudo-distributed mode.

⁠Usage

To start Jupyter notebook simply use the command docker run -p 8888:8888 lamastex/hsjupyter and use the token printed to stdout to enter the webui at localhost:8888. To start Jupyter along with hdfs use docker run -p 8888:8888 -p 50070:50070 lamastex/hsjupyter bash and run the script start.sh in the work directory.

The Toree Scala and SQL interpreters are preinstalled for using Spark with Scala and SQL. For using PySpark the package findspark is also installed. To use PySpark and have a Spark context available as sc, start a Python3 notebook and run

import findspark
findspark.init()
import pyspark
sc = pyspark.SparkContext()

⁠Ports

The following ports are exposed by the dockerfile:

• Jupyter webUI: 8888
• hdfs webUI: 50070
• Yarn webUI: 8088
• Spark master webUI: 7070
• Spark worker webUI: 8081

Note that the Spark master webUI port is not the default 8080 to avoid collision with Zeppelin which also uses port 8080 by default.