marcelmittelstaedt/spark_base repository overview

This docker image serves the purpose of experimental usage and for educational purposes. It should not be used in a productive manner in any way.

Source Files: https://github.com/marcelmittelstaedt/BigData/tree/master/docker/hiveserver_base⁠

Used within lecture "Big Data", see:

• https://github.com/marcelmittelstaedt/BigData⁠ or
• http://wwwlehre.dhbw-stuttgart.de/~mittelstaedt/⁠ for more details.

Image Contains:

• Ubuntu 18.04 LTS
• OpenJDK, Vim, Curl, PDSH, HTOP
• Hadoop 3.1.2 (configured for single node pseudo-distributed mode)
• Spark 2.3.4
• Hive 3.1.2 (configured for single node pseudo-distributed mode)
• HiveServer2

Usage:

• Switch to hadoop user: ~/ sudo su hadoop
• Start Hadoop Cluster: ~/ start-all.sh
• Start Spark Shell (with Yarn Master): ~/ spark-shell --master yarn
• Start PySpark Shell (with Yarn Master): ~/ pyspark --master yarn
• Start Jupyter Notebook: ~/ jupyter notebook
• Stop Hadoop Cluster: ~/ stop-all.sh