spsenthil/spark repository overview

docker images

docker run -it -v /Users/<yourId>/programs:/programs/notebook -p8080:8080 -p8088:8088 -p7077:7077 -p8888:8888 <imageId> sh sp-master.sh

(you can find the spark master IP in the output of above script)

docker run -it <imageId> sh sp-worker.sh <ip-of-spark-master>

Test these in the browser: Juypter - IP:8888 SparkUI - IP:8080 Yarn RM - IP:8088

Find the output of the first command (to get master ip - 172.17.0.2) STARTUP_MSG: Starting NodeManager STARTUP_MSG: host = 052e0ed35cc5/172.17.0.2

In case, you want to run all these through Kubernetes cluster with autoscaling of workers, look at this link [ https://github.com/senthilaru/spark-yarn-jupyter-kubernetes⁠ ]

Any Issues/Improvements: Pls. log here [https://github.com/senthilaru/spark-yarn-jupyter-kubernetes/issues⁠]