Public Repository

Last pushed: 5 months ago
Short Description
spark-master for kubernetes cluster
Full Description


This is only for mscp platform


the version no. such as 2.0.1 means we are using spark 2.0 , and the the last number is for our version number


  1. Add interface to use zookeeper enable master ha

all you need is to pass in the zookeeper service name by using ZK_SERVER_LIST environment virable
for instance , assume we have a zookeeper cluster with hostnames zk1-service, zk2-service, zk3-service

docker run --rm -e ZK_SERVER_LIST=zk1,zk2,zk3 archerbj/spark-master:2.0.2


  1. start spark-master with ${SPARK_HOME}/bin/spark-class org.apache.spark.deploy.master.Master

which enables to run spark process at foreground , that will benefit us from running spark in docker (kubernetes).
Once there is something wrong with the process , it will shut down , and kubernetes rc will help us recover the pod

Docker Pull Command