Public Repository

Last pushed: 7 months ago
Short Description
spark-master for kubernetes cluster
Full Description

WARNNING

This is only for mscp platform

info

the version no. such as 2.0.1 means we are using spark 2.0 , and the the last number is for our version number

v2.0.2

  1. Add interface to use zookeeper enable master ha

#Usage
all you need is to pass in the zookeeper service name by using ZK_SERVER_LIST environment virable
for instance , assume we have a zookeeper cluster with hostnames zk1-service, zk2-service, zk3-service

docker run --rm -e ZK_SERVER_LIST=zk1,zk2,zk3 archerbj/spark-master:2.0.2

v2.0.1

  1. start spark-master with ${SPARK_HOME}/bin/spark-class org.apache.spark.deploy.master.Master

which enables to run spark process at foreground , that will benefit us from running spark in docker (kubernetes).
Once there is something wrong with the process , it will shut down , and kubernetes rc will help us recover the pod

Docker Pull Command
Owner
archerbj

Comments (0)