tomihararznde/spark-cluster repository overview

⁠Spark Cluster

The utilization is very simple, pull the image and run as following:

First create a Bridge network to the containers can communicate

docker network create --driver bridge spark-cluster

ex: master

docker run -d \
    --name=spark-master \
    --net spark-cluster \
    -p 8080:8080 \
    -p 7077:7077 \
    -e SPARK_TYPE=master \
    tomihararznde/spark-cluster:latest

ex: slave

 docker run -d \
    --net spark-cluster \
    -p 7078:7078  \
    -e SPARK_TYPE=slave \
    -e MASTER_SPARK_URL=spark-master:7077 \
    -e EXTRA_PARAMS='--port 7078' \
    tomihararznde/spark-cluster:latest

Environment variables:

SPARK_TYPE: 
    master
    slave

If is a slave type, a extra variable is needed:

MASTER_SPARK_URL
   master_hostname:master_portnumber

If you have a additional configuration, add it in the following environment variable

EXTRA_PARAMS:
    '--port 7078'

⁠Kubernetes / Openshift

It´s ready to be used with Kubernetes/Openshift, you just need to create the variables and configure it to master and slave, when you resize the slaves it will be recognized automatically by master.

There are examples in my github of master, slaves and master services deployment

Enjoy and don't forget to gimme a Star if it works to you!