bumory1987/spark3 repository overview

Setup spark standalone(3.4.0 & java17)

(1) 8083, 7077, 8090 ,8091 ,8092, 8093 ,8889 port should be released before running command
(2) you have create docker network(bridge) called spark
(3) you have to generate worker as worker_0, worker_1, .... . you have to keep this name format.
(4) your spark master name should be "master".
(5) controller has a jupylter lab to run spark standalone (6) {your_docker_volumn} in here, there should be any csv file that spark will read

#controller setup ( took 3 min for generate jupyter lab, juypterlab password=1234 )
<1>
docker run -d --network spark --name controller -v {your_docker_volumn}:/home/sung/inventory -p 8889:8889 bumory1987/spark3:controllerm

from this moment, we can access to jupyterlab via localhost:8889

#master setup
<2>
docker run -d --cpus="2" --memory="4g" --network spark --name master -v {your_docker_volumn}:/home/sung/inventory -p 8083:8083 -p 7077:7077 -e "SPARK_MASTER_HOST=master" bumory1987/spark3:masterkube

#worker setup
<3>
docker run -d --cpus="1" --memory="3g" --network spark --name worker_0 -p 8090:8090 -v {your_docker_volumn}:/home/sung/inventory -e "SPARK_MASTER_HOST=master" -e "SPARK_MASTER_PORT=7077" -e "SPARK_MASTER_WEBUI_PORT=8083" -e "SPARK_WORKER_WEBUI_PORT=8090" bumory1987/spark3:workerkube

docker run -d --cpus="1" --memory="3g" --network spark --name worker_1 -p 8091:8090 -v {your_docker_volumn}:/home/sung/inventory -e "SPARK_MASTER_HOST=master" -e "SPARK_MASTER_PORT=7077" -e "SPARK_MASTER_WEBUI_PORT=8083" -e "SPARK_WORKER_WEBUI_PORT=8090" bumory1987/spark3:workerkube

docker run -d --cpus="1" --memory="3g" --network spark --name worker_2 -p 8092:8090 -v {your_docker_volumn}:/home/sung/inventory -e "SPARK_MASTER_HOST=master" -e "SPARK_MASTER_PORT=7077" -e "SPARK_MASTER_WEBUI_PORT=8083" -e "SPARK_WORKER_WEBUI_PORT=8090" bumory1987/spark3:workerkube

docker run -d --cpus="1" --memory="3g" --network spark --name worker_3 -p 8093:8090 -v /home/sung/inventory:/home/sung/inventory -e "SPARK_MASTER_HOST=master" -e "SPARK_MASTER_PORT=7077" -e "SPARK_MASTER_WEBUI_PORT=8083" -e "SPARK_WORKER_WEBUI_PORT=8090" bumory1987/spark3:workerkube