Docker container for spark stand alone cluster
To run master execute:
To run worker execute:
You can run multiple workers. Every worker would be able to find master by it's container name "spark_master".
To run spark shell against this cluster execute:
You can run multiple shells. Every shell would be able to find master by it's container name "spark_master".
If you like to run another container against this cluster, please read explanation how to prepare driver container.
If you need to increase memory or core count or pass any other parameter to spark, please use:
./spark-shell.sh --executor-memory 300M --total-executor-cores 3 ./start-worker.sh --memory 700M
If you execute these images without scripts mentioned above, please:
- Remeber to name master container as spark_master for correct working on linkage.
- Read documentation to understand what's going on.