hashicorp/spark-nomad
Apache Spark distribution with HashiCorp Nomad scheduler support
5.5K
This image provides a build of Apache Spark 2.1.1 with the addition of support for scheduling against HashiCorp Nomad. The PR that implements this support can be found here: https://github.com/apache/spark/pull/18209.
Full usage instructions can be found here: https://github.com/barnardb/spark/blob/nomad/docs/running-on-nomad.md
Start a Nomad agent in developer mode on a machine with Docker:
$ sudo nomad agent -dev
This will create a Nomad server and client so that we can submit or example Spark job to.
Run the following to create a Spark job that will run on the Nomad agent created in Spark client mode:
$ docker run --network=host -it hashicorp/spark-nomad ./opt/spark/bin/spark-submit \
--master nomad \
--docker-image hashicorp/spark-nomad \
--distribution local:///opt/spark \
--driver-memory 512m \
--executor-memory 512m \
--class org.apache.spark.examples.SparkPi \
local:/opt/spark/examples/jars/spark-examples_2.11-2.1.1.jar \
10
Run the following to create a Spark job that will run on the Nomad agent created in Spark cluster mode:
$ docker run --network=host -it hashicorp/spark-nomad ./opt/spark/bin/spark-submit \
--master nomad \
--deploy-mode cluster \
--docker-image hashicorp/spark-nomad \
--distribution local:///opt/spark \
--driver-memory 512m \
--executor-memory 512m \
--class org.apache.spark.examples.SparkPi \
local:/opt/spark/examples/jars/spark-examples_2.11-2.1.1.jar \
10
docker pull hashicorp/spark-nomad