Public | Automated Build

Last pushed: 2 years ago
Short Description
Short description is empty for this repo.
Full Description

Mesos Spark

Spark for mesos.

Usage

Docker images are tagged by spark versions.

The Master URLs for Mesos are in the form mesos://host:5050
for a single-master Mesos cluster, or mesos://zk://host:2181
for a multi-master Mesos cluster using ZooKeeper.

Spark distribution url should point to spark distribution
for your version of spark. You can obtain url from
spark download page or host it yourself (preferred option).
Note that you need compiled spark version, not source code.

Spark shell

docker run --rm -t -i --net=host bobrik/mesos-spark:1.1.1 /opt/spark/bin/spark-shell \
  --master <mesos master> --conf spark.executor.uri=<spark distribution url>

Check that everything works as expected:

sc.parallelize(1 to 1000).count()

Spark submit

docker run --rm --net=host -v /app/on/host:/app/in/docker bobrik/mesos-spark:1.1.1 \
  /opt/spark/bin/spark-submit --master <mesos master> \
  --conf spark.executor.uri=<spark distribution url> /app/in-docker

Here /app/on/host could be java, scala or python app. You should
check out docs for examples.

Note that for java you also need to specify --class.

Docker Pull Command
Owner
bobrik
Source Repository

Comments (0)