qulo/rancher-spark repository overview

⁠spark-rancher

A Apache Spark⁠ image for Rancher⁠ based on the docker container [gettyimages/docker-spark] (https://github.com/gettyimages/docker-spark⁠). Used to spin up a easy to manage spark cluster on Rancher.

⁠Versions

• Oracle Java⁠ - v8.131_11
• Apache Spark⁠ - v2.1.1
• Apache Hadoop⁠ - v2.8.0
• Livy⁠ - v0.4.0-SNAPSHOT

⁠Running locally

# starting spark master:
docker run qulo/rancher-spark master <hostname or ip of the master>

# starting spark worker:
docker run qulo/rancher-spark worker <hostname or ip of the master>

⁠Running with docker-compose

# starting
docker-compose up

# skaling workers to 3
docker-compose scale spark-worker=3

# stopping
docker-compose stop

⁠Running in Rancher

⁠WARNING: this will open port 8080 on the spark-master to public

create a service stack and use docker-compose.yml⁠ service creation.