Public Repository

Last pushed: 2 years ago
Short Description
personal spark master
Full Description

drpaulbrewer/spark-master is a personal build of spark (1.3.1 as of April 2015) with scripting to be a master

The version, build options, settings, etc. are from drpaulbrewer/spark-roasted-elephant and subject to change.

This is not an official public build and there is NO WARRANTY for this code. ALL USE IT AT YOUR OWN RISK. If it works for you, great. But don't expect it to always work, or to always have the same options compiled in.

Here's a short shell script to start a master and set up the networking on a firewalled home LAN.

We use pipework to set up the networking. You'll probably want to change the IP address.

./run-docker-spark-master

#!/bin/bash
sudo -v
MASTER=$(docker run --name="master" --expose=1-65535 --env SPARKDIR=/spark/spark-1.3.1 --env SPARK_MASTER_IP=192.168.1.10 --env SPARK_LOCAL_IP=192.168.1.10 -v /data:/data -v /tmp:/tmp -d drpaulbrewer/spark-master:latest)
sudo pipework eth0 $MASTER 192.168.1.10/24@192.168.1.1

To cleanup, use this shell file on the host (not the container):

./stop-docker-spark-master.sh

#!/bin/bash
docker kill master
docker rm master
Docker Pull Command
Owner
drpaulbrewer