Public | Automated Build

Last pushed: 7 months ago
Short Description
spark-1.2.0 image with supervisord, hadoop-2.4.0, scala-2.10.4 and yarn
Full Description

This is my minimal spark image (based on hadoop-2.4.0, scala-2.10.4)

Run the Containers

SSH keys Container

First we fire up the keys_host so that we can mount the .ssh volume in
spark container.


sudo docker run --name keyhost ezhaar/key-host

Spark Container

Now we run the Spark container


    sudo docker run -it \
    --volumes-from keyhost \
    --name spark-test \
    -h master.localdomain \
    --dns-search=localdomain \
    ezhaar/docker-spark
  • -h sets the hostname and adds an entry in the /etc/hosts file.
  • --dns-search updates the /etc/resolve.conf for reverse DNS lookups.

Once the container has started, format the namenode the namenode, we are ready
to play with hadoop and spark.


# start hadoop and spark
/root/start-bdas.sh

Note: Make sure to verify the spark-env.sh.

Create a Spark Cluster on a Single Host

To create a spark cluster look at my
github

Docker Pull Command
Owner
ezhaar
Source Repository