yuiskw/docker-spark-dev repository overview

⁠README

This docker bootstrap script is used for developing Apache Spark. It seems that there is no bootstrap script for development environment for Apache Spark as far as I know.

⁠Create a Docker Container

A docker image named spark-dev will be created, if we run the below command. This container is based on Ubuntu Trusty Tahr (14.04 LTS) and is conposed of Oracle Java7 and Scala 2.10.4.

# Run docker daemon If you don't
boot2docker init
boot2docker up

# Create a docker container
bash build.sh
docker run -i -t spark-dev /bin/bash

⁠Compiled Apache Spark master branch

This bootstrap script already build the master branch of Apache Spark in order to pre-fetch dependent libraries. The source code exists in /opt/spark. So adding your github repository, you can immediatly use the development environment.

cd /opt/spark
git remote add dev git://your-github/forked/spark

⁠Share Spark Source between Host and a Docker Image

If you want to deal with Spark source on a docker image without checking out, Docker provides you to share a directory between host machine and a docker image.

docker run -i -t -v /your/spark/source:/container/directory spark-dev /bin/bash