Public | Automated Build

Last pushed: 3 years ago
Short Description
Apache Spark development environment
Full Description


This docker bootstrap script is used for developing Apache Spark.
It seems that there is no bootstrap script for development environment for Apache Spark as far as I know.

Create a Docker Container

A docker image named spark-dev will be created, if we run the below command.
This container is based on Ubuntu Trusty Tahr (14.04 LTS) and is conposed of Oracle Java7 and Scala 2.10.4.

# Run docker daemon If you don't
boot2docker init
boot2docker up

# Create a docker container
docker run -i -t spark-dev /bin/bash

Compiled Apache Spark master branch

This bootstrap script already build the master branch of Apache Spark in order to pre-fetch dependent libraries.
The source code exists in /opt/spark.
So adding your github repository, you can immediatly use the development environment.

cd /opt/spark
git remote add dev git://your-github/forked/spark

Share Spark Source between Host and a Docker Image

If you want to deal with Spark source on a docker image without checking out, Docker provides you to share a directory between host machine and a docker image.

docker run -i -t -v /your/spark/source:/container/directory spark-dev /bin/bash
Docker Pull Command
Source Repository