Apache Spark Image for the Spark Fundamentals I course
This Docker image should be used for creating environments for conducting hands-on labs for the Spark Fundamentals I course on the www.bigdatauniversity.com. You can use it to create Spark environment on your own laptop/desktop or on one of the supported public clouds.
This docker image contained pre-deployed IBM STC Spark with Hadoop.
Set up Docker environment on your laptop
How to use this image ?
Start Kitematic in Docker folder
type bigdatauniversity in the search box to filter the Docker Hub catalog to Big Data University provided images
Click on Create button on the spark image to create Docker container using this image
Docker Quickstart Terminal (CLI)
-- "Applications -> Docker -> Docker Quickstart Terminal"
-- "Start -> Program -> Docker -> Docker Quickstart Terminal".
Then run the below steps within this terminal.
1) Pull (download) this Docker image
Run this command in your terminal window:
docker pull bigdatauniversity/spark
- Note: it may take a while to pull this image over the internet
2) Start Docker container as daemon
docker run -it --hostname bigdatauniversitySpark --name bdu_spark -P -p 8080:8080 -p 8081:8081 bigdatauniversity/spark:latest /etc/bootstrap.sh -bash
docker run -d --hostname bigdatauniversitySpark --name bdu_spark -P -p 8080:8080 -p 8081:8081 bigdatauniversity/spark:latest /etc/bootstrap.sh -d
3) Start Spark
- To start Scala Spark shell:
- To start Python Spark shell:
- All hands-on lab files are located in:
- How to restart and attach to the container
If you exit from Docker Container, you can always restart and attach to it later by running the below:
docker start bdu_spark docker attach bdu_spark
- Start a new command in a running container
docker exec -it bdu_spark <command>
The supported tags stands for version of Spark.
Supported Docker versions
- This image is officially supported on Docker version 1.6.0.
- Support for older versions (down to 1.0) is provided on a best-effort basis.
- Zhong Yu (Leo) Wu ( email@example.com )
- Emerging Technology Team ( firstname.lastname@example.org ), IBM Analytics Platform
Like this image? Give us a star at the top of this page!