Public | Automated Build

Last pushed: 2 years ago
Short Description
Docker spark
Full Description

Docker container for spark stand alone cluster

This repository contains a set of scripts and configuration files to run a Apache Spark standalone cluster from Docker container.

Upgraded to Spark 1.6, Scala 2.11.7

To run master execute:

./start-master.sh

To run worker execute:

./start-worker.sh

You can run multiple workers. Every worker would be able to find master by it's container name "spark_master".

To run spark shell against this cluster execute:

./spark-shell.sh

You can run multiple shells. Every shell would be able to find master by it's container name "spark_master".

If you like to run another container against this cluster, please read explanation how to prepare driver container.

If you need to increase memory or core count or pass any other parameter to spark, please use:

./spark-shell.sh --executor-memory 300M --total-executor-cores 3
./start-worker.sh --memory 700M

If you execute these images without scripts mentioned above, please:

  • Remeber to name master container as spark_master for correct working on linkage.
  • Read documentation to understand what's going on.
Docker Pull Command
Owner
segy
Source Repository

Comments (0)