Public Repository

Last pushed: 2 years ago
Short Description
A Docker image with Apache Spark and Model Evaluation Tools
Full Description

This Repository contains a Docker files to build a Docker image with Apache Spark and Model Evaluation Tools. This Docker image depends on the sequenceiq/spark:1.6.0 docker image, which is available at the SequenceIQ GitHub page.

1 Pull the image from the Docker Repository

#docker pull bingrao/met

2 Run the image

%if using boot2docker make sure your VM has more than 2GB memory
%in your /etc/hosts file add $(boot2docker ip) as host 'sandbox' to make it easier to access your sandbox UI
%open yarn UI ports when running container

#docker run -it -p 8088:8088 -p 8042:8042 -h sandbox bingrao/met bash
or

#docker run -d -h sandbox bingrao/met -d

3 Run test

1) run the spark shell

#spark-shell --master yarn-client --driver-memory 1g --executor-memory 1g --executor-cores 1

2) Estimating Pi (yarn-cluster mode):

#spark-submit \
--class org.apache.spark.examples.SparkPi \
--files $SPARK_HOME/conf/metrics.properties \
--master yarn-cluster \
--driver-memory 1g \
--executor-memory 1g \
--executor-cores 1 \
$SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar

3) Estimating Pi (yarn-client mode):

#spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn-client \
--driver-memory 1g \
--executor-memory 1g \
--executor-cores 1 \
$SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar

4) Model Evaluation tools
Install directory: /usr/local/src/met-5.1
Execute files: /usr/local/src/met-5.1/bin

Docker Pull Command
Owner
bingrao

Comments (0)