Sign inSign up

ludwigprager/spark-python

By ludwigprager

•Updated almost 7 years ago

spark-submit ready docker container for evaluation purposes

Image
0

208

ludwigprager/spark-python repository overview

Docker container with python and spark

⁠tl;dr

This container ist ready for your spark-submit. It sports a running spark master and worker.

⁠Build the container

. set-env.sh
docker build  -t ${SPARK_PYTHON_IMAGE} .

⁠Start a shell

./shell.sh

After calling this command you can run a 'spark-submit' command:

lprager@mycomputer:~/work/spark/spark-python$ ./shell.sh 
starting org.apache.spark.deploy.master.Master, logging to /spark-2.4.4-bin-hadoop2.7/logs/spark--org.apache.spark.deploy.master.Master-1-debian.out
starting org.apache.spark.deploy.worker.Worker, logging to /spark-2.4.4-bin-hadoop2.7/logs/spark--org.apache.spark.deploy.worker.Worker-1-debian.out

example:
spark-submit --master spark://debian:7077 /spark-2.4.4-bin-hadoop2.7/examples/src/main/python/pi.py

root@debian:/my_project# spark-submit --master spark://debian:7077 /spark-2.4.4-bin-hadoop2.7/examples/src/main/python/pi.py
19/10/29 16:29:52 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/10/29 16:29:53 INFO SparkContext: Running Spark version 2.4.4
19/10/29 16:29:53 INFO SparkContext: Submitted application: PythonPi
.
.
.
.

⁠short cut in your .bashrc

with the following shortcut in your .bashrc you are able to start this container by typing 'sp'

sp() {

docker run \
  -ti --rm \
  --hostname debian \
  -v $(pwd)/:/my_project/ \
  ludwigprager/spark-python
}

Tag summary

Content type

Image

Digest

Size

921.5 MB

Last updated

almost 7 years ago

docker pull ludwigprager/spark-python