Public Repository

Last pushed: 10 months ago
Short Description
spark image 2.0.1 hadoop 2.7.2 with r and python
Full Description

spark image includes

  • gfortran
  • r
  • python3
  • spark 2.0.1
  • hadoop 2.7.2

tags

  • cest
    includes timezone set to Europe/Vienna in container

for worker
moutn volumes for work and scratch
add properties file in conf

ENV SPARK_CONF_DIR /conf
ENV SPARK_LOCAL_DIRS /tmp/scratch
ENV SPARK_WORKER_DIR /tmp/work
EXPOSE 7012-7016 # depends on conf
EXPOSE 8881 # depends on conf
EXPOSE 8081 #UI port
CMD ['bin/spark-class','org.apache.spark.deploy.worker.Worker','spark://master:7077']

for master
FROM samuelst/sparkbase
ENV MASTER spark://master:7077
ENV SPARK_CONF_DIR /conf
ENV SPARK_PUBLIC_DNS localhost
ENV SPARK_LOCAL_DIRS /tmp/work
EXPOSE 7001-7006 # conf dependent
EXPOSE 7077
EXPOSE 6066
EXPOSE 8080 # UI port
WORKDIR $SPARK_HOME
CMD ['bin/spark-class','org.apache.spark.deploy.master.Master','-h','master']

Docker Pull Command
Owner
samuelst

Comments (0)