vinston/spark repository overview

Spark/Hadoop Image Saprk 1.6.1 image details Java 1.8
Spark 1.6.1
Hadoop 2.7.1
Hive 2.1.1

Docker Spark 2.0.0 Configuration Setting up Apache Spark standalone cluster with Docker containers

To run the Spark2.0.0, use the below docker-compose script spark-master: image: vinston/spark:2.0.0 command: bin/spark-class org.apache.spark.deploy.master.Master -h spark-master hostname: spark-master environment: MASTER: spark://spark-master:7077 SPARK_CONF_DIR: /conf SPARK_PUBLIC_DNS: 192.168.99.100 expose: - 7001 - 7002 - 7003 - 7004 - 7005 - 7006 - 7077 - 6066 ports: - 4040:4040 - 6066:6066 - 7077:7077 - 8080:8080 volumes: - ./conf/spark-master:/conf - ./data:/tmp/data

spark-worker-1: image: vinston/spark:2.0.0 command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077 hostname: spark-worker-1 environment: SPARK_CONF_DIR: /conf SPARK_PUBLIC_DNS: 192.168.99.100 SPARK_WORKER_CORES: 2 SPARK_WORKER_MEMORY: 2g SPARK_WORKER_PORT: 8881 SPARK_WORKER_WEBUI_PORT: 8081 links: - spark-master expose: - 7012 - 7013 - 7014 - 7015 - 7016 - 8881 ports: - 8081:8081 volumes: - ./conf/spark-worker-1:/conf - ./data:/tmp/data

Once started, access the master node Web UI in browser using following URL: http://192.168.99.100:8080⁠