Sign inSign up

pschatzmann/spark

By pschatzmann

•Updated almost 8 years ago

Apache Spark-2.1.0 on Alpine

Image
1

2.1K

pschatzmann/spark repository overview

Docker swarm is used to start a spark cluster with one master, multiple workers and one proxy for the web gui. Information about spark can be found at http://spark.apache.org/⁠

You can start a spark swarm cluster with the following compose file:

  • The master is running on the manager node so that we have a defined point of entry.
  • The worker nodes are scheduled by the docker storm cluster.
  • In order to make the web gui of each worker accessible we finally start a proxy.

⁠Docker Compose Example

The cluster can bee addessed with spark://:7077 and the gui is available at http://:8006

version: '3.0' 
 services:
  spark-master:
    image: pschatzmann/spark
    command: bin/spark-class org.apache.spark.deploy.master.Master
    environment:
      SPARK_MASTER_HOST: 0.0.0.0
      SPARK_MASTER_WEBUI_PORT: 8004
      SPARK_CONF_DIR: /conf
    ports:
      - 6066:6066
      - 7077:7077
      - 8004
    networks:  
       - spark
    deploy:
      placement:
        constraints:
          - node.role == manager

   spark-worker:
     image: pschatzmann/spark
    depends_on: 
      - spark-master
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark-master:7077
    environment:
      SPARK_CONF_DIR: /conf
      SPARK_WORKER_CORES: 2
      SPARK_WORKER_MEMORY: 2g
      SPARK_WORKER_PORT: 8881
      SPARK_WORKER_WEBUI_PORT: 8004   
    networks:  
       - spark
    deploy:
        restart_policy:
          condition: on-failure

  spark-ui-proxy:
    image: pschatzmann/spark-ui-proxy
    command: spark-master:8004 8006 
    depends_on: 
      - spark-master
      - spark-worker
    ports:
      - 8006:8006
    networks:  
       - spark
    deploy:
      placement:
        constraints:
          - node.role == manager

⁠Limitations

The setup which has been described above is working fine if all involved systems are on Linux. Unfortunately this is not working for OS/X. In order to overcome this limitation you can used e.g. the weaveworks network driver⁠

Tag summary

Content type

Image

Digest

Size

276.9 MB

Last updated

almost 8 years ago

docker pull pschatzmann/spark