tcjennings/spark-docker repository overview

⁠Environment

• SPARK_NODE_MODE: "master" or "slave". Defaults to "slave".
• HOSTNAME_COMMAND: Command to execute within container to determine value for SPARK_PUBLIC_DNS name. Defaults to hostname which will return the container ID.

⁠Reverse Proxy

Spark is configured within the container to use "reverse proxy" whereby the Master node proxies all web-ui requests for workers and applications. Only the master node, then, needs to be accessible by name and port from a host external to the Docker network. Use SPARK_PUBLIC_DNS explicitly or via HOSTNAME command to help build the Spark reverse proxy configuration correctly.

Reverse Proxy makes it possible to run Spark in a docker swarm overlay network environment where only the Master needs a public hostname and a mapped port.

⁠Example Compose File

@docker-compose.yml

version: '3.2'
networks:
  default:
    driver: overlay
services:
  spark-master:
    image: tcjennings/spark-docker:latest
    environment:
       HOSTNAME_COMMAND: curl http://169.254.169.254/latest/meta-data/public-hostname
       SPARK_NODE_MODE: master
    ports:
       - "8080:8080"
       - "7077:7077"
    deploy:
       restart_policy:
          condition: on-failure
       placement:
          constraints:
             - node.role == manager
  spark-worker:
    image: tcjennings/spark-docker:latest
    environment:
       HOSTNAME_COMMAND: curl http://169.254.169.254/latest/meta-data/public-hostname
       SPARK_MASTER_HOST: spark-master
    deploy:
       mode: replicated
       replicas: 3
       restart_policy:
          condition: on-failure