adolphlwq/docker-spark repository overview

⁠Spark docker image

⁠Info

• Spark 1.6.0
• Mesos 0.28.1
• Jre 1.7

⁠Usage

• SPARK_HOME :/usr/local/spark-1.6.0-bin-hadoop2.6

• Try on docker image

  sudo docker run -it --name spark-try --net host adolphlwq/docker-spark bash
  

  Copy

⁠Use it on Mesos Cluster as Mesos Spark executor

Refer Spark on Mesos⁠ And Spark configuration⁠ for more details.

• Instance SparkConf

  conf = SparkConf()
  conf.setMaster(mesos_endpoint)
  conf.set("spark.mesos.executor.docker.image", "adolphlwq/docker-spark")
  conf.set("spark.mesos.executor.home", "/usr/local/spark-1.6.0-bin-hadoop2.6")
  

  Copy

• Instance SparkContext

  sc = SparkContext(conf=conf)
  below is your code
  ...
  

  Copy

⁠TODOs

No todos. Any issues is welcome.