Sign inSign up

mangalaman93/spark

By mangalaman93

•Updated over 9 years ago

Spark image container spark binaries as well as mesos lib for running spark with mesos

Image
0

358

mangalaman93/spark repository overview

⁠dspark

Run spark in docker containers. For more details, checkout wiki⁠

⁠Run spark shell

docker run --net=compose_default --rm -it mangalaman93/executor:2.1.0 /opt/spark/bin/spark-shell --master mesos://zk://$HOST_IP:2181,$HOST_IP:2182,$HOST_IP:2183/mesos --conf spark.mesos.executor.docker.image=mangalaman93/spark --conf spark.mesos.executor.home=/opt/spark
val textFile = sc.textFile("/opt/spark/README.md")
textFile.map(line => line.split(" ").size).reduce((a, b) => if (a > b) a else b)

⁠Todo

  • Avoid running shuffle service in the host network mode
  • Avoid running namenode and datanode services in the host network mode
  • Setup an HA cluster of HDFS with at least 3 datanodes

⁠Issues/Enhancements

  • Hostname look up issue with mesos
  • Fix spark tmp directory as volume only when necessary

Tag summary

Content type

Image

Digest

Size

310.8 MB

Last updated

over 9 years ago

docker pull mangalaman93/spark