[{"_1":2,"_171":-5,"_172":-5},"loaderData",{"_3":4,"_13":14,"_21":22,"_150":151},"root",{"_5":-5,"_6":-7,"_7":8,"_9":-7,"_10":11,"_12":-7},"currentUser","primaryEmail","siteVerification","6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI","toast","enableDev",false,"incidentStatus","routes/_layout",{"_15":-5,"_16":17,"_18":17,"_19":11,"_20":11},"username","isAskGordonEnabled",true,"isUsingNewTemplate","showMinimalLanding","showHoneypotTrap","routes/_layout.r.$namespace.$reponame",{"_23":24,"_25":-5,"_26":11,"_27":11,"_28":29,"_30":31,"_32":-5,"_33":11,"_34":35,"_84":85,"_105":106,"_143":-5,"_144":11,"_145":17,"_146":11,"_147":-5,"_148":149},"canonicalUrl","https://hub.docker.com/r/lab41/python-datatools/tags","mcpServer","canSeeDockerfile","canSeeAutobuilds","namespace","lab41","reponame","python-datatools","cloudBuildSource","isAutobuildsAvailable","repo",{"_36":37,"_38":39,"_41":42,"_43":44,"_45":11,"_46":47,"_48":49,"_50":51,"_53":31,"_28":29,"_54":55,"_56":-5,"_57":58,"_59":58,"_60":61,"_62":-5,"_63":64,"_65":66,"_67":11,"_68":29,"_69":70,"_75":17,"_76":77,"_81":29,"_82":83},"categories",[],"content_types",[40],"image","date_registered","2015-06-15T18:41:39.806203Z","description","Follow installation instructions at https://github.com/Lab41/ipython-spark-docker","is_private","last_updated","2016-03-15T00:01:42.542057Z","last_modified","2024-10-16T13:48:34.145251Z","media_types",[52],"application/vnd.docker.distribution.manifest.v1+prettyjws","name","pull_count",505,"repository_type","star_count",1,"status","status_description","active","source","collaborator_count",0,"full_description","# ipython-spark-docker\n\nThis repo provides [Docker](http://www.docker.io) containers to run:\n - [Spark](https://spark.apache.org) master and worker(s) for running Spark in standalone mode on dedicated hosts\n - [Mesos](http://mesos.apache.org)-enhanced containers for Mesos-mastered Spark jobs\n - [IPython](http://ipython.org) web interface for interacting with Spark or Mesos master via [PySpark](https://spark.apache.org/docs/0.9.0/python-programming-guide.html)\n\nPlease see the accompanying blog posts for the technical details and motivation behind this project:\n - Using Docker to Build an IPython-driven Spark Deployment\n - Triplewide Trailer, Part 1: Looking to Hitch IPython, Spark, and Mesos to Docker\n\n## Architecture\nDocker containers provide a portable and repeatable method for deploying the cluster:\n

\n $\"hadoop-docker-client$ \n

\n\n### CDH5 Tools and Libraries\n\n\n \n \n \n \n \n \n \n \n

Hbase

\n\n\n### Python Packages and Modules\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n

Pattern	NLTK	Pandas	NumPy	SciPy	SymPy	Seaborn
Cython	Numba	Biopython	Rmagic	0MQ	Matplotlib	Scikit-Learn
Statsmodels	Beautiful Soup	NetworkX	LLVM	Bokeh	Vincent	MDP

\n\n\n## Usage\n\n### Option 1. Mesos-mastered Spark Jobs\n1. Install Mesos with Docker Containerizer and Docker Images: Install a Mesos cluster configured to use the Docker containerizer, which enables the Mesos slaves to execute Spark tasks within a Docker container.\n\n A. End-to-end Installation: The script ```mesos/1-setup-mesos-cluster.sh``` uses the Python library [Fabric](http://www.fabfile.org/) to install and configure a cluster according to [How To Configure a Production-Ready Mesosphere Cluster on Ubuntu 14.04](https://www.digitalocean.com/community/tutorials/how-to-configure-a-production-ready-mesosphere-cluster-on-ubuntu-14-04). After installation, it also pulls the Docker images that will execute Spark tasks. To use:\n - Update IP Addresses of Mesos nodes in ```mesos/fabfile.py```. Find instances to change with:\n

grep 'ip-address' mesos/fabfile.py

\n\n - Install/configure the cluster:\n

./mesos/1-setup-mesos-cluster.sh

\n Optional: ```./1-build.sh``` if you prefer instead to build the docker images from scratch (rather than the script pulling from Docker Hub)\n\n B. Manual Installation: Follow the general steps in ```mesos/1-setup-mesos-cluster.sh``` to manually install:\n\n - Install mesosphere on masters\n - Install mesos on slaves\n - Configure zookeeper on all nodes\n - Configure and start masters\n - Configure and start slaves\n - Load docker images:\n

docker pull lab41/spark-mesos-dockerworker-ipython\ndocker pull lab41/spark-mesos-mesosworker-ipython

\n\n\n2. Run the client container on a client host (replace 'username-for-sparkjobs' and 'mesos-master-fqdn' below):

./5-run-spark-mesos-dockerworker-ipython.sh username-for-sparkjobs mesos://mesos-master-fqdn:5050

\n*Note: the client container will create username-for-sparkjobs when started, providing the ability to submit Spark jobs as a specific user and/or deploy different IPython servers for different users.\n\n\n\n### Option 2. Spark Standalone Mode\n**Installation and Deployment** - Build each Docker image and run each on separate dedicated hosts\n

Tip: Build a common/shared host image with all necessary configurations and pre-built containers, which you can then use to deploy each node. When starting each node, you can pass the container run scripts as User data to initialize that container at boot time

\n\n0. Prerequisites\n - Deploy Hadoop/HDFS cluster. Spark uses a cluster to distrubute analysis of data pulled from multiple sources, including the Hadoop Distrubuted File System (HDFS). The ephemeral nature of Docker containers make them ill-suited for persisting long-term data in a cluster. Instead of attempting to store data within the Docker containers' HDFS nodes or mounting host volumes, it is recommended you point this cluster at an external Hadoop deployment. Cloudera provides complete resources for [installing](http://www.cloudera.com/content/cloudera/en/documentation/cdh5/v5-0-0/CDH5-Installation-Guide/cdh5ig_cdh5_install.html) and [configuring](http://www.cloudera.com/content/cloudera/en/documentation/cdh5/v5-0-0/CDH5-Installation-Guide/cdh5ig_cdh5_cluster_deploy.html) its distribution (CDH) of Hadoop. This repo has been tested using CDH5.\n\n1. Build and configure hosts\n 1. Install Docker v1.5+, jq JSON processor, and iptables. For example, on an Ubuntu host:\n

./0-prepare-host.sh

\n 2. Update the Hadoop configuration files in ```runtime/cdh5//``` with the correct hostnames for your Hadoop cluster. Use ```grep FIXME -R .``` to find hostnames to change.\n 3. Generate new SSH keypair (```dockerfiles/base/lab41/spark-base/config/ssh/id_rsa``` and ```dockerfiles/base/lab41/spark-base/config/ssh/id_rsa.pub```), adding the public key to ```dockerfiles/base/lab41/spark-base/config/ssh/authorized_keys```.\n 4. (optional) Update ```SPARK_WORKER_CONFIG``` environment variable for Spark-specific options such as executor cores. Update the variable via a shell ```export``` command or by updating ```dockerfiles/standalone/lab41/spark-client-ipython/config/service/ipython/run```.\n 5. (optional) Comment out any unwanted Python packages in the base Dockerfile image ```dockerfiles/base/lab41/python-datatools/Dockerfile```.\n\n2. Get Docker images:\n

Option A: If you prefer to pull from Docker Hub:\n

docker pull lab41/spark-master\ndocker pull lab41/spark-worker\ndocker pull lab41/spark-client-ipython

Option B: If you prefer to build from scratch yourself:\n

./1-build.sh

If you are creating common/shared host images, this would be the point to snapshot the host image for replication.

\n\n3. Deploy cluster nodes\n

Ensure each host has a Fully-Qualified-Domain-Name (i.e. master.domain.com; worker1.domain.com; ipython.domain.com) for the Spark nodes to properly associate

\n 1. Run the master container on the master host:

./2-run-spark-master.sh

\n 2. Run worker container(s) on worker host(s) (replace 'spark-master-fqdn' below):

./3-run-spark-worker.sh spark://spark-master-fqdn:7077

\n 3. Run the client container on the client host (replace 'spark-master-fqdn' below):

./4-run-spark-client-ipython.sh spark://spark-master-fqdn:7077

\n","has_starred","hub_user","immutable_tags_settings",{"_71":11,"_72":73},"enabled","rules",[74],".*","is_automated","permissions",{"_78":17,"_79":11,"_80":11},"read","write","admin","user","storage_size",43888930818,"orgData",{"_86":87,"_88":89,"_90":91,"_92":29,"_93":94,"_95":94,"_96":94,"_97":94,"_98":99,"_100":101,"_102":94,"_103":94,"_104":17},"id","44779c8c74c011e4bea40242ac11001b","uuid","44779c8c-74c0-11e4-bea4-0242ac11001b","type","organization","orgname","full_name","","location","company","profile_url","date_joined","2014-03-11T17:53:37.440273Z","gravatar_url","https://www.gravatar.com/avatar/109973d4bd83260a40f201ae14c67544?s=80&r=g&d=mm","gravatar_email","badge","is_active","recentTags",[107],{"_108":40,"_109":110,"_111":112,"_113":114,"_86":115,"_116":117,"_46":132,"_133":110,"_134":29,"_135":52,"_53":136,"_137":138,"_139":123,"_140":132,"_141":61,"_142":17},"content_type","creator",14048,"digest","sha256:eb15b6e8f85eb407958d6774f1fae07667e3f1611b213f5e596fc7eeb055108e","full_size",2214954654,216328,"images",[118],{"_119":120,"_111":112,"_121":94,"_122":123,"_124":125,"_126":94,"_127":94,"_128":-5,"_129":130,"_59":61,"_131":-5},"architecture","amd64","features","last_pulled","2026-09-02T17:07:45.337106385Z","last_pushed","2016-03-15T00:01:42Z","os","os_features","os_version","size",2214954974,"variant","2016-03-15T00:01:41.936346Z","last_updater","last_updater_username","media_type","latest","repository",228762,"tag_last_pulled","tag_last_pushed","tag_status","v2","sbxKitMetadata","isPotentialSpam","isHubSbxKitsEnabled","displaySecurityMetadata","tagAttestations","tagsHaveAttestations",[],"routes/_layout.r.$namespace.$reponame.tags",{"_34":152,"_159":160,"_148":169,"_170":11},{"_36":153,"_38":154,"_41":42,"_43":44,"_45":11,"_46":47,"_48":49,"_50":155,"_53":31,"_28":29,"_54":55,"_56":-5,"_57":58,"_59":58,"_60":61,"_62":-5,"_63":64,"_65":66,"_67":11,"_68":29,"_69":156,"_75":17,"_76":158,"_81":29,"_82":83},[],[40],[52],{"_71":11,"_72":157},[74],{"_78":17,"_79":11,"_80":11},"tags",{"_161":58,"_162":-5,"_163":-5,"_164":165},"count","next","previous","results",[166],{"_108":40,"_109":110,"_111":112,"_113":114,"_86":115,"_116":167,"_46":132,"_133":110,"_134":29,"_135":52,"_53":136,"_137":138,"_139":123,"_140":132,"_141":61,"_142":17},[168],{"_119":120,"_111":112,"_121":94,"_122":123,"_124":125,"_126":94,"_127":94,"_128":-5,"_129":130,"_59":61,"_131":-5},[],"canDeleteTags","actionData","errors"]

lab41/python-datatools

Tags for lab41/python-datatools