Public | Automated Build

Last pushed: a year ago
Short Description
basis build
Full Description

Stream-Processing - FLINK/STORM/SPARK

<div>
<IMG title="Stay cool! :)" alt="Orignal Ressource: http://vignette2.wikia.nocookie.net/fallout/images/b/bf/Vault_Boy_Dock_Icon_by_oloff3.png/revision/20110820161751" style="float: right;margin-left: 10px;" align="right" alt="Banner" border="0" src="https://github.com/FutureApp/stream-processing/raw/master/pics/FallOutBoy.png">
<p>
This repo contains the source code for the bachelor thesis named "Streamprocessing im Big Data Umfeld: Datenintegrität und Fehlertoleranz". Goal of the thesis: How fault-tolerant and what kind of data integrity are guaranteed by these technologies. With this GIT-Repo you will able to run three common streaming-technologies named apache-flink, apache-storm and apache-spark throw docker. All three technologies are running in cluster mode out of the box. This project is inspired by the benchmark of yahoo for these technologies. To see what they do, follow and read the links on the next chapter.
</p>
</div>

YAHOO - Benchmark

Article and Results - Yahoo Benchmark

https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-computation-engines-at

Git-Repo - Yahoo Benchmark

https://github.com/yahoo/streaming-benchmarks

Pre-Requisites

Install Docker and that's all. =)

Usage

First of all:
All hubs are listen to <b>localhost:9000</b>.
If you are running the commandos first time, it will take a longer time because all the images need to be downloaded first.

Assumed that you are running a flink-cluster and you want to navigate to the flink-hub / flink-web-ui just enter localhost:9000 in your browser to access this ressource. This URL( localhost:9000) is pre-configured for all technologies(flink,storm,spark)

Well, you have a couple of functions to interact with your docker-x-cluster.

All methods for interaction. Look at interact/cluster_service.sh:

"stop all container" "kill all container" "kill container #NameOfContainer" "kill everything" "build all images"

Particular mention should be made of:

"run cluster flink" 1 #NumberOfWorkers
"run cluster storm" 1 #NumberOfWorkers
"run cluster spark" 1 #NumberOfWorkers

Run a flink-cluster with 1 master and 2 workers:

Lets say you want to have a cluster with 1x master and 2x worker.
To do that enter ->

bash cluster-service.sh "run cluster flink" 1 2

Run a storm-cluster with 1 master and 2 workers:

Lets say you want to have a cluster with 1x master and 2 worker. Keep in mind that one worker will start automaticaly on the master.
To do that enter ->
bash cluster-service.sh "run cluster storm" 1 2

Now, you need to wait around 2 minutes before everything is ready. Time to take a coffee!

Run a spark-cluster with 1 master and 2 workers:

Lets say you want to have a cluster with 1x master and 2 worker. Keep in mind that one worker will start automaticaly on the master.
To do that enter ->
bash cluster-service.sh "run cluster spark" 1 2

Stop all containers & Remove all containers:

bash cluster-service.sh "kill everything"

Docker Pull Command
Owner
futureapplications
Source Repository