Sign inSign up

lee212/e222

By lee212

•Updated over 8 years ago

Docker Hadoop with Examples

Image
0

286

lee212/e222 repository overview

⁠Interactive shell

PageRank sudo docker run -it lee212/e222:pagerank /etc/bootstrap.sh -bash

Statistics sudo docker run -it lee212/e222:exer1 /etc/bootstrap.sh -bash

⁠Example: Pagerank

cd /cloudmesh/pagerank
./compileAndExecHadoopPageRank.sh PageRankDataGenerator/pagerank5000g50.input.0 5000 1

*********************************************
*           Hadoop PageRank                 *
*********************************************
Hadoop CreateGraph starts...

...
⁠Result
head output.pagerank/part-r-00000
0       2.9999999999999997E-5
1       2.9999999999999997E-5
2       2.9999999999999997E-5
3       2.9999999999999997E-5
4       2.9999999999999997E-5

⁠Path for Examples

/cloudmesh/

⁠Sample Run

⁠Getting Min/Max/Avg/Std values from a single input file

export HADOOP_CLASSPATH=`$HADOOP_PREFIX/bin/hadoop classpath
mkdir /root/exer1/dest
javac -classpath $HADOOP_CLASSPATH -d /root/exer1/dest root/exer1/src/exercise/MinMaxAvgStd.java
jar -cvf exer1.jar -C /root/exer1/dest/ /root/exer1/
export PATH=$PATH:/$HADOOP_PREFIX/bin
hadoop fs -mkdir exer1_input
hadoop fs -put input_data/* exer1_input
hadoop fs -ls exer1_input/
hadoop jar exer1.jar exercise.MinMaxAvgStd exer1_input/data_1000.txt exer1_output_1000

⁠Result

doop fs -cat exer1_output_1000/part-r-00000

⁠With Multiple Files

hadoop jar exer1.jar exercise.MinMaxAvgStd exer1_input/ exer1_output_all

⁠Result

doop fs -cat exer1_output_all/part-r-00000

⁠Dockerfile

https://github.com/lee212/docker-hadoop⁠

Tag summary

Content type

Image

Digest

Size

896.1 MB

Last updated

over 8 years ago

docker pull lee212/e222:hadoop2.7.5-dev