labianchin/hadoop repository overview

⁠Yak shaving an elefant

You want to try out hadoop 2.3? Go to the zoo and shave a yak⁠. Or simply just use docker⁠.

docker run -it --rm labianchin/hadoop /etc/bootstrap.sh -bash

⁠Hive

This image is a fork of sequenceiq/hadoop-docker with the addition of hive.

⁠Play with hive

$ export PATH=$HADOOP_PREFIX/bin:$HIVE_HOME/bin:$PATH
$ cat > /tmp/pokes_foo.csv << EOL
foo,bar
1,"raa"
2,"quee"
3,"other val"
EOL
$ hive
hive> CREATE TABLE pokes (foo INT, bar STRING) row format delimited fields terminated by ',';
hive> LOAD DATA LOCAL INPATH '/tmp/pokes_foo.csv' OVERWRITE INTO TABLE pokes;
hive> SELECT * FROM pokes;

⁠Testing

You can run one of the stock examples:

cd $HADOOP_PREFIX
 # run the mapreduce
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar grep input output 'dfs[a-z.]+'

 # check the output
bin/hdfs dfs -cat output/*

⁠too long didn't read

I had problems installing hadoop 2.3 and by googling i stumbled upon this email thread⁠, which references an alternative hadoop docs⁠ deployed on github.

By following that description i run into an other issue: hadoop is delivered with 32 bit native libraries. No big deal ...

⁠Hadoop native libraries

Of course there is an official Native Libraries Guide⁠ it instructs you to simple download the sources and mvn package. But than you face a new issue: missing protobuf. Eeeasy ...

⁠Protobuf 2.5

Unfortunately yum install protobuf installs an older 2.3 version, which is close but no cigar. So you download protobuf source, and ./configure && make && make install

To succeed on that one you have to install a couple of development packages, and there you go.

⁠Bintray

I wanted to save you those steps so created a binary distro of the native libs compiled with 64 bit CentOS. So I created Bintray r̨epo⁠. Enjoy

⁠Automate everything

As I'm an automation fetishist, a Docker file was created, and released in the official docker repo⁠

⁠Additional classpath

It is possible to add libraries to the classpath of a yarn container, by passing an environment variable as follows:

ACP_SRC=http://repo1.maven.org/maven2/org/apache/mahout
ACP=$ACP_SRC/mahout-core/0.9/mahout-core-0.9.jar,$ACP_SRC/mahout-math/0.9/mahout-math-0.9.jar,$ACP_SRC/mahout-integration/0.9/mahout-integration-0.9.jar
docker run -e "ACP=$ACP" -t -d -h sandbox --name sandbox  my-sandbox /etc/bootstrap.sh -bash

The ACP environment variable is a comma separated list of urls where the needed resources can be downloaded from. Libraries are copied to the $HADOOP_PREFIX/share/hadoop/common/ folder.