Public | Automated Build

Last pushed: 7 months ago
Short Description
SURFsara Hadoop service client as a Docker container
Full Description

hathi-client

This repository contains client configuration for the SURFsara Hadoop cluster
Hathi. At the moment it contains configuration for Hadoop 2.7.1 and Pig 0.16.0
and Spark 2.1.1.

Prerequisites

This software is tested on Linux and OSX. On Linux you need to make sure that
Git, Java 7 and the Kerberos client libraries are installed. On OSX these
should already be installed.

Debian-based Linux (Debian, Ubuntu, Mint):

apt-get install git openjdk-7-jre-headless krb5-user

Enterprise Linux (Redhat, CentOS, Fedora):

yum install git java-1.7.0-openjdk krb5-workstation

Note that when using the Oracle JDK (e.g. OSX) you will need to install the
Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files
for your specific JVM version.

For Oracle Java 7 they can be found here:
http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html

For Oracle Java 8:
http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

Usage

The first time you need to download the official Hadoop/Pig/Spark software from
Apache and put the SURFsara configuration in the right location. We provide a
helper script that will do this automatically:

git clone --depth 1 https://github.com/sara-nl/hathi-client
/path/to/hathi-client/bin/get.sh hadoop
/path/to/hathi-client/bin/get.sh pig
/path/to/hathi-client/bin/get.sh spark

Whenever you want to use the cluster you need to perform the following once per
session.

1) Setup the environment:

eval $(/path/to/hathi-client/bin/env.sh)

(You can add this line to your ~/.profile so that it is run automatically on
login).

2) Now you can authenticate using Kerberos:

kinit USERNAME

And use the Hadoop, Pig and Spark utilities:

hdfs dfs -ls /

yarn jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar pi 5 5

spark-submit --class org.apache.spark.examples.SparkPi \
             --master yarn  --deploy-mode cluster \
             $SPARK_HOME/lib/spark-examples*.jar 10

Browser setup

In order to authenticate to the webapps you will need to use Firefox and alter
the about:config (promise to be careful). Search for the key
network.negotiate-auth.trusted-uris and add the value hathi.surfsara.nl.

In addition, Firefox needs to be aware of the Kerberos setup. For this the
Kerberos configuration conf/krb5.conf needs to be placed in the right
location (you will need root access for this). Note that if you work with
different Kerberos realms you can also add the KDC configuration (the
[realms] section) from the hathi-client file to any existing Kerberos
configuration file. To copy (and overwrite any existing files) the
configuration to the correct location:

For OSX:

cp git/hathi-client/conf/krb5.conf $HOME/Library/Preferences/edu.mit.Kerberos

For Linux:

sudo cp git/hathi-client/conf/krb5.conf /etc/

The resource manager of the cluster can then be found at
http://head05.hathi.surfsara.nl.

The namenode of the cluster is located at http://head02.hathi.surfsara.nl.

Support

For more information about the SURFsara Hadoop cluster see
https://userinfo.surfsara.nl/systems/hadoop.

For any questions using Hadoop at SURFsara contact the SURFsara
helpdesk
.

Docker Pull Command
Owner
surfsara
Source Repository