Public Repository

Last pushed: 2 years ago
Short Description
Hadoop cluster, single and multi-node
Full Description

Installation

  1. open jdk7
  2. hadoop 2.6.0
  3. ssh

Hadoop Single Cluster

docker pull rickhw/hadoop:single
docker run -h hadoop-single -p 8088:8088 -p 50070:50070 -it rickhw/hadoop:single /bin/bash
service ssh start
start-dfs.sh
start-yarn.sh

Hadoop ResourceManager Web: http://{docker-machine-ip}:8088/
NameNode HDFS Web: http://{docker-machine-ip}:50070/

  • Updated: run these images with Linux instead of OS X

Multi-Node Cluster

1. clone following images: rickhw/hadoop:namenode, rickhw/hadoop:datanode

docker pull rickhw/hadoop:namenode
docker pull rickhw/hadoop:datanode

2. Start Hadoop-Master

docker run -h hadoop-master -p 8088:8088 -p 50070:50070 -it rickhw/hadoop:master /bin/bash
service ssh start

3. Start Data Node 1-3

docker run --name hadoop-datanode1 -h hadoop-datanode1 --link hadoop-master -it rickhw/hadoop:datanode /bin/bash
docker run --name hadoop-datanode2 -h hadoop-datanode2 --link hadoop-master -it rickhw/hadoop:datanode /bin/bash
docker run --name hadoop-datanode3 -h hadoop-datanode3 --link hadoop-master -it rickhw/hadoop:datanode /bin/bash

# Start SSH Service of DataNode[1-3]
service ssh start

4. Add DataNode Host to Master

# Go to hadoop-master, edit /etc/hosts, add following:
172.17.0.3      hadoop-datanode1
172.17.0.4      hadoop-datanode2
172.17.0.5      hadoop-datanode3

# Test Connect from master to data node
ssh hadoop-datanode1
ssh hadoop-datanode2
ssh hadoop-datanode3

5. Start DFS and YARN in Hadoop-Master

start-dfs.sh
start-yarn.sh

Hadoop ResourceManager Web: http://{docker-machine-ip}:8088/
NameNode HDFS Web: http://{docker-machine-ip}:50070/

Docker Pull Command
Owner
rickhw

Comments (0)