Sign inSign up

lyingbo/hadoop

By lyingbo

•Updated over 6 years ago

Run Hadoop Cluster within Docker Containers

Image
2

322

lyingbo/hadoop repository overview

⁠Run Hadoop Cluster within Docker Containers

⁠A. 3 Nodes Hadoop Cluster
⁠1. pull docker image
sudo docker pull lyingbo/hadoop:3.2.0
⁠2. clone github repository
git clone https://github.com/lyingbo/hadoop-cluster-docker.git
⁠3. start container
cd hadoop-cluster-docker
sudo ./start-container.sh

output:

start hadoop-master container...
start hadoop-slave1 container...
start hadoop-slave2 container...
root@hadoop-master:/#
  • start 3 containers with 1 master and 2 slaves
  • you will get into the /root directory of hadoop-master container
⁠4. start hadoop
start-all.sh

output

Starting namenodes on [hadoop-master]
hadoop-master: Warning: Permanently added 'hadoop-master,172.18.0.2' (ECDSA) to the list of known hosts.
Starting datanodes
hadoop-slave2: Warning: Permanently added 'hadoop-slave2,172.18.0.4' (ECDSA) to the list of known hosts.
hadoop-slave1: Warning: Permanently added 'hadoop-slave1,172.18.0.3' (ECDSA) to the list of known hosts.
Starting secondary namenodes [hadoop-master]
Starting resourcemanager
Starting nodemanagers
⁠5. run wordcount
run-wordcount.sh

output

input file1.txt:
Hello Docker

input file2.txt:
Hello Hadoop

input file3.txt:
Hello MapReduce

wordcount output:
Docker  1
Hadoop  1
Hello   3
MapReduce       1
⁠B. Arbitrary size Hadoop cluster
⁠1. clone github repository
do 2 like section A
⁠2. rebuild docker image
sudo ./resize-cluster.sh 5
  • specify parameter > 1: 2, 3..
  • this script just rebuild hadoop image with different workers file, which pecifies the name of all nodes
⁠3. start container
sudo ./start-container.sh 5
  • use the same parameter as the step 2
⁠4. run hadoop cluster
do 4~5 like section A

Tag summary

Content type

Image

Digest

Size

528.1 MB

Last updated

over 6 years ago

docker pull lyingbo/hadoop:3.2.0