neshkeev/hadoop repository overview

⁠A Hadoop docker image packaged by neshkeev

DISCLAMER:

⁠TLDR

There is a functional docker-compose.yml⁠ file in the repository. Run the application using it as shown below:

curl -sSL https://raw.githubusercontent.com/neshkeev/containers/master/hadoop/examples/simple/docker-compose.yml > docker-compose.yml
docker compose up

⁠Overview of Apache Hadoop⁠

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.

⁠Goal

The packaging is ideal for demo, testing or teaching purposes, because it contains full-blown Hadoop Distribution inside a single docker image. Client applications can perform basic operations using Hadoop MapReduce and the Hadoop Distributed File System (HDFS).

⁠Architectures

The docker image can be used to run a Hadoop Cluster on the following architectures:

• amd64⁠,
• aarch64/arm64⁠.

⁠Tags

⁠Configuration

In order to cofigure a Hadoop cluster there are two options that can be used in combination.

⁠Mount config files

One can mount prepared config files:

• core-site.xml -> /opt/hadoop/etc/hadoop/core-site.xml
• hdfs-site.xml -> /opt/hadoop/etc/hadoop/hdfs-site.xml,
• yarn-site.xml -> /opt/hadoop/etc/hadoop/yarn-site.xml,
• mapred-site.xml -> /opt/hadoop/etc/hadoop/mapred-site.xml,
• and so on

⁠Environment Variables

Existing config files can be amended with environment variables.

In order to amend a config properties one needs to construct an environment variable which name matches the following pattern: PREFIX + config.name.

For example, the value for fs.defaultFS should be stored in the CORE-SITE.XML_fs.defaultFS environment variable like this:

CORE-SITE.XML_fs.defaultFS=hdfs://namenode:9000

The prefix is derived from the file the config belongs to. See the examples' docker-compose.yml⁠ files for references.

⁠Precedence

The values from environment variables takes precedence over the mounted config files.

⁠Examples

⁠Compute Pi on a single node Hadoop cluster

1. Download docker-compose.yml⁠:

curl -sSL https://raw.githubusercontent.com/neshkeev/containers/master/hadoop/examples/multi-node-cluster/docker-compose.yml > docker-compose.yml

1. Start the docker compose services:

docker compose up

1. Enter the hadoop docker compose service:

docker compose exec -it hadoop bash

1. Run the Pi computation task:

yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar pi 10 15

1. Explore the workload on the web interface:
   • NameNode: http://localhost:9870/explorer.html⁠
   • ResourceManager: http://localhost:8088⁠
2. Exit the hadoop docker container:

exit

1. Stop the cluster:

docker compose down

⁠Run a Map Reduce task on a multi-node Hadoop cluster

Run the grep example that comes with the Hadoop distribution by default:

1. Download docker-compose.yml⁠:

curl -sSL https://raw.githubusercontent.com/neshkeev/containers/master/hadoop/examples/multi-node-cluster/docker-compose.yml > docker-compose.yml

1. Start the docker compose services:

docker compose up

1. Wait until all the services are healthy:

docker compose ps

1. Enter the hadoop docker compose service:

docker compose exec -it hadoop bash

1. Create a directory for the user that is required to execute MapReduce jobs:

hdfs dfs -mkdir -p /user/hadoop

1. Create the input directory:

hdfs dfs -mkdir -p input

1. Copy the input files into HDFS:

hdfs dfs -put -f /opt/hadoop/etc/hadoop/*.xml input

1. Start the Map Reduce job:

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar grep input output 'dfs[a-z.]+'

1. Explore the workload on the web interface:
   • NameNode: http://localhost:9870/explorer.html⁠
   • ResourceManager: http://localhost:8088⁠
2. Review the results:

hdfs dfs -cat output/*

1. Copy the results to the local file system:

hdfs dfs -get output ~/output

1. Analyze the results:

find ~/output

1. Remove the results:

hdfs dfs -rm -r output

1. Exit the resourcemanager docker container:

exit

1. Stop the cluster:

docker compose down

⁠Connect to Hadoop Services Over JMX

1. Start a Hadoop Cluster

curl -sSL https://raw.githubusercontent.com/neshkeev/containers/master/hadoop/examples/jmx/docker-compose.yml > docker-compose.yml
docker compose up -d

1. Open jconsole (or visualvm):

jconsole

1. Create a remote JMX Connection:
   • NameNode: localhost:59870
   • DataNode: localhost:59864
   • ResourceManager: localhost:58088
   • NodeManager: localhost:58042
   • HistoryServer: localhost:59888