Sign inSign up

crazycapivara/hadoop

By crazycapivara

•Updated almost 9 years ago

hadoop in a box!

Image
0

473

crazycapivara/hadoop repository overview

⁠hadoop in a box

run hadoop inside a docker container

⁠preface

out of the box you can run hadoop in

  • non-distributed or
  • pseudo-disributed

mode and run a MapReduce job either locally or on YARN as documented here:

It should also be possible to run multiple containers to set up fully-distributed clusters (generally by adjusting the config files), but I did not test it yet.

⁠build

# with url
$ docker build -t crazycapivara/hadoop github.com/crazycapivara/hadoop-sandbox

# or after pull
$ docker build -t crazycapivara/hadoop . 

# older version
$ docker build -f Dockerfile-hadoop-2.8.2 -t crazycapivara/hadoop:2.8.2 .

to build other versions you can set the environment variable HADOOP_VERSION in the Dockerfile

⁠pull

# latest
$ docker pull crazycapivara/hadoop

# older version (check available tags)
$ docker pull crazycapivara/hadoop:2.8.2

⁠run

# check version
$ docker run --rm crazycapivara/hadoop hadoop version

# run MapReduce example locally as standalone operation
$ docker run --rm crazycapivara/hadoop examples/map-reduce.sh

# start HDFS and run MapReduce example locally as pseudo-distributed operation
$ docker run --rm -e START_DFS=yes crazycapivara/hadoop examples/map-reduce.sh

# start HDFS and YARN
# and run MapReduce example as pseudo-distributed operation on YARN
$ docker run --rm -P \
    -e START_DFS=yes \
    -e START_YARN=yes \
    crazycapivara/hadoop examples/map-reduce.sh

# start HDFS and an interactive bash session
$ docker run --rm -it -e START_DFS=yes crazycapivara/hadoop

⁠environment variables

  • START_DFS:
    set to yes to set up HDFS and run hadoop's start-dfs.sh (defaults to no)
  • START_YARN:
    set to yes to set up YARN and run hadoop's start-yarn.sh (defaults to no)

⁠ssh

# run in detached mode
$ docker run -t -d -p 10022:22 -e START_DFS=yes --name hadoop_sandbox crazycapivara/hadoop

# get key
$ docker cp hadoop_sandbox:/root/.ssh/id_rsa ~/.ssh/

# connect via ssh
$ ssh locahost -p 10022 -i ~/.ssh/id_rsa

⁠web interfaces

# publish port(s)
$ docker run --rm -it \
    -p 50070:50070 -p 8088:8088 \
    -e START_DFS=yes -e START_YARN=yes crazycapivara/hadoop

browse the web interface for the NameNode

browse the web interface for the ResourceManager

Tag summary

Content type

Image

Digest

Size

584.9 MB

Last updated

almost 9 years ago

docker pull crazycapivara/hadoop