Sign inSign up

nutthaphon/hadoop-master

By nutthaphon

•Updated about 7 years ago

This is masternode in cluster is consist of namenode, resource manager and so on.

Image
0

122

nutthaphon/hadoop-master repository overview

My Hadoop cluster is consist of :

  • Master node: hdmaster (this image)
  • Slave node: hdslave1
  • Slave node: hdslave2

Add below parameters in docker-compose-cluster.yml

version: "3"

services:
  hdmaster:
    image: nutthaphon/hadoop-master:3.1.2-spark2.4.3-jdk8
    container_name: hdmaster
    hostname: hdmaster
    working_dir: "/spark"
    #user: "1000:1000"
    stdin_open: true
    tty: true
    environment:
      - CLUSTER_NAME=poc
    ports:
      # SSH
      - "2220:22"
      # Hadoop Namenode Web
      - "9870:9870"
      # Hadoop Datanode Web
      - "9864:9864"
      # YARN Resource Manager
      - "8088:8088"
      # MapReduce JobHistory Server 
      - "19888:19888"
      # Spark Context Web UI
      - "4040:4040"
      # Spark History Server]
      - "18080:18080"
    volumes:
      - hdmaster-hadoop:/hadoop
      - java-program:/java
      - spark-program:/spark
volumes:
  java-program:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /home/nutt/Docker/bigdata/java/jdk1.8.0_211
  spark-program:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /home/nutt/Docker/bigdata/spark/spark-2.4.3-bin-hadoop2.7
  hdmaster-hadoop:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /home/nutt/Docker/bigdata/hadoop-3.1.2-cluster/hdmaster

JDK, Spark and Hadoop home are outside containers to modifying some files easier.

Make sure you prepare those three home before start container

To create new or re-create container

docker-compose -f docker-compose-cluster.yml up hdmaster

To start an existing container

docker-compose -f docker-compose-cluster.yml start hdmaster

To delete container

docker-compose -f docker-compose-cluster.yml down hdmaster

To stop running container

docker-compose -f docker-compose-cluster.yml stop hdmaster

Start all three nodes services

$ docker exec -it --user hadoop --workdir /hadoop hdmaster sh -c /hadoop/start.sh
Starting namenodes on [hdmaster]
Starting datanodes
Starting secondary namenodes [hdmaster]
Starting resourcemanager
Starting nodemanagers
Show services
[ hdmaster ]
4465 WebAppProxyServer
3585 SecondaryNameNode
4610 Jps
4003 NodeManager
3363 DataNode
4555 JobHistoryServer
3100 NameNode
3854 ResourceManager
[ hdslave1 ]
768 NodeManager
921 Jps
637 DataNode
[ hdslave2 ]
770 NodeManager
923 Jps
639 DataNode

Stop all three nodes services

$ docker exec -it --user hadoop --workdir /hadoop hdmaster sh -c /hadoop/stop.sh
Stopping nodemanagers
Stopping resourcemanager
Stopping proxy server [hdmaster]
Stopping namenodes on [hdmaster]
Stopping datanodes
Stopping secondary namenodes [hdmaster]
Show services
[ hdmaster ]
2896 Jps
[ hdslave1 ]
558 Jps
[ hdslave2 ]
560 Jps

Tag summary

Content type

Image

Digest

Size

1.4 GB

Last updated

about 7 years ago

docker pull nutthaphon/hadoop-master:3.1.2-spark2.4.3-jdk8