martinmares/hdpkub repository overview

⁠Hortonworks Hadoop cluster deployment with Kubernetes

The point is to get the Hadoop cluster with needed components, running as docker container at Kubernetes cluster.

Files: Dockerfile⁠ ambari.repo⁠ bp.json⁠ map.json⁠ start.sh⁠ must be placed at accessible from hub.docker.com git repository.

⁠1. Create Docker Automated Build.

⁠1.1 Register at hub.docker.com and link your account to your git repository while creating Automated Build

Register at hub.docker.com and link account to github, Create Automated Build, setting git repo smaryn/hdpkub⁠ as a "Source Repository"

⁠1.2 Go to the "Build Settings" tab and click "Trigger" button to run docker image building.

The build may take a while and should be finished successfully, you can always check the build status at Build Details tab

⁠2. Prepare kubectl utility to be able to connect to Kubeletes master (198.18.8.161):

⁠2.1 Download kubectl binary file from corresponding resource and place it somewhere in the $PATH:

⁠For Linux workstation:

wget -c https://storage.googleapis.com/kubernetes-release/release/v1.1.8/bin/linux/amd64/kubectl -O /usr/local/bin/kubectl

⁠For Mac OS X workstation:

wget -c https://storage.googleapis.com/kubernetes-release/release/v1.1.8/bin/darwin/amd64/kubectl -O /usr/local/bin/kubectl

⁠2.2 Configure kubectl utility:

⁠Set Kubeletes cluster:

kubectl config set-cluster ubuntu --server=http://198.18.8.161:8080 --insecure-skip-tls-verify=true

⁠Set context:

kubectl config set-context ubuntu --cluster=ubuntu --user=ubuntu

⁠Add user for created cluster and password:

kubectl config set-credentials ubuntu --username=admin --password=HAKlQaGHHmU6W0W

⁠Inform kubectl to use desired context:

kubectl config use-context ubuntu

⁠Check kubectl configuration:

kubectl config view

⁠Check connection to configured cluster:

kubectl get nodes

⁠3. Kubernetes pod and service by prepared yaml files

⁠3.1 Create Kubernetes pod for Hadoop container(s) using hdp.yaml⁠:

kubectl create -f kube/hdp.yaml

⁠3.2 Create Kubernetes service for Hadoop using hdp-service.yaml⁠

⁠At this stage in the command output you should notice the TCP port which is assigned for Ambari server management:

kubectl create -f kube/hdp-service.yaml

⁠3.3 Check created service status:

kubectl logs hdpkub

⁠3.4 Check cluster deployment

⁠Connect to Ambari Management Console

Open web browser and type Kubernetes node IP address and TCP port noticed at 2.3 stage. Use default username and password (admin) to see how cluster is deployed.

⁠3. Cluster termination and Kubernetes pod and service removal

⁠3.1 Delete Kubernetes service:

kubectl delete -f kube/hdp-service.yaml

⁠3.2 Delete Kubernetes pod:

kubectl delete -f kube/hdp.yaml

⁠3.3 Obtain the root shell at the Kubernetes node and remove unused Docker containers and images

List all containers:

docker ps -a

Delete unused containers by ID:

docker rm -f <ID1> <ID2> <...>

List all images:

docker images

Delete uneeded images by ID:

docker rmi -f <ID1> <ID2> <...>