Sign inSign up

jcai/roar-deployer

By jcai

•Updated almost 10 years ago

Automating deploy big data software to cluster

Image
0

1.3K

jcai/roar-deployer repository overview

⁠roar-deployer

⁠简述

本项目主要是一键部署大数据软件，支持ubuntu 16.04 LTS 操作系统，实现集群化对操作系统进行设置、软件安装、远程命令支持。

⁠安装步骤

  1. 前期准备

  • 新建集群配置，假设集群名称为me,在config/deploy下建立自己集群文件me.rb以及文件夹me，示例见：config/deploy/keyten

  • 配置NTP服务器,如果有NTP服务器则可忽略此步

  • 配置集群机器的hosts文件，config/deploy/me/hosts;示例见: config/deploy/keyten/hosts

  • 更改本机的/etc/hosts文件,以方便操作远程服务器

  • 配置ubuntu机器使用的镜像源设置文件,config/deploy/me/sources.list;示例见: config/deploy/keyten/sources.list

  • 配置当前集群，修改config/deploy/me.rb;示例见: config/deploy/keyten.rb

  • 设置NGINX文件服务器，供下载软件使用,软件目录包含如下:

    hadoop-2.5.2.tar.gz hadoop-client-2.5.2.jar hbase-0.98.20-hadoop2-bin.tar.gz server-jre-7u79-linux-x64.tar.gz

  • 安装docker.

      sudo apt-get install docker.io
  sudo usermod -aG docker $(whoami)

    退出当前用户，重新登录系统. 如果无互联网环境，需要导入roar-deployer的docker文件:

      docker load < roar-deployer.tgz
  1. 启动文件服务器
  docker run -d -p 80:80 -v /opt/software:/usr/share/nginx/html/software --name file.roar jcai/roar-deployer:file
  1. 运行管理系统
  bin/manager
  1. 设置操作系统，(用户支持sudo权限,假设用户名为ubuntu)
ROAR_USER=ubuntu cap me ubuntu:setup:user ubuntu:setup:hosts ubuntu:setup:init ubuntu:setup:ntp ubuntu:setup:jdk

操作完之后,通常需要重启服务器,下面命令中的s2.roar为你要重启的机器 sh ROAR_USER=ubuntu cap me -z s2.roar ubuntu:reboot

  1. 配置GIT服务器，把当前配置内容push到git服务器中 初始化git服务器
cap me git:setup

添加内容比较提交到远程服务器

git add .
git commit -a -m 'add my configruation files'
git remote add me [email protected]:/opt/apps/roar_deployer.git
git push me master
  1. 部署hadoop
cap me hadoop:setup
  1. 部署hbase
cap me hbase:setup
  1. 部署配置
cap me deploy
  1. 操作hadoop集群

  • 启动hadoop集群

    cap me hadoop:start

  • 启动yarn

    cap me hadoop:start_yarn

  • 停止yarn

    cap me hadoop:stop_yarn

  • 停止hadoop

    cap me hadoop:stop

  • 访问hdfs集群

    cap me hadoop:fs[-ls /]
  1. 操作hbase集群

  • 启动hbase集群

    cap me hbase:start

  • 停止hbase集群

    cap me hbase:stop

  • 打开hbase的shell

    cap me hbase:shell
  1. 常用命令
cap deploy                         # Deploy a new release
cap git:setup                      # setup git server
cap hadoop:data:start              # start data server
cap hadoop:data:stop               # stop data server
cap hadoop:fs[fs_cmd]              # exexcute fs shell
cap hadoop:name:format             # format name server
cap hadoop:name:start              # start name server
cap hadoop:name:stop               # stop name server
cap hadoop:setup                   # setup hadoop system
cap hadoop:start                   # start all hadoop component
cap hadoop:start_yarn              # start yarn cluster
cap hadoop:stop                    # stop all hadoop component
cap hadoop:stop_yarn               # stop yarn cluster
cap hadoop:yarn:start_manager      # start yarn resource manager
cap hadoop:yarn:start_node         # start yarn node manager
cap hadoop:yarn:stop_manager       # stop yarn resource manager
cap hadoop:yarn:stop_node          # stop yarn node manager
cap hbase:master:start             # start master server
cap hbase:master:stop              # stop master server
cap hbase:region:start             # start region server
cap hbase:region:stop              # stop region server
cap hbase:setup                    # setup hbase system
cap hbase:shell                    # execute hbase shell on first master server
cap hbase:start                    # start all hbase component
cap hbase:stop                     # stop all hbase component
cap hbase:zk:start                 # start zookeeper server
cap hbase:zk:stop                  # stop zookeeper server
cap hive:beeline                   # execute beeline shell on first hive
cap hive:init                      # init hive component
cap hive:setup                     # setup hive system
cap hive:start                     # start hive server
cap hive:stop                      # stop hive server
cap install                        # Install Capistrano, cap install STAGES=staging,production
cap java:bin[command]              # execute command under java/bin directory
cap java:gcutil[process]           # view gcutil of java process
cap java:vmargs[process]           # view vmargs of java process
cap roar:install_scala             # install scala library
cap roar:link                      # link roar product files
cap roar:update                    # update roar system
cap ubuntu:poweroff                #
cap ubuntu:reboot                  #
cap ubuntu:setup:hosts             # update hosts
cap ubuntu:setup:init              # setup ubuntu system
cap ubuntu:setup:jdk               # install jdk
cap ubuntu:setup:ntp               # setup ntpdate
cap ubuntu:setup:user              # setup roar user
cap uptime                         # Report Uptimes

Tag summary

Content type

Image

Digest

Size

35.1 MB

Last updated

almost 10 years ago

docker pull jcai/roar-deployer