atline/dpm repository overview

⁠Dapeng

Dapeng is a distributed scheduler service aims to dispatch tasks to proper resources.

⁠Reference Manual

You can find the latest Dapeng reference manual in the doc folder of this repository.
This README file only contains basic setup instructions.

⁠Ecosystem

⁠Architecture

NOTE: For detail data flow explanation you can go to the reference manual.

⁠Build Dapeng

Dapeng is built using Sbt⁠. To build Dapeng, run:

sbt distclean release

You can also use dpbuild docker image to build dapeng:

docker pull atline/dpbuild
docker run --rm -v /path/to/dapeng_repo:/dpsrc atline/dpbuild

⁠Install Dapeng

⁠1. System Requirements

NOTE: You can use curl https://get.docker.com/ | sudo sh to install docker or visit docker official website⁠ to get the latest guide.

⁠2. Install Dapeng Components

User can use boot script to install/configure/start dapeng components.

(You can use curl -O https://bitbucket.sw.nxp.com/projects/MSVC/repos/dapeng/raw/docker/script/dp.init to get the dp.init. If not accessable, you can also get it from sourcecode of dapeng, located at docker/script/dp.init)

When use dp.init to install dapeng component, an environment variable must be set to determine the version to be installed, this environment variable just needed when install.

$ export DP_VERSION=$version

Boot script command usage:

$ dp.init -c $component --install #  install specified component to your server
$ dp.init -c $component --configure #  reconfigure specified component
$ dp.init -c $component --start #  start specified component if not auto start it during install

NOTE: dp.init internal use dpinit docker image to install/configure/start dapeng components:

It supports five kinds of components, belongs to server & client parts:

⁠2.1. Server Parts

⁠component: inf

This component is a virtual one to install/configure/start zookeeper, redis, mesos-master services.

$ dp.init -c inf --install #  install component to your server
$ dp.init -c inf --configure #  reconfigure component
$ dp.init -c inf --start #  start component if not auto start it during install

⁠component: cas

This component is for cassandra instance, dps component will default setup cassandra instance on its same server, however, we set consistent level of cassandra to QUORUM, this means we at least need 3 replicas to tolerate one cassandra node failure. Meanwhile, if one node failure, we still need 3 nodes to continue afford db service, then at least 4 cassandra nodes need to be set in our environments.

If we did not set enough dps (distributed scheduler instance), then we need to set up some standalone cassandra instance to make cluster tolerate failure.

$ dp.init -c cas --install #  install component to your server
$ dp.init -c cas --configure #  reconfigure component
$ dp.init -c cas --start #  start component if not auto start it during install

⁠component: dps

This component is for dapeng scheduler server, it affords resource router service for user request.

$ dp.init -c dps --install #  install dps to your server
$ dp.init -c dps --configure #  reconfigure dps
$ dp.init -c dps --start #  start dps if not auto start it during install

⁠component: dpm

This component is for dapeng mesos framework, it affords the adapter between mesos & dapeng scheduler.

$ dp.init -c dpm --install #  install dpm to your server
$ dp.init -c dpm --configure #  reconfigure dpm
$ dp.init -c dpm --start #  start dpm if not auto start it during install

⁠2.2. Client Parts

⁠component: dpc

This component is for dapeng client, it affords interface for user application to use dapeng scheduler service, meanwhile, also afford path for scheduler to call user application.

$ dp.init -c dpc --install #  install dpc to your server
$ dp.init -c dpc --configure #  reconfigure dpc
$ dp.init -c dpc --start #  start dpc if not auto start it during install

(The End)