Public | Automated Build

Last pushed: 2 months ago
Short Description
image for mesos exector
Full Description

linkerProcessorSample Intro

A sample code for processing infomation from Kafka and store to Cassandra using Apache Spark

Tag introductions

tag v0.1

realizations:

  1. collect info from kafka topic
  2. calculate cpu total usage as well as each cpu usage
  3. save cpu usage to cassandra
  4. test successfully on Mesos cluster with Spark client mode

tag v0.2

  1. the same as v0.1
  2. calculate cpu_all usage

tag v0.3

  1. the same as v0.1
  2. realize calculating muli-machine cpu usage
  3. build spark driver image and executor image for Mesos Cluster

Usage

Prerequisite

First, you should set up zookeeper, cassandra and kafka broker. Then create kafka topic and cassandra keyspace

Second:

  1. download Apache Spark
  2. run linkerConnector
  3. git clone linkerProcessSample(https://github.com/adolphlwq/linkerProcessorSample.git)

submit code to Spark

Please refer Here for detail

Reference

TODOs

  • [X] collect info from kafka
  • [X] save processes info to cassandra
  • [X] save machine info to cassandra
  • [X] save kafka message to cassandra directly
  • [X] overall cpu usage from linkerConnector (via Kafka)
    • [X] calculate cpu usage and save to cassandra
    • [X] calculate cpu usage from multi machine
  • [X] Mesos agent usage from linkerConnector (via Kafka)
  • [X] build Mesos Spark executor docker image for testing code on Mesos cluster
  • [ ] research Spark streaming's "window" and improve code
Docker Pull Command
Owner
adolphlwq
Source Repository

Comments (0)