Public Repository

Last pushed: 3 months ago
Short Description
Kafka Consumer Lag Checking Application for monitoring
Full Description

Remora

A simple API application, Built using akka and scala, in front of kafka to give offset information to act as a Kafka lag checker

Background

Due to using akka streams we could not get the current offset from the api. Instead we used linkedin burrow
but contains some performance issues

All we did was wrap and hack the kafka consumer group command in akka http.

Prerequisites

  • Kafka 0.10.2.1 (for 0.10.0.1 please see the 0.1.0 release as the api has changed since)
  • Store offsets in kafka

Server Arguments

  • SERVER_PORT - default 9000
  • KAFKA_ENDPOINT - default localhost:9092

Show Active Consumers

http://localhost:9000/consumers

which gives

["consumer-1", "consumer-2", "consumer-3"]

Show specific consumer info

curl http://localhost:9000/consumers/<ConsumerGroupId>

State can be "Empty", "Dead", "Stable", "PreparingRebalance", "AwaitingSync"

{  
   "state":"Empty",
   "partition_assignment":[  
      {  
         "group":"console-consumer-20891",
         "coordinator":{  
            "id":0,
            "id_string":"0",
            "host":"foo.company.com",
            "port":9092
         },
         "topic":"products-in",
         "partition":1,
         "offset":3,
         "lag":0,
         "consumer_id":"-",
         "host":"-",
         "client_id":"-",
         "log_end_offset":3
      },
      {  
         "group":"console-consumer-20891",
         "coordinator":{  
            "id":0,
            "id_string":"0",
            "host":"foo.company.com",
            "port":9092
         },
         "topic":"products-in",
         "partition":0,
         "offset":3,
         "lag":0,
         "consumer_id":"consumer-1-7baba9b9-0ec3-4241-9433-f36255dd4708",
         "host":"/xx.xxx.xxx.xxx",
         "client_id":"consumer-1",
         "log_end_offset":3
      }
   ]
}

Health

curl http://localhost:9000/health returns OK

Metrics

curl http://localhost:9000/metrics

Contributing

We are happy to accept contributions. First, take a look at our contributing guidelines.

You can see our current status in this task board.

TODO

Please check the Issues Page
for contribution ideas.

Contact

Feel free to contact one of the maintainers.

License

MIT

Docker Pull Command
Owner
zalandoremora

Comments (0)