Sign inSign up

ifloor/kafka-real-time-data-scrapper

By ifloor

•Updated almost 4 years ago

Tool to export some metrics from a running kafka broker to elasticsearch

Image
0

1.0K

ifloor/kafka-real-time-data-scrapper repository overview

Application to scrap a running Kafka broker and export some metrics of it to an elasticsearch database

Current features:

  1. Lag of consumer groups (groups can be tweaked by exclusion / inclusion rules)
  2. Calculate time between a request message, and it's result (topic patterns for 'request' and 'result' can be tweaked)

⁠Environment variables to run:

⁠Required variables
Var nameFeature relatedDescriptionDefault valueAvailable options
KAFKA_BOOTSTRAP_SERVERS1, 2Kafka broker(s) to connect[IP:PORT,...]
ELASTIC_SERVER1, 2The elastic server in which the data will be writtenhttp(s) url
ZOOKEEPER_SERVERS1The zookeeper of the kafka broker aboveIP:PORT
KAFKA_SCHEMA_REGISTRY2Schema registry to deserialize messageshttp(s) url
CALCULATION_REQUEST_TOPIC_SUFFIX2Suffix that requests topics have at the endAny string
CALCULATION_RESULT_TOPIC_SUFFIX2Suffix that results topics have at the endAny string
CALCULATION_LINK_ID_PATH2Path to the value that is the ID that links the request with the result. Per example: request.idAny string
⁠Optional variables
Var nameFeature relatedDescriptionDefault valueAvailable options
ENABLE_SCRAPING1Enable lag and productions scrapingfalsetrue, false
ENABLE_TIME_CALCULATIONS2Enable calculate of time between requests and resultsfalsetrue, false
LOG_LEVEL1, 2Kafka broker(s) to connectINFOTRACE, DEBUG, INFO, WARN, ERROR
EXPORT_ELASTIC_INDEX1, 2Index to write the scrapped datakafka-real-time-data-scrapperAny string
ELASTIC_SEC_USER1, 2User to authenticate on elasticAny string
ELASTIC_SEC_PASS1, 2Password to authenticate on elasticAny string
ELASTIC_SSL_CA_PATH1, 2Custom CA path to connect to elastic
REGEX_EXCLUDED_CONSUMER_GROUPS1Consumer groups which lag won't be scrapped
REGEX_FORCE_INCLUDE_CONSUMER_GROUPS1,Consumers that can be excluded in the above param, can be forced to be include using this var
TIMEOUT_ON_QUERIES1Timeout when querying data on zookeeper15000Any integer milliseconds
INTERVAL_SECONDS_BETWEEN_SCRAPS1The interval between scraping data60Any integer seconds
CALCULATIONS_LISTENER_GROUP_ID2The group id that will be used to listen to messageskafka-real-time-data-scrapperAny string
CALCULATIONS_SECONDS_LIMIT_WAIT_RESULT2The maximum amount of time in seconds the calculation will wait for a result. After that, it will be cleaned600Any integer seconds

Tag summary

Content type

Image

Digest

sha256:e94e1938a…

Size

565.1 MB

Last updated

almost 4 years ago

docker pull ifloor/kafka-real-time-data-scrapper:1.3.0