| Name | Kafka Consumer |
| Version | v1.0.0 |
| GitHub | weeve-modules/kafka-consumer |
| Authors | Jakub Grzelak |
Consumer to connect with Apache Kafka and input data from the cluster. The module uses utf-8 decoding for value deserializer.
The following module configurations can be provided in a data service designer section on weeve platform:
| Name | Environment Variables | Type | Description |
|---|---|---|---|
| Topic | TOPIC | string | Kafka topic to subscribe to. |
| Bootstrap Servers | BOOTSTRAP_SERVERS | string | List of comma (,) separated Kafka bootstrap servers that the consumer should contact to bootstrap initial cluster metadata. |
| Client ID | CLIENT_ID | string | A name for this client. This string is passed in each request to servers and can be used to identify specific server-side log entries that correspond to this client. |
| Group ID | GROUP_ID | string | The name of the consumer group to join for dynamic partition assignment, and to use for fetching and committing offsets. |
| Auto Offset Reset | AUTO_OFFSET_RESET | string | A policy for resetting offsets 'earliest' will move to the oldest available message, 'latest' will move to the most recent. |
| Enable Auto Commit | ENABLE_AUTO_COMMIT | boolean | If True, the consumer offset will be periodically committed in the background. |
Other features required for establishing the inter-container communication between modules in a data service are set by weeve agent.
| Environment Variables | type | Description |
|---|---|---|
| MODULE_NAME | string | Name of the module |
| MODULE_TYPE | string | Type of the module (INGRESS, PROCESS, EGRESS) |
| EGRESS_URLS | string | HTTP ReST endpoint for the next module |
requests
kafka-python
Input to this module is data stored on Kafka cluster.
Output to this module is data stored on Kafka cluster in JSON format.
Content type
Image
Digest
sha256:b318250ca…
Size
48.8 MB
Last updated
over 3 years ago
docker pull weevenetwork/kafka-consumer:v1.0.1