optnc/producer-file
Utilitaire d'Intégration d'un fichier plat dans Kafka (similaire Kafkacat) avec conversion CSV/JSON
54
Utilitaire permettant d'intégrer un fichier dans un topic Kafka (à la manière de kafkacat
) avec les fonctionnalités suivantes :
Prérequis :
NOTES:
Basée sur le quickstart Redpanda
cat <<EOF | docker compose -f - -p demo up -d
---
version: "3.7"
name: demo
networks:
redpanda_network:
driver: bridge
volumes:
redpanda-0: null
services:
redpanda-0:
command:
- redpanda
- start
- --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
# Address the broker advertises to clients that connect to the Kafka API.
# Use the internal addresses to connect to the Redpanda brokers'
# from inside the same Docker network.
# Use the external addresses to connect to the Redpanda brokers'
# from outside the Docker network.
- --advertise-kafka-addr internal://redpanda-0:9092,external://localhost:19092
- --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082
# Address the broker advertises to clients that connect to the HTTP Proxy.
- --advertise-pandaproxy-addr internal://redpanda-0:8082,external://localhost:18082
- --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081
# Redpanda brokers use the RPC API to communicate with eachother internally.
- --rpc-addr redpanda-0:33145
- --advertise-rpc-addr redpanda-0:33145
# Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system.
- --smp 1
# The amount of memory to make available to Redpanda.
- --memory 1G
# Mode dev-container uses well-known configuration properties for development in containers.
- --mode dev-container
# enable logs for debugging.
- --default-log-level=debug
image: docker.redpanda.com/redpandadata/redpanda:v23.1.4
container_name: redpanda-0
volumes:
- redpanda-0:/var/lib/redpanda/data
networks:
- redpanda_network
ports:
- 18081:18081
- 18082:18082
- 19092:19092
- 19644:9644
console:
container_name: redpanda-console
image: docker.redpanda.com/vectorized/console:v2.2.3
networks:
- redpanda_network
entrypoint: /bin/sh
command: -c 'echo "\$\$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console'
environment:
CONFIG_FILEPATH: /tmp/config.yml
CONSOLE_CONFIG_FILE: |
kafka:
brokers: ["redpanda-0:9092"]
schemaRegistry:
enabled: true
urls: ["http://redpanda-0:8081"]
redpanda:
adminApi:
enabled: true
urls: ["http://redpanda-0:9644"]
ports:
- 8080:8080
depends_on:
- redpanda-0
EOF
echo "ID,VALEUR,DATE\n1,Un,05/04/2023 15:17:22\n2,Deux,05/04/2023 15:18:33" > /tmp/demo.csv
producer-file
en JSON 🚀docker run --rm --network demo_redpanda_network -v /tmp/demo.csv:/tmp/demo.csv optnc/producer-file -v -b redpanda-0:9092 -t demo -J ',' -K '{{ID}}' /tmp/demo.csv
👉🏼 pour plus d'infos
docker run --rm optnc/producer-file -h
kcat -b localhost:19092 -t demo -C -K ";"
ou bien sur l'interface http://localhost:8080/topics/demo
ℹ️ À noter que les dates ont été converties en ISO
rm /tmp/demo.csv
docker compose -p demo down
docker volume rm demo_redpanda-0
docker pull optnc/producer-file