jerrinot/pg_gregor repository overview

⁠pg_gregor - Kafka Protocol for PostgreSQL

⚠️ EARLY DEVELOPMENT - EXPERIMENTAL

A PostgreSQL extension that speaks Kafka protocol. Send messages with any Kafka client, store them in PostgreSQL tables with full SQL access.

⁠Why This Exists

Ever wanted to:

• Use Kafka clients but store data in PostgreSQL for complex queries?
• Get Kafka's simple producer API with PostgreSQL's ACID guarantees?
• Bridge microservices using Kafka protocol while keeping data in your existing PostgreSQL infrastructure?

pg_gregor lets you use standard Kafka clients (kafka-python, Java, etc.) while storing everything in PostgreSQL tables that you can query with SQL.

⁠Quick Try (One-Liner)

docker run -p 5432:5432 -p 9092:9092 jerrinot/pg_gregor:latest

This starts:

• PostgreSQL on port 5432 (user: postgres, password: postgres, db: pg_gregor)
• Kafka protocol server on port 9092

⁠Test It Out

Once the container is running, test with Python:

# pip install kafka-python psycopg2-binary
from kafka import KafkaProducer
import psycopg2

# Send message via Kafka protocol
producer = KafkaProducer(bootstrap_servers=['localhost:9092'])
producer.send('test-topic', b'Hello from Kafka client!')
producer.close()

# Query it with SQL
conn = psycopg2.connect("host=localhost port=5432 user=postgres password=postgres dbname=pg_gregor")
cur = conn.cursor()
cur.execute("SELECT * FROM kafka_messages;")
print(cur.fetchall())  # Your Kafka message is here!

⁠Current Status

What Works:

• ✅ Kafka producer protocol (send messages)
• ✅ Standard Kafka clients (kafka-python, Java, etc.)
• ✅ Message storage in PostgreSQL tables
• ✅ Topic auto-creation
• ✅ Docker deployment

What's Missing (coming soon):

• ❌ Kafka consumers (can only send, not receive via Kafka protocol)
• ❌ Consumer groups
• ❌ Production hardening
• ❌ Performance optimization

This is experimental software! Use for prototyping and testing only.

⁠Use Cases

• Data ingestion: Use Kafka clients for simple message sending, PostgreSQL for complex analytics
• Microservice integration: Services speak Kafka, data lives in PostgreSQL for reporting
• Migration path: Start with Kafka interface, gradually move to pure PostgreSQL
• Development/testing: Mock Kafka infrastructure with persistent PostgreSQL storage

⁠Project Goals

1. Protocol compatibility: Work with existing Kafka clients without changes
2. PostgreSQL integration: Full ACID transactions, SQL queries, PostgreSQL ecosystem
3. Simple deployment: Single container, no separate Kafka infrastructure
4. Development bridge: Easy transition between Kafka and PostgreSQL patterns

⁠Architecture

Kafka Client --[Kafka Protocol]--> pg_gregor --[SQL]--> PostgreSQL
      ↓                               ↓                    ↓
  Standard API                 Protocol Bridge         ACID Storage

⁠Database Schema

Messages are stored in simple tables:

-- Your messages end up here
SELECT topic, message_key, message_value, created_at 
FROM kafka_messages 
WHERE topic = 'your-topic';

-- Topic metadata
SELECT * FROM kafka_topics;

⁠Development Status

This project is in very early development. Expect:

• API changes
• Limited Kafka feature support
• Rough edges and bugs
• Breaking changes between versions

Contributions welcome! Check the GitHub repository⁠ for source code and development setup.

⁠Support

• 🐛 Issues: Report bugs on GitHub
• 💡 Ideas: Feature requests welcome
• 🤝 Contributing: PRs appreciated
• ❓ Questions: GitHub discussions

Remember: This is experimental software. Not for production use yet!