linjifan/elasticrelay repository overview

⁠ElasticRelay - Multi-Source CDC Gateway to Elasticsearch

⁠Overview

ElasticRelay is a high-performance, real-time Change Data Capture (CDC) gateway that synchronizes data from multiple database sources (MySQL, PostgreSQL, MongoDB) to Elasticsearch. Built in Go for optimal performance and reliability, it offers zero-code configuration with built-in data governance and transformation capabilities.

⁠Key Features

• Real-time CDC: Transaction log-level change capture with precise checkpointing
• Multi-Database Support: MySQL (production-ready), PostgreSQL & MongoDB (upcoming)
• Zero-Code Configuration: JSON-based configuration with automatic format detection
• Multi-Table Dynamic Indexing: Automatic per-table Elasticsearch index creation
• Built-in Governance: Data transformation, anonymization, and type conversion
• Dead Letter Queue (DLQ): Comprehensive error handling with automatic retry
• Parallel Processing: Advanced chunking strategies for large table optimization
• High Reliability: Idempotent writes, fault tolerance, and automatic recovery

⁠Quick Start

⁠1. Run with Docker Compose

# Download example configuration
curl -O https://raw.githubusercontent.com/yogoosoft/elasticrelay/main/docker-compose.yml

# Start the stack
docker-compose up -d

⁠2. Run Standalone

# Create configuration file
cat > config.json << EOF
{
  "version": "3.0",
  "data_sources": [{
    "id": "mysql-main",
    "type": "mysql",
    "host": "mysql-host",
    "port": 3306,
    "user": "elastic_user",
    "password": "password",
    "database": "your_database",
    "server_id": 100,
    "table_filters": ["users", "orders", "products"]
  }],
  "sinks": [{
    "id": "es-main",
    "type": "elasticsearch",
    "addresses": ["http://elasticsearch:9200"],
    "options": {
      "index_prefix": "elasticrelay"
    }
  }],
  "global": {
    "log_level": "info",
    "grpc_port": 50051
  }
}
EOF

# Run ElasticRelay
docker run -d \
  --name elasticrelay \
  -p 50051:50051 \
  -v $(pwd)/config.json:/app/config/config.json \
  elasticrelay/elasticrelay:latest \
  --config /app/config/config.json

⁠Configuration

⁠Multi-Source Configuration (v3.0)

{
  "version": "3.0",
  "data_sources": [
    {
      "id": "mysql-main",
      "type": "mysql",
      "host": "localhost",
      "port": 3306,
      "user": "elastic_user",
      "password": "password",
      "database": "elasticrelay",
      "server_id": 100,
      "table_filters": ["users", "orders", "products"]
    }
  ],
  "sinks": [
    {
      "id": "es-main",
      "type": "elasticsearch",
      "addresses": ["http://localhost:9200"],
      "options": {
        "index_prefix": "elasticrelay"
      }
    }
  ],
  "global": {
    "log_level": "info",
    "grpc_port": 50051,
    "dlq_config": {
      "enabled": true,
      "storage_path": "dlq",
      "max_retries": 3,
      "retry_delay": "30s"
    }
  }
}

⁠Legacy Configuration Support

ElasticRelay automatically detects and migrates legacy configuration formats, ensuring backward compatibility.

⁠Environment Variables

⁠Volume Mounts

• /app/config: Configuration files
• /app/logs: Application logs
• /app/dlq: Dead Letter Queue storage

⁠Health Check

The container includes a built-in health check endpoint:

curl http://localhost:50051/health

⁠Supported Platforms

• linux/amd64
• linux/arm64

⁠Database Prerequisites

⁠MySQL

• Version: 5.7+ or 8.x
• Enable binary logging:

  # Add to my.cnf
  [mysqld]
  log-bin=mysql-bin
  server-id=1
  binlog-format=row
  

  Copy

⁠Elasticsearch

• Version: 7.x or 8.x
• Ensure sufficient heap size for bulk operations

⁠Example Use Cases

• E-commerce: Sync product catalogs, orders, and user data to Elasticsearch for search
• Analytics: Real-time data pipeline for business intelligence and reporting
• Microservices: Decouple database changes from search index updates
• Data Migration: Zero-downtime migration of data to Elasticsearch

⁠Links

• Source Code: GitHub Repository⁠
• Documentation: docs.elasticrelay.com⁠
• Issues: GitHub Issues⁠
• Discussions: GitHub Discussions⁠

⁠License

Apache License 2.0 - see LICENSE⁠ for details.

Current Status: Active development (MVP ~95% complete). Production-ready for MySQL to Elasticsearch synchronization.