ssingh3339/es-bulk-proxy repository overview

⁠es-bulk-proxy

A lightweight, transparent Elasticsearch proxy that aggregates small _bulk requests into larger batches to improve indexing performance and reduce disk I/O.

⁠🚀 Overview

es-bulk-proxy sits between your data source (e.g., Zenarmor, log shippers) and Elasticsearch. It buffers incoming _bulk requests in memory and periodically flushes them as larger batches, significantly reducing write amplification and improving overall cluster efficiency.

At the same time, it behaves as a fully compatible Elasticsearch proxy, forwarding all non-bulk requests (search, mappings, health checks, etc.) directly without modification.

⁠✨ Key Features

• 🔄 Smart _bulk request aggregation
• ⚡ Reduces Elasticsearch disk IOPS and segment merges
• 🧠 In-memory buffering (no disk usage)
• 🔍 Transparent proxy for all Elasticsearch APIs
• ⚙️ Configurable batching (time + size based)
• 🪶 Lightweight and fast (minimal resource usage)

⁠🧱 Architecture

Client (Zenarmor, etc)
        ↓
   es-bulk-proxy
        ↓
   Elasticsearch

⁠🎯 Use Cases

• High-frequency small bulk ingestion
• Network/security analytics pipelines (e.g., Zenarmor)
• Log ingestion systems
• Reducing Elasticsearch write overhead

⁠⚙️ Configuration

Configured via environment variables:

• ES_URL – Elasticsearch endpoint (required)
• FLUSH_INTERVAL – Batch flush interval (default: 3s)
• MAX_BATCH_SIZE – Max batch size (default: 5MB)
• MAX_BUFFER_SIZE – Max buffer size (default: 50MB)

⁠🐳 Quick Start

docker run -d \
  -p 8080:8080 \
  -e ES_URL=http://localhost:9200 \
  ssingh3339/es-bulk-proxy

⁠📌 Notes

• Designed for real-time ingestion pipelines
• Works best when paired with optimized Elasticsearch settings
• No persistent queue — data in buffer may be lost on crash

Built for efficient and scalable Elasticsearch ingestion pipelines.