A service for managing PostgreSQL logical replication between source and target databases.
1.5K
A Docker-based service for managing PostgreSQL logical replication between source and target databases.
A Docker-based service for managing PostgreSQL logical replication between source and target databases.
🐳 Docker Hub: https://hub.docker.com/r/darkmatter08/postgres-replica
For a complete working example, check out the db-sync-test repository:
This example repository demonstrates how to set up replication between multiple PostgreSQL databases with actual data and realistic scenarios.
📦 Example Repository: dark-matter08/db-sync-test - Complete working example with sample data and configurations
For logical replication to work, both source and target databases need proper configuration:
# Required PostgreSQL settings for the source database
wal_level = logical # Enable logical replication
max_replication_slots = 10 # Allow replication slots
max_wal_senders = 10 # Allow WAL senders
max_logical_replication_workers = 10 # Logical replication workers
max_worker_processes = 16 # Total worker processes
# Required PostgreSQL settings for target databases
wal_level = logical # Enable logical replication
max_replication_slots = 10 # Allow replication slots
max_logical_replication_workers = 10 # Logical replication workers
max_worker_processes = 16 # Total worker processes
The included docker-compose.yml automatically configures:
Source Database (source-db):
users and posts tablesTarget Databases (target-db-1, target-db-2):
Docker Compose Configuration:
Important: The Docker containers provide PostgreSQL instances with replication configuration, but you must create your own database schemas and initial data.
If using existing databases, ensure they meet these requirements:
# Add these settings to postgresql.conf
wal_level = logical
max_replication_slots = 10
max_wal_senders = 10
max_logical_replication_workers = 10
max_worker_processes = 16
-- On source database: verify tables exist
\dt
-- On target database: create matching schema
-- Use the provided scripts/setup-schema.sh or manually create tables
-- On source database
GRANT SELECT ON ALL TABLES IN SCHEMA public TO sourceuser;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO sourceuser;
-- On target database
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO targetuser1;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO targetuser1;
# Add to pg_hba.conf on source database
host replication sourceuser target-host/32 md5
# Add to pg_hba.conf on target database
host all targetuser1 source-host/32 md5
Note: The provided pg_hba_source.conf and pg_hba_target.conf files are configured for Docker networking and can be used as templates for your setup.
Since the Docker containers don't include init scripts, you'll need to set up your schemas:
# Connect to source database
docker exec -it postgres-source psql -U sourceuser -d sourcedb
# Create your tables, indexes, and data
CREATE TABLE users (id SERIAL PRIMARY KEY, name VARCHAR(100), email VARCHAR(100));
-- ... create your schema
# Connect to target databases and create matching schema
docker exec -it postgres-target-1 psql -U targetuser1 -d targetdb1
-- Create the same tables (without initial data)
# Apply your schema to source
docker exec -i postgres-source psql -U sourceuser -d sourcedb < your-schema.sql
# Apply same schema to targets (without data)
docker exec -i postgres-target-1 psql -U targetuser1 -d targetdb1 < your-schema-no-data.sql
docker exec -i postgres-target-2 psql -U targetuser2 -d targetdb2 < your-schema-no-data.sql
# Set environment variables for your databases
export SOURCE_HOST=localhost SOURCE_PORT=5432 SOURCE_USER=sourceuser SOURCE_PASS=sourcepass SOURCE_DB=sourcedb
export TARGET_HOST=localhost TARGET_PORT=5433 TARGET_USER=targetuser1 TARGET_PASS=targetpass1 TARGET_DB=targetdb1
# Copy schema from source to target
./scripts/setup-schema.sh copy-schema
To add or remove tables from replication:
Add tables to your target databases (if adding new tables):
-- Connect to each target database and create the new table structure
CREATE TABLE new_table (
id SERIAL PRIMARY KEY,
-- ... your columns
);
Update your replication-config.yml:
replication:
tables:
- "existing_table1"
- "existing_table2"
- "new_table" # Add this line
Restart the replication service:
docker-compose restart postgres-replica
The service will automatically:
Note: Always create table structures in target databases BEFORE adding them to the config.
The service reads its configuration from YAML files in the following order of preference:
CONFIG_FILE environment variable path/config/replication-config.yml (Docker volume mount)./config/replication-config.yml (local config directory)./replication-config.yml (current directory)../replication-config.yml (parent directory)REPLICATION_CONFIG environment variable (fallback)The repository includes a complete example setup:
replication-config.yml - Example configurationdocker-compose.yml - Complete multi-database setuppg_hba_source.conf - Source database authentication rulespg_hba_target.conf - Target database authentication rulesreplication:
publication_name: "example_publication"
# Source database (publisher)
source:
host: "source-db"
port: 5432
user: "sourceuser"
password: "sourcepass"
database: "sourcedb"
# Target databases (subscribers)
targets:
- name: "primary_replica"
subscription_name: "primary_subscription"
host: "target-db-1"
port: 5432
user: "targetuser1"
password: "targetpass1"
database: "targetdb1"
settings:
enable_initial_sync: true
disable_triggers_during_sync: true
# Tables to replicate
tables:
- "users"
- "posts"
# Global settings
settings:
max_wait_attempts: 30
wait_interval_seconds: 5
enable_initial_sync: true
disable_triggers_during_sync: true
# Pull the latest image
docker pull darkmatter08/postgres-replica:latest
# Run with your config file
docker run --rm \
--name postgres-replica \
-v "/path/to/your/replication-config.yml:/config/replication-config.yml:ro" \
-p 3001:3000 \
darkmatter08/postgres-replica:latest
Use the included docker-compose.yml for a complete setup with source and target databases:
# Clone the repository
git clone <your-repo-url>
cd postgres-replica
# Start all services
docker-compose up -d
# Check logs
docker-compose logs -f postgres-replica
# Check health
curl http://localhost:3001/health
This will start:
Add to your existing docker-compose.yml:
services:
postgres-replica:
image: darkmatter08/postgres-replica:latest
volumes:
- ./replication-config.yml:/config/replication-config.yml:ro
ports:
- "3001:3000" # Health check endpoint
depends_on:
- your-source-db
- your-target-db
restart: unless-stopped
# Clone and build locally
git clone <your-repo-url>
cd postgres-replica
# Build the image
docker build -t postgres-replica .
# Run with your config
docker run --rm \
--name postgres-replica \
-v "./replication-config.yml:/config/replication-config.yml:ro" \
-p 3001:3000 \
postgres-replica
# Clone the repository
git clone <your-repo-url>
cd postgres-replica
# Start all services (source DB + target DBs + replication service)
docker-compose up -d
# Check the replication status
curl http://localhost:3001/health
# View logs
docker-compose logs -f postgres-replica
What this sets up:
http://localhost:3001/healthNote: You need to create your own database schemas. The containers only provide the PostgreSQL instances with replication configuration.
# Pull the image
docker pull darkmatter08/postgres-replica:latest
# Create your replication-config.yml (see sample below)
# Then run:
docker run --rm \
-v "./replication-config.yml:/config/replication-config.yml:ro" \
-p 3001:3000 \
darkmatter08/postgres-replica:latest
Once running, monitor the service:
# Check overall health
curl http://localhost:3001/health
# Pretty print the JSON response
curl -s http://localhost:3001/health | python -m json.tool
# Install dependencies
npm install
# Build the project
npm run build
# Run in development mode
npm run dev
REPLICATION_CONFIG: The replication configuration (required)The service intelligently manages table changes:
The service includes comprehensive error handling:
Errors are logged with detailed information for troubleshooting.
❌ Missing required tables in target database: users, posts
Solution: Target databases must have the same schema as source
# Option 1: Use the provided schema setup script
./scripts/setup-schema.sh copy-schema
# Option 2: Manually copy schema
pg_dump -h source-host -U source-user --schema-only source-db | \
psql -h target-host -U target-user target-db
Check these settings in postgresql.conf:
wal_level = logicalmax_replication_slots >= 1max_wal_senders >= 1 (source only)max_logical_replication_workers >= 1 (target only)FATAL: no pg_hba.conf entry for replication connection
Solution: Add replication entry to pg_hba.conf on source database:
# Allow replication connections
host replication sourceuser target-host/32 md5
-- Check publication exists
SELECT * FROM pg_publication WHERE pubname = 'your_publication';
-- Check published tables
SELECT * FROM pg_publication_tables WHERE pubname = 'your_publication';
-- Check subscription status
SELECT * FROM pg_subscription WHERE subname = 'your_subscription';
-- Check replication worker status
SELECT * FROM pg_stat_subscription WHERE subname = 'your_subscription';
The service provides detailed logging:
# View replication service logs
docker-compose logs -f postgres-replica
# View database logs
docker-compose logs source-db
docker-compose logs target-db-1
ISC
Content type
Image
Digest
sha256:fd0270303…
Size
55.9 MB
Last updated
7 months ago
docker pull darkmatter08/postgres-replica