soolutions/s8-backupper-mongodb

•Updated 11 months ago

handy backup utility for mongodb in s3 buckets

Image

1.9K

soolutions/s8-backupper-mongodb repository overview

⁠MongoDB Backup System

A production-ready MongoDB backup solution with automated scheduling, S3 storage, and comprehensive restore capabilities.

⁠✨ Features

Automated Backups: Full and incremental backups with cron scheduling
S3 Integration: Secure cloud storage (no versioning required)
Docker Ready: Complete containerized solution
Health Monitoring: Built-in health checks and system status
Easy Restore: Simple restore procedures with verification
Production Ready: Comprehensive logging and error handling
Simplified Operations: All scripts work with standard S3 permissions

⁠🚀 Quick Start

⁠Prerequisites

Docker and Docker Compose
AWS S3 bucket with appropriate permissions
MongoDB instance (local or remote)

⁠Setup

Clone and configure:

git clone <repository-url>
cd cloud-databse-backupper
cp .env.example .env

Configure environment (edit .env):

# MongoDB
MONGODB_URL=mongodb://localhost:27017/your-database

# S3 Storage
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
S3_BUCKET=your-backup-bucket
S3_REGION=us-east-1

# Backup Settings
BACKUP_RETENTION_DAYS=30
FULL_BACKUP_SCHEDULE="0 2 * * 0"    # Weekly at 2 AM
INCREMENTAL_BACKUP_SCHEDULE="0 2 * * 1-6"  # Daily at 2 AM

Deploy:
```
docker-compose up -d
```

⁠Verify Installation

# Check system status
docker-compose exec mongodb-backup /scripts/system-status.sh

# Run health check
docker-compose exec mongodb-backup /scripts/health-check.sh

# Test MongoDB connection
docker-compose exec mongodb-backup /scripts/test-mongodb-connection.sh

⁠📋 Configuration

⁠Environment Variables

Variable	Description	Default
`MONGODB_URL`	MongoDB connection string	Required
`S3_BUCKET`	S3 bucket name	Required
`AWS_ACCESS_KEY_ID`	AWS access key	Required
`AWS_SECRET_ACCESS_KEY`	AWS secret key	Required
`S3_REGION`	AWS region	`us-east-1`
`BACKUP_RETENTION_DAYS`	Days to keep backups	`30`
`FULL_BACKUP_SCHEDULE`	Cron for full backups	`0 2 * * 0`
`INCREMENTAL_BACKUP_SCHEDULE`	Cron for incremental	`0 2 * * 1-6`

⁠MongoDB Connection Examples

# Local MongoDB
MONGODB_URL=mongodb://localhost:27017/myapp

# MongoDB with authentication
MONGODB_URL=mongodb://user:password@host:27017/myapp

# MongoDB Atlas
MONGODB_URL=mongodb+srv://user:[email protected]/myapp

# Replica Set
MONGODB_URL=mongodb://host1:27017,host2:27017/myapp?replicaSet=rs0

⁠🔧 Operations

⁠Monitoring

# System status
docker-compose exec mongodb-backup /scripts/system-status.sh

# Health check
docker-compose exec mongodb-backup /scripts/health-check.sh

# View logs
docker-compose logs mongodb-backup
docker-compose exec mongodb-backup tail -f /var/log/backup/backup.log

⁠Manual Backups

# Trigger full backup
docker-compose exec mongodb-backup /scripts/full-backup.sh

# Trigger incremental backup
docker-compose exec mongodb-backup /scripts/incremental-backup.sh

# List available backups
docker-compose exec mongodb-backup /scripts/list-backups.sh

⁠Restore Operations

# Quick restore (latest full backup)
docker-compose exec mongodb-backup /scripts/restore.sh full

# Restore latest incremental backup
docker-compose exec mongodb-backup /scripts/restore.sh incremental

# Restore specific version (when S3 versioning is enabled)
docker-compose exec mongodb-backup /scripts/restore.sh full abc123def456
docker-compose exec mongodb-backup /scripts/restore.sh incremental xyz789ghi012

# Dry run (test restore without making changes)
docker-compose exec mongodb-backup /scripts/restore.sh full --dry-run
docker-compose exec mongodb-backup /scripts/restore.sh full abc123def456 --dry-run

# Verify database after restore
docker-compose exec mongodb-backup /scripts/verify-restore.sh

📖 For detailed restore procedures, see RESTORE.md⁠

🚨 For emergency recovery, see DISASTER_RECOVERY_PLAN.md⁠

⁠🛠️ Troubleshooting

⁠Common Issues

Connection Failed:

# Test MongoDB connection
docker-compose exec mongodb-backup /scripts/test-mongodb-connection.sh

S3 Upload Failed:

# Check S3 permissions and list available backups
docker-compose exec mongodb-backup /scripts/list-backups.sh

Backup Failed:

# Check logs
docker-compose exec mongodb-backup tail -50 /var/log/backup/backup.log

Container Won't Start:

# Check container logs
docker-compose logs mongodb-backup

Getting Specific Version IDs (when S3 versioning is enabled):

# Get version ID for full backup
docker-compose exec mongodb-backup aws s3api head-object \
  --bucket s8-db-backups \
  --key montabook16-dev-mongodb-full.tar.gz \
  --query 'VersionId' --output text

# List all versions of a backup
docker-compose exec mongodb-backup aws s3api list-object-versions \
  --bucket s8-db-backups \
  --prefix montabook16-dev-mongodb-full.tar.gz

⁠Debug Mode

Enable detailed logging:

# Add to .env file
LOG_LEVEL=DEBUG

# Restart container
docker-compose restart mongodb-backup

⁠🔒 Security

⁠AWS IAM Permissions

Required S3 permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket",
        "arn:aws:s3:::your-bucket/*"
      ]
    }
  ]
}

⁠MongoDB Security

Use dedicated backup user with minimal permissions
Enable authentication and SSL/TLS
Restrict network access to backup container

⁠Container Security

Runs as non-root user
No unnecessary ports exposed
Secrets managed via environment variables

⁠📊 Monitoring Integration

⁠Prometheus Metrics

Metrics available at /metrics endpoint:

Backup success/failure rates
Backup duration and size
System resource usage

⁠Webhook Notifications

Configure webhooks for backup events:

WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK
WEBHOOK_EVENTS=backup_success,backup_failure,health_check_failure

⁠🔄 Maintenance

⁠Log Rotation

Logs are automatically rotated:

Daily rotation for backup logs
30-day retention by default
Compressed archives for space efficiency

⁠Backup Cleanup

Old backups are automatically cleaned based on BACKUP_RETENTION_DAYS:

# Manual cleanup
docker-compose exec mongodb-backup /scripts/cleanup-old-backups.sh

⁠Updates

# Update to latest version
docker-compose pull
docker-compose up -d

⁠🏗️ Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   MongoDB       │    │  Backup System  │    │   AWS S3        │
│   Database      │◄───┤  (Docker)       ├───►│   Storage       │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                              │
                              ▼
                       ┌─────────────────┐
                       │   Monitoring    │
                       │   & Alerts      │
                       └─────────────────┘

Components:

Backup Scripts: Automated backup and restore logic
Cron Scheduler: Manages backup timing
Health Monitor: Continuous system monitoring
S3 Integration: Secure cloud storage
Logging System: Comprehensive audit trail

⁠📞 Support

Documentation: RESTORE.md⁠ | DISASTER_RECOVERY_PLAN.md⁠
Issues: Create GitHub issue with logs and configuration
Emergency: Follow disaster recovery procedures

Production Ready ✅ | Docker Native 🐳 | S3 Integrated ☁️ | Monitoring Enabled 📊

Tag summary

Recent tags

Content type

Image

Digest

sha256:441dfadf0…

Size

201.5 MB

Last updated

11 months ago

docker pull soolutions/s8-backupper-mongodb