trusmith/dropbox-consumer repository overview

⁠Dropbox Consumer

File monitoring service that copies new files from a source directory to a destination. Built for document management systems like Paperless-NGX.

⁠Why I Built This

I created this out of a need to use paperless-ngx⁠ without having it devour my documentation. I needed a way to continue using my existing organization structure while also using paperless. It does create two sources for the same information, yes, but that's what I needed. And since there was nothing else like it out there, I chose to build this small, efficient, and simple tool.

⁠Features

Real-time monitoring, duplicate detection, file stability checks, persistent state, file filtering.

⁠Quick Start

Complete docker-compose.yml example:

version: "3.8"
services:
  dropbox_consumer:
    image: trusmith/dropbox-consumer:latest
    container_name: dropbox_consumer
    user: "${PUID:-1000}:${PGID:-1000}"
    volumes:
      - /your/source/path:/source:ro
      - /your/destination/path:/consume:rw
      - ./state:/app/state:rw
    environment:
      - SOURCE=/source
      - DEST=/consume
      - RECURSIVE=true
      - PRESERVE_DIRS=false
      - COPY_EMPTY_DIRS=false
      - STATE_DIR=/app/state
      - STATE_CLEANUP_DAYS=30
      - STATE_BACKUP_COUNT=3
      - LOG_LEVEL=INFO
      - DEBOUNCE_SECONDS=1.0
      - STABILITY_INTERVAL=0.5
      - STABILITY_STABLE_ROUNDS=2
      - COPY_TIMEOUT=60
      - MAX_WORKERS=4
      - FILE_INCLUDE_PATTERNS=
      - FILE_EXCLUDE_PATTERNS=
      - DRY_RUN=false
      - DELETE_SOURCE=false
      - MAX_FILE_SIZE_MB=0
      - COMPRESS_FILES=false
      - WEBHOOK_URL=
      - RETRY_ATTEMPTS=3
      - RETRY_DELAY=2.0
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
    restart: unless-stopped

Start the service:

docker-compose up -d

Monitor activity:

docker-compose logs -f dropbox_consumer

⁠Configuration

Environment variables control behavior. Set in docker-compose.yml or via command line.

⁠Core Settings

⁠Performance Tuning

⁠File Filtering

When include patterns are specified, only files matching at least one pattern are processed. Exclude patterns are applied first and take precedence.

Examples:

# Only copy PDF and image files
environment:
  - FILE_INCLUDE_PATTERNS=*.pdf,*.jpg,*.png,*.jpeg

# Exclude temporary and system files
environment:
  - FILE_EXCLUDE_PATTERNS=*.tmp,*.swp,.DS_Store,Thumbs.db

# Combine both
environment:
  - FILE_INCLUDE_PATTERNS=*.pdf,*.docx
  - FILE_EXCLUDE_PATTERNS=*draft*,*temp*

⁠Advanced Options

⁠How It Works

On startup, the service snapshots all existing files in the source directory and marks them as processed. Only new files added after startup are copied.

File stability checks prevent copying incomplete files. The service monitors file size and waits for it to remain unchanged for the configured stability period.

Duplicate detection uses SHA-256 hashes. Files with identical content are skipped regardless of filename.

State is persisted to disk and survives container restarts. Files processed before a restart will not be reprocessed.

⁠Directory Structure Preservation

When PRESERVE_DIRS=true, the service maintains the relative path structure:

source/2024/invoices/document.pdf → destination/2024/invoices/document.pdf

When disabled, all files are copied to the destination root:

source/2024/invoices/document.pdf → destination/document.pdf

⁠Permissions

The service runs as the user specified by PUID and PGID. Set these to match your destination directory ownership:

id $(whoami)

Update docker-compose.yml:

user: "${PUID:-1000}:${PGID:-1000}"

The state directory must be writable by this user.

⁠Logging

Logs are written to stdout and managed by Docker. Configure rotation in docker-compose.yml:

logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"

⁠Development

Run locally without Docker:

python3 -m pip install -r requirements.txt
export SOURCE=/path/to/source
export DEST=/path/to/destination
python3 main.py

⁠Troubleshooting

⁠Files Not Being Copied

Check logs for errors:

docker-compose logs -f dropbox_consumer

Verify permissions:

docker-compose exec dropbox_consumer ls -la /source
docker-compose exec dropbox_consumer ls -la /consume

⁠Permission Errors

Ensure PUID and PGID match the destination directory owner:

ls -la /your/destination/path
id $(whoami)

Set correct values in docker-compose.yml.

⁠State Persistence Issues

Verify state directory is writable:

ls -la ./state/
docker-compose exec dropbox_consumer ls -la /app/state

Fix permissions if needed:

sudo chown -R $(id -u):$(id -g) ./state/

⁠High CPU Usage

Reduce worker threads or increase debounce time:

environment:
  - MAX_WORKERS=1
  - DEBOUNCE_SECONDS=5.0

⁠Docker Hub

Image: trusmith/dropbox-consumer:latest

Pull the image:

docker pull trusmith/dropbox-consumer:latest

Available tags:

• latest - Most recent stable release
• v3.0 - Version 3.0 release
• main - Latest development build

⁠License

Licensed under the MIT License.

See the LICENSE⁠ file for details.

⁠Support

If this project helped you, please consider supporting my work: