cpremus/documcp repository overview

⁠DocuMCP

A documentation server that exposes knowledge bases through the Model Context Protocol⁠ (MCP), enabling AI agents to search, read, and manage documentation.

DocuMCP gives AI agents structured access to your documentation via MCP tools and prompts. It handles document ingestion, full-text search, and OAuth 2.1 authorization. Written in Go and shipped as a distroless container (~45 MB).

⁠Features

• MCP Server -- 17 tools and 6 prompts via the official Go MCP SDK⁠, stateless Streamable HTTP. Search, read, create, update, replace, and delete documents. Federated search across documents, ZIM archives, and Git templates.
• OAuth 2.1 Authorization Server -- PKCE, device authorization (RFC 8628⁠), dynamic client registration (RFC 7591⁠), and RFC 9728⁠ Protected Resource Metadata for automatic discovery.
• Document Pipeline -- Upload PDF, DOCX, XLSX, HTML, EPUB, or Markdown. Text is extracted, indexed via PostgreSQL full-text search, and searchable within seconds.
• External Integrations -- Kiwix ZIM archives (federated article search) and Git template repositories.
• Background Jobs -- River⁠ Postgres-native job queue with 8 worker types, 3 queues (high, default, low), and 7 periodic schedules (enabled with SCHEDULER_ENABLED=true).
• Admin UI -- Vue 3 + TypeScript SPA for managing documents, users, OAuth clients, external services, and queue status.
• Observability -- OpenTelemetry tracing with automatic instrumentation for database queries (otelpgx), Redis commands (redisotel), and outbound HTTP (otelhttp). Prometheus metrics covering HTTP, database pool, Redis pool, search, queue, and OAuth token replay. Structured logging with slog (trace/span ID injection). Optional Sentry/GlitchTip error tracking. See docs/OBSERVABILITY.md⁠ for architecture and configuration.
• OIDC Authentication -- User login via any OpenID Connect provider.

⁠Quick Start

Docker Compose is the fastest way to run DocuMCP. The stack includes the application, PostgreSQL 18, Redis 8, and Traefik v3.7.

1. Create a .env file:

cat > .env <<EOF
# Database
DB_DATABASE=documcp
DB_USERNAME=documcp
DB_PASSWORD=$(openssl rand -hex 24)

# Redis
REDIS_PASSWORD=$(openssl rand -hex 24)

# Secrets
OAUTH_SESSION_SECRET=$(openssl rand -base64 32)
ENCRYPTION_KEY=$(openssl rand -hex 32)
INTERNAL_API_TOKEN=$(openssl rand -hex 32)
HKDF_SALT=$(openssl rand -base64 24)

# Public URL and TLS
APP_URL=https://documcp.example.com
TRAEFIK_HOST=documcp.example.com
[email protected]
EOF

1. Start the stack:

docker compose up -d

Migrations run automatically on first start. The compose file passes the whole .env to the app, so any variable from .env.example can be added there. It also turns on the job scheduler (SCHEDULER_ENABLED=true) and connects to the bundled Postgres without TLS (DB_SSLMODE=disable). Set DB_SSLMODE=require if you point DB_HOST at an external database.

1. The application is available at https://documcp.example.com via Traefik (ports 80/443), and directly on port 8080 of the host. To run without Traefik, remove the traefik service; to keep Traefik as the only entry point, remove the ports mapping from the app service.

See docs/OAUTH_CLIENT_GUIDE.md⁠ for connecting AI agents and CLI tools. MCP clients that don't yet implement RFC 8707 resource indicators (e.g. Open WebUI) can be supported via the opt-in OAUTH_ACCEPT_EMPTY_RESOURCE flag — see Accepting non-RFC-8707 clients⁠.

⁠Development

⁠Prerequisites

• Go 1.27.1 (the toolchain directive in go.mod)
• Node.js 24 (frontend)
• PostgreSQL (with pg_trgm and unaccent extensions)
• Redis 8+ (distributed rate limiting and SSE events)

⁠Build and Run

go build -o bin/documcp ./cmd/documcp    # Build binary

# Cobra subcommands:
go run ./cmd/documcp serve --with-worker # HTTP server + queue workers (dev default)
go run ./cmd/documcp serve               # HTTP server only (River insert-only)
go run ./cmd/documcp worker              # Queue workers only + health endpoint (:9090)
go run ./cmd/documcp migrate             # Run database migrations and exit
go run ./cmd/documcp rekey               # Re-encrypt secrets after rotating ENCRYPTION_KEY
go run ./cmd/documcp version             # Print version info
go run ./cmd/documcp health              # Check readiness (for Docker healthchecks)

⁠Test

go test ./...                        # Unit tests
go test -race ./...                  # With race detection
go test -cover ./...                 # With coverage
go test -tags integration ./...      # Unit + integration tests (needs Docker for testcontainers)

⁠Code Quality

gofmt -w .                           # Format
goimports -w .                       # Fix imports
golangci-lint run                    # Lint (v2.14.0)

⁠Frontend

cd frontend
npm ci
npm run build              # vue-tsc + Vite build -> web/frontend/dist/
npm run dev                # Dev server with HMR
npm run test               # Vitest
npm run test:coverage      # Vitest with coverage
npm run lint               # vue-tsc -b + ESLint
npm run format:check       # Prettier

⁠Architecture

cmd/documcp/             Entry point (serve, worker, migrate, rekey, version, health)
internal/
  app/                   App lifecycle (Foundation + ServerApp + WorkerApp)
  archive/               Shared zip/tar.gz builder with path-traversal guard
  auth/middleware/       Bearer-token, scope, and RFC 8707 audience middleware
  auth/oauth/            OAuth 2.1 server (PKCE, device flow, dynamic registration)
  auth/oidc/             OIDC client for user authentication
  auth/scope/            Scope registry and consent ceilings
  auth/session/          Redis-backed browser sessions
  client/kiwix/          ZIM archive reader (Kiwix)
  client/git/            Git template repository sync
  config/                Configuration loading (env, YAML)
  cron/                  Cron schedule definitions
  crypto/                AES-256-GCM encryption for secrets at rest
  database/              PostgreSQL connection and migrations (goose)
  dto/                   Data transfer objects
  extractor/             Text extraction (PDF, DOCX, XLSX, HTML, EPUB, Markdown)
  handler/api/           REST API handlers
  handler/mcp/           MCP tool and prompt handlers
  handler/oauth/         OAuth endpoint handlers
  model/                 Domain models
  observability/         Tracing, metrics, structured logging, Sentry
  queue/                 River job queue (workers, events, periodic jobs)
  repository/            Data access layer (pgx, handwritten SQL)
  search/                Full-text search (tsvector/tsquery + pg_trgm)
  security/              Path traversal and SSRF guards
  server/                HTTP server setup and routing (chi v5)
  service/               Business logic orchestration
  storage/               Blob storage abstraction (local filesystem + S3 via aws-sdk-go-v2)
  stringutil/            Shared string utilities
  testutil/              Test helpers and fixtures
frontend/                Vue 3 + TypeScript SPA source (admin panel)
web/frontend/            Embedded SPA (//go:embed dist/)
migrations/              SQL migration files (goose)
docs/contracts/          OpenAPI spec, MCP contract, OAuth flow reference

The application uses a single Cobra binary. serve and worker scale independently; migrate, rekey, version, and health are one-shot commands. A shared Foundation holds dependencies (database pool, Redis client, repositories, search). ServerApp handles HTTP; WorkerApp handles River queue processing. Redis provides distributed rate limiting across server instances and cross-instance SSE event delivery via Pub/Sub. Repositories use pgxpool.Pool directly, services accept repository interfaces, and handlers accept services. Background jobs run via River, a Postgres-native job queue.

⁠MCP Tools

Read tools need the mcp:read scope and write tools need mcp:write, on top of mcp:access for the endpoint itself. ZIM and Git template tools are always registered; they return empty results until a Kiwix server or Git template repository is configured in the admin panel.

⁠Configuration

DocuMCP is configured through environment variables. .env.example is the authoritative list; docs/CONFIGURATION.md⁠ groups every variable by concern (Application / Server / TLS / Database / Redis / OIDC / OAuth / Storage / Kiwix / Git Templates / Security / Queue Workers / Lifecycle / MCP Server / Observability / Scheduler) with defaults and descriptions.

The minimum set needed to start:

⁠Running multiple replicas

DocuMCP is designed to scale horizontally behind a load balancer. Two things have to be true:

1. Shared storage -- set STORAGE_DRIVER=s3 and point it at any S3-compatible service. The filesystem driver is node-local and will not work with more than one replica.
2. At least one worker replica -- uploaded documents are extracted by queue workers, and scheduled jobs (Kiwix and Git template sync, OAuth token and expired scope-grant cleanup, orphaned-file cleanup, soft-delete purge, external-service health checks, search-query retention) run via River's periodic-job enqueuer. River elects a single leader across the cluster via the river_leader table, and only the leader enqueues periodic jobs. If every replica runs in insert-only serve mode (without --with-worker), nothing is extracted and scheduled jobs never fire. Run at least one serve --with-worker or worker replica, and set SCHEDULER_ENABLED=true on it.

The MCP endpoint (/documcp) runs in Streamable HTTP stateless mode: every request creates a temporary session that closes after the response, so any replica can serve any request without sticky affinity. GET /documcp returns 405 Method Not Allowed -- we don't offer a standalone SSE stream because our tools are pure request/response (no sampling, elicitation, or server-initiated messages). The REST and admin surfaces are stateless against Postgres and also require no affinity.

Cross-replica cache invalidation is already handled: admin edits to Kiwix external services publish a message on a dedicated Redis pub/sub channel (documcp:control:cache.kiwix.invalidate) that all replicas subscribe to. Other caches are read-through against Postgres and don't need invalidation.

Per-replica health is reported at /health/ready, which checks Postgres and Redis and needs no token (the container healthcheck calls it). Worker-only replicas serve the same check on WORKER_HEALTH_PORT (default 9090).

⁠Documentation

⁠License

MIT⁠