aeternalabshq/pullmd repository overview

⁠PullMD

Anything in, clean Markdown out. Self-hosted conversion of web pages, documents, images, audio and YouTube videos into token-lean Markdown, for you and for your AI agents.

Quick start⁠ · Use with AI agents⁠ · API⁠ · Configuration⁠ · Changelog⁠

⁠Why PullMD

Agents are bad at reading the web. Raw HTML burns context on navigation, cookie banners and scripts, and built-in fetch tools often stumble on JavaScript-heavy pages, PDFs and comment threads. PullMD sits in between: one URL or file in, clean Markdown out, with source and metadata in YAML frontmatter instead of the body.

curl "https://pullmd.example.com/api?url=https://example.com/article"

What that saves, measured on three public pages (tokens counted with the cl100k_base tokenizer, 2026-10-16):

The curl column is what a naive fetch tool hands an agent. The last column is query-scoped extraction⁠: only the sections relevant to a question.

⁠Highlights

Reads almost anything

• Web pages through a fallback chain: Cloudflare's native Markdown, then Readability + Trafilatura (best result wins), then headless Chromium for JavaScript-heavy sites
• Reddit and Hacker News threads with their comment trees (depth and count configurable)
• Documents: PDF, Word, PowerPoint, Excel, EPUB, by URL or upload, plus an optional OCR tier for table-grade PDFs
• YouTube title, description and transcript with clickable timecodes, no API key needed
• Images and audio via any OpenAI-compatible or local model (captions, transcripts)

Built for agents

• MCP server for Claude Code and other MCP clients, with OAuth 2.1 for claude.ai and Claude Desktop
• Query-scoped extraction: ?query= returns only the sections relevant to a question, typically 70-95% fewer tokens on long pages
• Lean output: clean body, metadata in frontmatter, unpadded tables, configurable field allowlist
• Live share links: /s/:id re-fetches when the cached copy is older than an hour and serves the last good snapshot if the source dies

Comfortable to self-host

• One docker compose up, zero config, multi-arch images (amd64/arm64) on Docker Hub and GHCR
• Every sidecar is optional, and PullMD degrades gracefully without it
• Auth modes from open homelab to multi-user, SSRF protection, rate limits, a health endpoint for Uptime Kuma
• Per-site extraction recipes as declarative JSON

Four ways in: PWA · REST API · MCP server · Claude Code skill

⁠Quick start

mkdir pullmd && cd pullmd
curl -O https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/docker-compose.yml
docker compose up -d
# → http://localhost:3000

No .env needed: every variable has a sensible default. Add a .env next to the compose file to override anything (see Configuration⁠). Then connect your agent: see AI-agent integration⁠.

⁠docker-compose.yml (zero-config, abridged)

services:
  pullmd:
    image: aeternalabshq/pullmd:latest
    container_name: pullmd
    restart: unless-stopped
    ports:
      - "${PORT:-3000}:3000"
    environment:
      - PUBLIC_URL=${PUBLIC_URL:-http://localhost:${PORT:-3000}}
      - TRAFILATURA_URL=http://trafilatura:8001/extract
      - PLAYWRIGHT_URL=http://playwright:8002/render
      - MARKITDOWN_URL=http://markitdown:8003/convert
      - CACHE_DB=/data/cache.db
    volumes:
      - ./data:/data
    networks:
      - pullmd-internal
    depends_on:
      - trafilatura
      - playwright
      - markitdown

  trafilatura:
    image: aeternalabshq/pullmd-trafilatura:latest
    container_name: pullmd-trafilatura
    restart: unless-stopped
    networks:
      - pullmd-internal

  playwright:
    image: aeternalabshq/pullmd-playwright:latest
    container_name: pullmd-playwright
    restart: unless-stopped
    networks:
      - pullmd-internal

  markitdown:
    image: aeternalabshq/pullmd-markitdown:latest
    container_name: pullmd-markitdown
    restart: unless-stopped
    mem_limit: ${MARKITDOWN_MEM_LIMIT:-1g}
    networks:
      - pullmd-internal

networks:
  pullmd-internal:
    driver: bridge

⁠Behind Traefik

For deployments behind Traefik with TLS, use docker-compose.traefik.yml instead. Same images, but with Traefik labels and the proxy external network. Set HOST_DOMAIN in .env:

curl -O https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/docker-compose.traefik.yml
echo "HOST_DOMAIN=pullmd.example.com" > .env
docker compose -f docker-compose.traefik.yml up -d

This file sets PULLMD_TRUST_PROXY=1: Traefik is the only peer (no port is published), so PullMD takes the client address and the https scheme from Traefik's forwarding headers.

⁠Behind another reverse proxy

PullMD ignores X-Forwarded-For, X-Forwarded-Proto and X-Forwarded-Host unless PULLMD_TRUST_PROXY says the connecting peer is a proxy. Behind nginx, Caddy, a tunnel or a load balancer, set:

PULLMD_TRUST_PROXY=1                          # one proxy hop, port not reachable directly
# or: PULLMD_TRUST_PROXY=172.18.0.0/16       # the proxy's address/network
PUBLIC_URL=https://pullmd.example.com

A hop count trusts whatever connects, so use it only when the container port is not published to anyone but the proxy; otherwise list the proxy's address. Without the setting, all clients share one rate-limit bucket (login, signup, share-link misses, OAuth, media budget) and PullMD logs a one-time warning. PUBLIC_URL keeps the session cookie Secure and is accepted as this site's host by the login form check. Do not send Referrer-Policy: no-referrer for PullMD's pages: browsers then post forms with Origin: null, which is refused. See MIGRATION.md⁠.

⁠Local development (no Docker)

git clone https://github.com/AeternaLabsHQ/pullmd.git
cd pullmd
npm install
npm start             # http://localhost:3000
npm test              # node --test

⁠Configuration

All variables go in .env (copy from .env.example):

PUBLIC_URL matters for self-hosting: the help page and downloadable skill embed it as the canonical endpoint. Set it correctly and your users get a copy-paste setup that points at your instance.

PullMD rotates its outbound User-Agent for the web fetch path from a pool of current desktop browsers, refreshed every 48 hours from a live feed of real-world UAs⁠ maintained by @WinFuture23⁠. A built-in seed pool ensures rotation works even when the feed is unreachable. Set PULLMD_USER_AGENT to pin a single UA, or PULLMD_UA_FEED_URL to point at your own feed. The Reddit path keeps its dedicated REDDIT_USER_AGENT because Reddit's API expects a stable, identifying UA.

DISABLE_PUBLIC_HISTORY=true is the privacy switch for shared instances (multi-tenant VPS, office deployments). Conversions still get cached and assigned share IDs; users just can't see what other users have fetched (the MCP list_recent tool follows the same rule, and /api/stats leaves out its per-domain lists). Anyone with a known /s/:id link still gets their markdown back. Share ids are 128 random bits, so a link cannot be guessed, and /s/:id throttles unknown-id lookups to 120 per minute and IP (valid links are never throttled). With sign-in enabled (PULLMD_AUTH_MODE=single-admin or multi-user) history is scoped per user anyway; the switch matters for instances without sign-in.

⁠Authentication (v2.0+)

PullMD ships with three auth modes. Pick one with PULLMD_AUTH_MODE:

In single-admin and multi-user modes, PULLMD_ADMIN_EMAIL + PULLMD_ADMIN_PASSWORD bootstrap the first admin user on first startup. After that, changing these env vars does not change the password — use the admin CLI:

docker compose exec pullmd node scripts/admin.js reset-password [email protected]

Create an account without opening self-registration (useful when PULLMD_ALLOW_SIGNUP is off):

docker compose exec pullmd node scripts/admin.js create-user [email protected]

Both commands read the password from stdin, so they need it attached: docker compose exec, docker exec -it, or a pipe (echo "…" | docker exec -i <container> node scripts/admin.js …). A plain docker exec without -i aborts with an error and exit code 2 instead of doing nothing.

⁠Auth boundary

/api/stats returns its per-domain lists (lowQualityDomains, fallbackByDomain) only to the admin, or to everyone when auth is off and public history is on; the aggregate counts are public. Signed-in callers of /api/history and MCP list_recent get their own conversions.

Cache deletes are scoped to the caller. An admin (and every caller in disabled mode) removes the shared, URL-deduped cache row, which affects every user's history. A regular user only unlinks the entry from their own history - the shared row and its /s/:id share link keep working. The response says which happened via "scope": "user" | "global".

⁠Authentication paths

1. Session cookies — POST /login sets pullmd_session (HttpOnly, SameSite=Lax, 90-day TTL with sliding expiry). The PWA uses this automatically. The cookie is Secure when the request arrived over HTTPS (directly, or through a proxy covered by PULLMD_TRUST_PROXY) or when PUBLIC_URL starts with https://. POST /login allows 10 attempts per minute and POST /signup 5 per hour per client address (429 with Retry-After beyond that). Login, signup, logout and OAuth consent form posts are accepted only when Origin (or Referer) names this host or the host of PUBLIC_URL; posts without either header are accepted.
2. API keys — generate at /settings, send via Authorization: Bearer pmd_<32-char-base62>. Stored as SHA-256 hashes; only shown once at creation.
3. Legacy PULLMD_AUTH_TOKEN — deprecated. single-admin mode only. Maps to admin user. Kept for migration compatibility; slated for removal in a future major release.

⁠Migration from v1.x

See MIGRATION.md for the full upgrade checklist. The TL;DR: leave PULLMD_AUTH_MODE unset and v2.0 behaves exactly like v1.x.

⁠OAuth 2.1 (claude.ai Web Connector)

PullMD ships with a full OAuth 2.1 Authorization Code flow so the claude.ai web app's Custom Connector feature can authenticate users against your PullMD instance. All endpoints needed by the spec are implemented: Dynamic Client Registration (RFC 7591), PKCE-S256 (RFC 7636), Authorization Server Metadata (RFC 8414), Protected Resource Metadata (RFC 9728), and Token Revocation (RFC 7009).

Setup:

1. Set PULLMD_AUTH_MODE to single-admin or multi-user (OAuth requires Phase-1 auth).
2. Set OAUTH_JWT_SECRET to a 32+ character random string (openssl rand -hex 32).
3. Set PUBLIC_URL to your instance's public origin (e.g. https://pullmd.example.com).
4. In claude.ai → Settings → Conn