aeternalabshq/pullmd repository overview

⁠PullMD

Self-hosted URL-to-Markdown service for humans and AI agents.

PullMD takes any web URL and returns clean, readable Markdown — no navigation, no ads, no boilerplate. It auto-detects Reddit and Hacker News threads (with full comment trees), uses Cloudflare's native Markdown when available, runs Mozilla Readability + Trafilatura on static HTML, and as a last resort renders JavaScript-heavy pages via headless Chromium (Playwright sidecar) before extracting.

As of v3, PullMD goes beyond web pages: it also converts documents (PDF, Office, EPUB), images, audio, and YouTube videos to Markdown, and emits a leaner, token-efficient body by default. See What's new in v3⁠ below.

It ships as:

• a PWA frontend with raw/rendered and live-frontmatter view toggles, one-tap sharing of the output to other apps (Web Share API), a download button that saves the result as a .md file under the server-suggested name, dark/paper themes, history, archive, share links, and conversion of local HTML files (drag-and-drop on desktop, file picker on desktop and mobile)
• a REST API at GET /api?url=…
• an MCP server at POST /mcp (Streamable-HTTP transport, stateless)
• a Claude Code skill as a downloadable zip

Every conversion gets a 32-hex share id that works as a stable live-endpoint: GET /s/:id returns the cached markdown and re-fetches from the source if older than one hour. Use the share id as a fixed URL that always returns fresh content — useful for subreddit feeds and similar.

⁠What's new in v3

PullMD v3 grows from a web-page reader into a general anything-to-Markdown service for agents, with a leaner default output. Everything beyond plain web extraction is opt-in and degrades gracefully - left unconfigured, v3 handles web pages exactly like v2, just with a cleaner body by default.

• Clean body by default - the Markdown body is now just # Title + content. The source URL, fetch date, and all metadata moved into the YAML frontmatter, so nothing is duplicated and you spend fewer tokens. Reddit posts follow the same rule: subreddit, author, upvotes, and publish date live in the frontmatter (subreddit, author, upvotes, published), not the body. This is the one breaking change: set PULLMD_SOURCE_HEADER=true to restore the old inline header, and use PULLMD_FRONTMATTER_FIELDS⁠ to trim which fields are emitted. See MIGRATION.md⁠.
• Documents → Markdown - PDF, Word, PowerPoint, Excel, EPUB and more, by URL or upload⁠ (POST /api/file, drag-and-drop in the PWA).
• High-quality PDF tables (OCR) - an opt-in, vendor-neutral OCR tier⁠ (?pdf=ocr) for table-grade PDF conversion, with automatic fallback to the free path.
• Images & audio → Markdown - opt-in captioning and transcription⁠ via any OpenAI-compatible or local model; runs inside pullmd, no extra container required.
• YouTube transcripts - title, description and transcript⁠ with clickable timecodes, no API key required.
• Richer frontmatter - extraction source, quality, and (for media/OCR) model + token/page usage⁠ for cost tracking, plus a configurable field allowlist.

Added in the 3.x line since then:

• Hacker News pipeline (3.1) - items, comment permalinks and listings through a purpose-built converter, plus Web Share and an instant frontmatter toggle in the PWA.
• X-Transcript-Status (3.2) - tells a transient YouTube rate-limit apart from a genuinely missing transcript.
• SSRF protection⁠ (3.3) - private, loopback, link-local, CGNAT and cloud-metadata targets are rejected by default, on every fetch path and every redirect hop.
• Query-scoped extraction⁠ (3.4) - ?query= returns only the sections relevant to a question, with a max_tokens budget.
• Site recipes opened up⁠ (3.5/3.6) - JSON-LD-to-frontmatter, a contributor guide, and select.content so a recipe can name the article body outright.
• Coverage guard (3.7) - recovers pages where extraction kept only a sliver of the body; see PULLMD_COVERAGE_GUARD⁠.
• Account controls (3.8) - a non-admin can clear entries from their own history, self-registration can be closed with PULLMD_ALLOW_SIGNUP⁠, and scripts/admin.js create-user creates accounts from the shell.
• Download button (3.9) - the PWA saves a result as a .md file, named by the server via X-Suggested-Filename⁠ and optionally date-prefixed.
• Sidecar health endpoint⁠ (3.10) - GET /api/status answers 503 when a configured sidecar stops responding, so a dead renderer shows up as an alert instead of quietly degrading extraction.
• Configurable cache retention (3.11) - PULLMD_CACHE_RETENTION_DAYS⁠ sets how long cache rows and share links live; default 90 days, 0 keeps them forever.

⁠Quick start

Pre-built multi-arch images (linux/amd64, linux/arm64) live on Docker Hub. Drop the compose file somewhere and run:

mkdir pullmd && cd pullmd
curl -O https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/docker-compose.yml
docker compose up -d
# → http://localhost:3000

That's it. No .env needed: every variable has a sensible default and PullMD listens on port 3000. Add a .env next to the compose file to override anything (see Configuration⁠).

⁠docker-compose.yml (zero-config, abridged)

services:
  pullmd:
    image: aeternalabshq/pullmd:latest
    container_name: pullmd
    restart: unless-stopped
    ports:
      - "${PORT:-3000}:3000"
    environment:
      - PUBLIC_URL=${PUBLIC_URL:-http://localhost:${PORT:-3000}}
      - TRAFILATURA_URL=http://trafilatura:8001/extract
      - PLAYWRIGHT_URL=http://playwright:8002/render
      - MARKITDOWN_URL=http://markitdown:8003/convert
      - CACHE_DB=/data/cache.db
    volumes:
      - ./data:/data
    networks:
      - pullmd-internal
    depends_on:
      - trafilatura
      - playwright
      - markitdown

  trafilatura:
    image: aeternalabshq/pullmd-trafilatura:latest
    container_name: pullmd-trafilatura
    restart: unless-stopped
    networks:
      - pullmd-internal

  playwright:
    image: aeternalabshq/pullmd-playwright:latest
    container_name: pullmd-playwright
    restart: unless-stopped
    networks:
      - pullmd-internal

  markitdown:
    image: aeternalabshq/pullmd-markitdown:latest
    container_name: pullmd-markitdown
    restart: unless-stopped
    mem_limit: ${MARKITDOWN_MEM_LIMIT:-1g}
    networks:
      - pullmd-internal

networks:
  pullmd-internal:
    driver: bridge

⁠Behind Traefik

For deployments behind Traefik with TLS, use docker-compose.traefik.yml instead. Same images, but with Traefik labels and the proxy external network. Set HOST_DOMAIN in .env:

curl -O https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/docker-compose.traefik.yml
echo "HOST_DOMAIN=pullmd.example.com" > .env
docker compose -f docker-compose.traefik.yml up -d

⁠Local development (no Docker)

git clone https://github.com/AeternaLabsHQ/pullmd.git
cd pullmd
npm install
npm start             # http://localhost:3000
npm test              # node --test

⁠Configuration

All variables go in .env (copy from .env.example):

PUBLIC_URL matters for self-hosting: the help page and downloadable skill embed it as the canonical endpoint. Set it correctly and your users get a copy-paste setup that points at your instance.

PullMD rotates its outbound User-Agent for the web fetch path from a pool of current desktop browsers, refreshed every 48 hours from a live feed of real-world UAs⁠ maintained by @WinFuture23⁠. A built-in seed pool ensures rotation works even when the feed is unreachable. Set PULLMD_USER_AGENT to pin a single UA, or PULLMD_UA_FEED_URL to point at your own feed. The Reddit path keeps its dedicated REDDIT_USER_AGENT because Reddit's API expects a stable, identifying UA.

DISABLE_PUBLIC_HISTORY=true is the privacy switch for shared instances (multi-tenant VPS, office deployments). Conversions still get cached and assigned share IDs; users just can't see what other users have fetched. Anyone with a known /s/:id link still gets their markdown back. Share ids are 128 random bits, so a link cannot be guessed, and /s/:id throttles unknown-id lookups to 120 per minute and IP (valid links are never throttled). Use this as a stopgap until per-user scoping lands.

⁠Authentication (v2.0+)

PullMD ships with three auth modes. Pick one with PULLMD_AUTH_MODE:

In single-admin and multi-user modes, PULLMD_ADMIN_EMAIL + PULLMD_ADMIN_PASSWORD bootstrap the first admin user on first startup. After that, changing these env vars does not change the password — use the admin CLI:

docker compose exec pullmd node scripts/admin.js reset-password [email protected]

Create an account without opening self-registration (useful when PULLMD_ALLOW_SIGNUP is off):

docker compose exec pullmd node scripts/admin.js create-user [email protected]

Both commands read the password from stdin, so they need it attached: docker compose exec, docker exec -it, or a pipe (echo "…" | docker exec -i <container> node scripts/admin.js …). A plain docker exec without -i aborts with an error and exit code 2 instead of doing nothing.

⁠Auth boundary

Cache deletes are scoped to the caller. An admin (and every caller in disabled mode) removes the shared, URL-deduped cache row, which affects every user's history. A regular user only unlinks the entry from their own history - the shared row and its /s/:id share link keep working. The response says which happened via "scope": "user" | "global".

⁠Authentication paths

1. Session cookies — POST /login sets pullmd_session (HttpOnly, SameSite=Lax, Secure over HTTPS, 90-day TTL with sliding expiry). The PWA uses this automatically.
2. API keys — generate at /settings, send via Authorization: Bearer pmd_<32-char-base62>. Stored as SHA-256 hashes; only shown once at creation.
3. Legacy PULLMD_AUTH_TOKEN — deprecated. single-admin mode only. Maps to admin user. Kept for migration compatibility; slated for removal in a future major release.

⁠Migration from v1.x

See MIGRATION.md for the full upgrade checklist. The TL;DR: leave PULLMD_AUTH_MODE unset and v2.0 behaves exactly like v1.x.

⁠OAuth 2.1 (claude.ai Web Connector)

PullMD ships with a full OAuth 2.1 Authorization Code flow so the claude.ai web app's Custom Connector feature can authenticate users against your PullMD instance. All endpoints needed by the spec are implemented: Dynamic Client Registration (RFC 7591), PKCE-S256 (RFC 7636), Authorization Server Metadata (RFC 8414), Protected Resource Metadata (RFC 9728), and Token Revocation (RFC 7009).

Setup:

1. Set PULLMD_AUTH_MODE to single-admin or multi-user (OAuth requires Phase-1 auth).
2. Set OAUTH_JWT_SECRET to a 32+ character random string (openssl rand -hex 32).
3. Set PUBLIC_URL to your instance's public origin (e.g. https://pullmd.example.com).
4. In claude.ai → Settings → Connectors → Add custom connector, point it at https://pullmd.example.com/mcp — claude.ai discovers everything else automatically via the well-known endpoints.
5. The first time the user clicks the connector, they'll be redirected to PullMD's /login, then to a consent screen, then back to claude.ai.

Tokens:

• Access tokens are JWTs (HS256), TTL 1 hour, audience-bound to your /mcp URL.
• Refresh tokens are opaque (pmd_rt_…), TTL 30 days, rotated on every refresh, with reuse-detection that invalidates the entire refresh chain on replay.
• Revoke a token via POST /oauth/revoke (RFC 7009).

Scope: Currently a single mcp:full scope (URL conversion + history read). Granular scopes are tracked for a future minor release.

Shipped in v2.3.0 (issues #6⁠ and #10⁠).

⁠AI-agent integration

Three install paths. Once your instance is running, ${PULLMD_URL}/help shows the same boxes with your URL pre-filled. Replace ${PULLMD_URL} below with your hostname (e.g. https://pullmd.example.com).

⁠1. Universal prompt

Drop into any chat agent (ChatGPT, Claude, Gemini, …):

When you need to read a web page, fetch it via PullMD instead of your
built-in fetch/browse tool - not just when that one fails:

  GE