aeternalabshq/pullmd repository overview

⁠PullMD

Anything in, clean Markdown out. Self-hosted conversion of web pages, documents, images, audio and YouTube videos into token-lean Markdown, for you and for your AI agents.

Documentation: aeternalabshq.github.io/pullmd⁠, with installation, configuration, guides, the API reference and the changelog.

Quick start⁠ · Use with AI agents⁠ · Docs⁠ · API reference⁠ · Configuration⁠ · Changelog⁠

⁠Why PullMD

Agents are bad at reading the web. Raw HTML burns context on navigation, cookie banners and scripts, and built-in fetch tools often stumble on JavaScript-heavy pages, PDFs and comment threads. PullMD sits in between: one URL or file in, clean Markdown out, with source and metadata in YAML frontmatter instead of the body.

curl "https://pullmd.example.com/api?url=https://example.com/article"

What that saves, measured on three public pages (tokens counted with the cl100k_base tokenizer, 2026-10-16):

The curl column is what a naive fetch tool hands an agent. The last column is query-scoped extraction⁠: only the sections relevant to a question.

⁠Highlights

Reads almost anything

• Web pages through a fallback chain: Cloudflare's native Markdown, then Readability + Trafilatura (best result wins), then headless Chromium for JavaScript-heavy sites
• Reddit and Hacker News threads with their comment trees (depth and count configurable; Reddit needs API credentials⁠)
• Documents: PDF, Word, PowerPoint, Excel, EPUB, by URL or upload, plus an optional OCR tier for table-grade PDFs
• YouTube title, description and transcript with clickable timecodes, no API key needed
• Images and audio via any OpenAI-compatible or local model (captions, transcripts)

Built for agents

• MCP server for Claude Code and other MCP clients, with OAuth 2.1 for claude.ai and Claude Desktop
• Query-scoped extraction: ?query= returns only the sections relevant to a question, typically 70-95% fewer tokens on long pages
• Lean output: clean body, metadata in frontmatter, unpadded tables, configurable field allowlist
• Live share links: /s/:id re-fetches when the cached copy is older than an hour and serves the last good snapshot if the source dies

Comfortable to self-host

• One docker compose up, zero config, multi-arch images (amd64/arm64) on Docker Hub and GHCR
• Every sidecar is optional, and PullMD degrades gracefully without it
• Auth modes from open homelab to multi-user, SSRF protection, rate limits, a health endpoint for Uptime Kuma
• Per-site extraction recipes as declarative JSON

Four ways in: PWA · REST API · MCP server · Claude Code skill

⁠Quick start

mkdir pullmd && cd pullmd
curl -O https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/docker-compose.yml
docker compose up -d
# -> http://localhost:3000

No .env needed: every variable has a default. Add a .env next to the compose file to override anything. This starts PullMD and its three optional sidecars (Trafilatura, Playwright, MarkItDown); the Playwright sidecar alone adds about 3.7 GB of browser binaries. Images are on Docker Hub and on GHCR (ghcr.io/aeternalabshq/pullmd, -trafilatura, -playwright, -markitdown).

More in the docs:

• Installation⁠: compose files, running without a sidecar, running without Docker
• Reverse proxy⁠: Traefik, nginx, Caddy, PULLMD_TRUST_PROXY
• Configuration⁠: every environment variable
• Authentication⁠: single-admin, multi-user, API keys, OAuth
• Upgrading⁠ and MIGRATION.md⁠

⁠Use with AI agents

Once your instance runs, /help on it shows every install path with your URL filled in. Replace ${PULLMD_URL} below with your instance's address.

MCP server (Claude Code, Cursor, other MCP clients; tools read_url, get_share, list_recent):

claude mcp add --transport http pullmd ${PULLMD_URL}/mcp

Claude Code skill:

curl -O ${PULLMD_URL}/pullmd.zip
mkdir -p ~/.claude/skills && unzip pullmd.zip -d ~/.claude/skills/

Any chat agent: paste the prompt from /help or from the docs.

Details: MCP server⁠ · claude.ai and Claude Desktop (OAuth)⁠ · Claude Code skill⁠ · Prompt for any agent⁠ · MCP tools reference⁠

⁠Reddit access

PullMD reads Reddit through Reddit's Data API, using REDDIT_CLIENT_ID and REDDIT_CLIENT_SECRET from an existing app. Without them Reddit blocks the requests and /api answers 502.

Reddit stopped accepting new requests for API access on 2026-10-31. It says it will start removing access for apps and users that have not registered on 2027-01-12, and close public access for all remaining apps in March 2027 (announcement⁠). PullMD's Reddit support ends with that access. PullMD has no login or session cookie option, and none is planned.

More: Reddit and Hacker News guide⁠.

⁠Documentation

AI tools can read the whole site as llms.txt⁠. The docs live in website/⁠; pages are Markdown, and pull requests are welcome.

⁠Architecture

• server.js - Express app factory (createApp) with dependency injection for tests. Exposes /api and /api/stream (SSE).
• lib/reddit.js - Reddit URL normalization, redirect resolution, post + comment extraction.
• lib/web.js - Orchestrator: Cloudflare-Markdown short-circuit, then static Readability + Trafilatura with pickBest, then optional Playwright re-render + re-extract on body-soup / low-quality output.
• lib/render-decision.js - Predicate that decides when to fall back to Playwright (readability-fellback + thin, body-soup signature, or quality < 0.5; plus force / skip overrides).
• lib/playwright-client.js - HTTP client for the Playwright sidecar with AbortSignal propagation for SSE-disconnect cancellation.
• lib/scoring.js - Quality scoring used to pick between extractors and as a render-trigger heuristic.
• lib/cache.js - SQLite cache (better-sqlite3) with configurable retention (default 90 days) and 32-hex (128-bit) share ids.
• lib/mcp.js - Stateless MCP server registering the three tools.
• lib/distrib.js - Public-URL substitution in /help and /pullmd.zip.
• trafilatura-sidecar/ - Python sidecar (FastAPI) wrapping Trafilatura.
• playwright-sidecar/ - Python sidecar (FastAPI + Playwright + Chromium) for JS-rendered pages.
• markitdown-sidecar/ - Python sidecar (FastAPI) wrapping MarkItDown for document conversion (PDF, Office, EPUB, …) plus the keyless /youtube transcript endpoint.
• lib/markitdown-client.js - HTTP client for the markitdown sidecar (documents + YouTube).
• lib/youtube.js - YouTube URL detection and normalization.
• lib/llm/ - provider resolver + adapters for image captioning (vision), audio transcription (STT), and PDF OCR against OpenAI-compatible endpoints.
• lib/frontmatter.js - YAML frontmatter builder with the PULLMD_FRONTMATTER_FIELDS allowlist.
• lib/coverage-guard.js - detects extractions that kept only a sliver of the page and recovers the container holding the body.
• lib/article-series.js - detects pages that split one piece of writing across several <article> blocks by the same author and joins them.
• lib/query-extract.js - Opt-in query-scoped extraction (?query=): BM25 over heading-based sections of the converted markdown, returning only the relevant sections/blocks. Composes lib/markdown-sections.js (block/section parser) and lib/bm25.js (tokenizer + scorer).
• public/ - PWA frontend (vanilla JS, dark/paper themes, service worker, EventSource client for /api/stream).
• skill/pullmd/ - Claude Code skill source (templated with __PULLMD_URL__).

⁠License

GNU AGPL v3⁠ - Copyright © 2026 Aeterna Labs.

PullMD is free software: you can redistribute it and modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, version 3 or later. If you run a modified version as a network service, you must make your modifications available to its users.