iarsalanshah/ghostfetch repository overview

⁠GhostFetch

Fetch the unfetchable. A stealthy, headless browser service for AI agents.

GhostFetch bypasses anti-bot protections to fetch content from difficult sites (like X.com and LinkedIn) and converts it into clean, LLM-ready Markdown. It handles the complexity of headless browsing, proxy rotation, and fingerprinting so your agent doesn't have to.

Powered by the 🦊 Phantom Fox — slipping past every wall, unseen.

⁠Why GhostFetch?

Fetching content for AI agents is hard. Simple requests or curl calls fail on modern sites due to JavaScript rendering and anti-bot checks. Heavy browser automation tools are slow and complex to manage.

GhostFetch solves this by providing:

• 🦊 Stealth by Design: "Ghost Protocol" fingerprinting to mimic real users.
• 📜 LLM-Native Output: Returns clean Markdown, not messy HTML.
• 📜 Smart Scrolling: Automatically expands infinite feeds (perfect for X/Twitter threads).
• 🔐 Authenticated Sessions: Bypass login walls for LinkedIn, X/Twitter, and other gated sites.
• ⚡ Zero-Config: Browsers auto-install and manage themselves.

⁠Architecture

URL → GhostFetch (Headless Browser + Ghost Protocol) → Markdown → AI Agent

⁠🚀 Quick Start

The fastest way to get started is via pip.

⁠1. Install

pip install ghostfetch

⁠2. Fetch a URL

Browsers will auto-install on the first run.

ghostfetch "https://x.com/user/status/123"

Output:

{
  "metadata": { "title": "...", "author": "..." },
  "markdown": "Captured content in markdown format..."
}

⁠3. Fetch Behind a Login Wall (LinkedIn, etc.)

# Open a visible browser to log in — session is saved for reuse
ghostfetch auth login --domain linkedin.com --login-url https://www.linkedin.com/login

# Fetch content using the saved session
ghostfetch "https://www.linkedin.com/in/profile" --auth-session-id <SESSION_ID>

⁠✨ Features

• ⚡ Synchronous & Async API: Flexible integration patterns.
• 🦊 Ghost Protocol: Advanced proxy rotation and cohesive browser fingerprinting.
• 📜 Smart Scrolling: Auto-detects and scrolls infinite feeds to capture full content.
• 🐦 X.com Optimized: Special handling for Twitter/X hydration and thread expansion.
• 🔍 Metadata Extraction: Auto-extracts title, author, date, and images.
• 📬 Job Queue: Built-in async job system with webhooks and retries.
• 🍪 Persistent Sessions: Cookie/localStorage persistence per domain.
• 🔐 Authenticated Sessions: Domain-locked session management for login-gated pages (LinkedIn, X, etc.).
• 🐳 Docker Ready: Production-ready container images included.

⁠📦 Installation

⁠Option 1: Python Package (Best for Agents)

pip install ghostfetch
# Usage:
# ghostfetch "url"             (CLI)
# from ghostfetch import fetch (Python SDK)

⁠Option 2: Docker (Best for Services)

docker run -p 8000:8000 iarsalanshah/ghostfetch
# Service available at http://localhost:8000

⁠Option 3: Manual / Source

git clone https://github.com/iArsalanshah/GhostFetch.git
cd GhostFetch
pip install -e .
playwright install chromium

⁠🧰 Usage

⁠CLI

# JSON output for parsing
ghostfetch "https://example.com" --json

# Metadata only
ghostfetch "https://example.com" --metadata-only

# Fetch behind a login wall
ghostfetch "https://linkedin.com/in/profile" --auth-session-id abc123

# Manage authenticated sessions
ghostfetch auth login --domain linkedin.com
ghostfetch auth status
ghostfetch auth revoke <SESSION_ID>

⁠Python SDK

from ghostfetch import fetch

# Simple fetch
result = fetch("https://example.com")
print(result['markdown'])

# Fetch behind a login wall
result = fetch("https://linkedin.com/in/profile", auth_session_id="abc123")
print(result['markdown'])

⁠REST API

Start the server:

ghostfetch serve

By default, fetch endpoints require API key auth:

export GHOSTFETCH_API_KEY="replace-with-strong-token"

Synchronous Fetch (Blocks until done):

curl -H "X-API-Key: $GHOSTFETCH_API_KEY" \
  "http://localhost:8000/fetch/sync?url=https://example.com"

Asynchronous Fetch (Background Job):

curl -X POST "http://localhost:8000/fetch" \
     -H "Content-Type: application/json" \
     -H "X-API-Key: $GHOSTFETCH_API_KEY" \
     -d '{"url": "https://example.com", "callback_url": "https://yourapp.com/webhook"}'

Fetch with Authenticated Session:

curl -X POST "http://localhost:8000/fetch/sync" \
     -H "Content-Type: application/json" \
     -H "X-API-Key: $GHOSTFETCH_API_KEY" \
     -d '{"url": "https://linkedin.com/in/profile", "auth_session_id": "abc123"}'

Import an Auth Session (programmatic):

curl -X POST "http://localhost:8000/auth/sessions/import" \
     -H "Content-Type: application/json" \
     -H "X-API-Key: $GHOSTFETCH_API_KEY" \
     -d '{"domain": "linkedin.com", "storage_state": {...}, "ttl_seconds": 86400}'

List Auth Sessions:

curl -H "X-API-Key: $GHOSTFETCH_API_KEY" \
  "http://localhost:8000/auth/sessions"

Revoke an Auth Session:

curl -X DELETE "http://localhost:8000/auth/sessions/abc123" \
  -H "X-API-Key: $GHOSTFETCH_API_KEY"

Check Health:

curl "http://localhost:8000/health"

Check Job Status:

curl -H "X-API-Key: $GHOSTFETCH_API_KEY" \
  "http://localhost:8000/job/a1b2c3d4-e5f6-7890"

⁠Response Format

All successful fetches return a standardized JSON structure:

{
  "metadata": {
    "title": "Page Title",
    "author": "Author Name",
    "publish_date": "2023-01-01",
    "images": ["image_url.jpg"]
  },
  "markdown": "# Page Title\n\nExtracted content...",
  "url": "https://example.com/original-url",
  "status": "success"
}

Auth Wall Detection: When fetching login-gated pages without a valid session, the response status will be one of:

• auth_required — The page requires a login.
• auth_expired — The saved session has expired.
• auth_challenge — An additional security challenge (e.g., CAPTCHA) was encountered.

⁠📊 Configuration

GhostFetch is configured via environment variables.

Proxies: Create a proxies.txt file in the working directory with one proxy per line: http://user:pass@host:port

Concurrency note: MAX_CONCURRENT_BROWSERS is a global browser-context cap shared by both sync and async fetch paths.

Auth session storage note: Authenticated session state files are stored under STORAGE_DIR/auth_sessions/ and may contain sensitive cookies. Keep STORAGE_DIR on a private filesystem with restricted access.

⁠📈 Advanced Usage

For GitHub integration, MCP Server configuration (Claude Desktop), and production deployment guides (Docker Compose, Proxy strategies), please see:

👉 Advanced Usage & Deployment Guide⁠

⁠🤖 Agent Plugins Integration

GhostFetch is packaged as a portable Agent Plugin conforming to the open, vendor-neutral Agent Plugins 1.0.0⁠ standard. Any conformant agent client (Cursor, Claude Code, OpenAI Codex, etc.) can discover and load GhostFetch as a plugin.

⁠Plugin Structure

The plugin consists of two manifest files at the repository root:

• plugin.json — The Agent Plugins manifest declaring the plugin metadata (name, version, author, license, keywords).
• mcp.json — The MCP server connection configuration in Agent Plugins format.

Additionally, the plugin includes:

• skills/ghostfetch/SKILL.md — An Agent Skill (per the Agent Skills specification⁠) that teaches agents how to use GhostFetch.

⁠Loading the Plugin

Conformant agent clients can load GhostFetch by pointing to this repository:

# Example: clone and install as a plugin
git clone https://github.com/iArsalanshah/GhostFetch.git
# The client reads plugin.json + mcp.json to discover capabilities

The mcp.json declares a single MCP server ghostfetch that runs via stdio:

• Command: python -m ghostfetch.mcp_server
• Environment: SYNC_TIMEOUT_DEFAULT=120, MAX_SYNC_TIMEOUT=300

⁠Standalone MCP Server

You can also run the MCP server directly without a plugin client:

# Start the MCP server (stdio transport)
python -m ghostfetch.mcp_server

# Or with custom timeouts
SYNC_TIMEOUT_DEFAULT=60 MAX_SYNC_TIMEOUT=180 python -m ghostfetch.mcp_server

The server exposes GhostFetch's fetching capabilities as MCP tools that any MCP-compatible client can invoke.

⁠🛠 Troubleshooting

• Browser Executable Missing: Run playwright install chromium.
• Timeouts: Increase timeout in request or SYNC_TIMEOUT_DEFAULT env var.
• Memory Issues: Reduce MAX_CONCURRENT_BROWSERS.

⁠🤝 Contributing

PRs welcome. Open an issue for major changes.

⁠⚠️ Legal Disclaimer

For educational and research purposes only. Users are responsible for complying with the Terms of Service, robots.txt, and applicable laws of the websites they access. This tool should not be used for unauthorized scraping or circumventing security measures in violation of law.

⁠License

MIT License. See LICENSE⁠ for details.