edgaras0x4e/web-loader-engine repository overview

⁠Web Loader Engine

High-performance web content extraction engine built in Rust. Primary purpose is serving as an external web loader for OpenWebUI⁠, but it's flexible enough for any use case that needs clean content extraction from web pages - RAG pipelines, content indexing, web scraping, archiving, and more.

⁠Features

• OpenWebUI Compatible - Native API support, drop-in replacement
• Multiple Output Formats - Markdown, HTML, plain text, screenshots
• Readability Extraction - Mozilla Readability algorithm for clean article content
• JavaScript Rendering - Chromium-based rendering for JS-heavy sites
• Smart Caching - Built-in response caching with configurable TTL
• Rate Limiting - Per-domain rate limiting and circuit breakers
• Batch Processing - Process multiple URLs concurrently
• Security - SSRF protection, blocked internal IPs, optional API key auth
• Egress Proxy Support - Honors HTTPS_PROXY/HTTP_PROXY/NO_PROXY for both HTTP client and Chromium browser traffic

⁠Quick Start

⁠Docker (Recommended)

docker build -t web-loader-engine .
docker run -d -p 14786:14786 --name web-loader web-loader-engine

⁠Docker Compose

services:
  web-loader:
    build: .
    ports:
      - "14786:14786"
    environment:
      - BROWSER_POOL_SIZE=10
      - CACHE_TTL=3600
      # - API_KEY=your-secret-key
    volumes:
      - screenshots:/app/screenshots
    restart: unless-stopped

volumes:
  screenshots:

docker-compose up -d

⁠Docker Hub (Pre-built Image)

services:
  web-loader:
    image: edgaras0x4e/web-loader-engine:latest
    ports:
      - "14786:14786"
    environment:
      - BROWSER_POOL_SIZE=10
      - CACHE_TTL=3600
      # - API_KEY=your-secret-key
    volumes:
      - screenshots:/app/screenshots
    restart: unless-stopped

volumes:
  screenshots:

docker-compose up -d

Then set OpenWebUI's web loader URL to http://web-loader:14786

⁠From Source

Requires Rust 1.70+ and Chrome/Chromium installed.

cp .env.example .env  # Configure settings
cargo build --release
./target/release/web-loader-engine

⁠Configuration

Copy the example environment file and adjust as needed:

cp .env.example .env

Environment variables:

⁠API

⁠OpenWebUI Endpoint

POST /

{"urls": ["https://example.com/article"]}

Returns:

[
  {
    "page_content": "# Article Title\n\nContent...",
    "metadata": {
      "source": "https://example.com/article",
      "title": "Article Title"
    }
  }
]

⁠Single URL

POST /load

{"url": "https://example.com"}

Response:

{
  "url": "https://example.com",
  "title": "Example Domain",
  "content": "# Example Domain\n\nThis domain is for examples...",
  "metadata": {
    "processing_time_ms": 1234,
    "cached": false
  }
}

⁠Batch

POST /load/batch

{"urls": ["https://example.com/1", "https://example.com/2"]}

Response:

{
  "results": [
    {
      "url": "https://example.com/1",
      "response": {
        "url": "https://example.com/1",
        "title": "Page Title",
        "content": "...",
        "metadata": {"processing_time_ms": 500, "cached": false}
      }
    }
  ],
  "total_processing_time_ms": 1234
}

⁠Health Check

GET /health

⁠Request Headers

⁠Request Body Options (all optional)

{
  "url": "https://example.com",
  "options": {
    "wait_for_selector": "#content",
    "target_selector": "article",
    "remove_selector": ".ads",
    "timeout": 60
  }
}

⁠Screenshots

Set x-respond-with to either screenshot (viewport only) or pageshot (full scrolling page). The API renders the page in headless Chromium, saves the PNG to SCREENSHOT_DIR, and returns a relative URL you can fetch from the same server.

⁠Viewport screenshot

curl -X POST http://localhost:14786/load \
  -H "Content-Type: application/json" \
  -H "x-respond-with: screenshot" \
  -d '{"url": "https://example.com"}'

Response:

{
  "url": "https://example.com",
  "title": null,
  "content": "",
  "screenshot_url": "/screenshots/httpsexamplecom_441d3714-d010-4eb4-a729-606873b081d9.png",
  "metadata": {"processing_time_ms": 1064, "cached": false}
}

Fetch the PNG:

curl -o page.png \
  http://localhost:14786/screenshots/httpsexamplecom_441d3714-d010-4eb4-a729-606873b081d9.png

⁠Full-page screenshot

curl -X POST http://localhost:14786/load \
  -H "Content-Type: application/json" \
  -H "x-respond-with: pageshot" \
  -d '{"url": "https://example.com"}'

⁠Wait for content before capturing

Combine with x-wait-for-selector so the screenshot is only taken once a specific element has rendered:

curl -X POST http://localhost:14786/load \
  -H "Content-Type: application/json" \
  -H "x-respond-with: screenshot" \
  -H "x-wait-for-selector: article h1" \
  -d '{"url": "https://example.com/post/123"}'

⁠With API key

curl -X POST http://localhost:14786/load \
  -H "Authorization: Bearer your-secret-key" \
  -H "Content-Type: application/json" \
  -H "x-respond-with: screenshot" \
  -d '{"url": "https://example.com"}'

curl -H "Authorization: Bearer your-secret-key" \
  -o page.png \
  http://localhost:14786/screenshots/httpsexamplecom_441d3714-d010-4eb4-a729-606873b081d9.png

⁠Storage

• Files are written to SCREENSHOT_DIR (default /app/screenshots in Docker, configurable via env)
• Filenames include a UUID to avoid collisions
• When using Docker, mount a volume at /app/screenshots to persist captures across container restarts

⁠User Agents

Three ways to control which User-Agent is sent with a request:

1. Configured default - set DEFAULT_USER_AGENT in the environment. Used when rotation is off and no header is provided.
2. Rotation pool - set USER_AGENT_ROTATION=round_robin or random plus USER_AGENT_POOL (or USER_AGENT_POOL_FILE). The server picks a different UA per request.
3. Per-request override - send x-user-agent on the individual call.

Resolution order per request: explicit header > rotation (if enabled) > configured default.

⁠Pool from an inline list

Separate UAs with | or newlines:

USER_AGENT_ROTATION=round_robin
USER_AGENT_POOL="Mozilla/5.0 ...Chrome/120...|Mozilla/5.0 ...Firefox/121..."

⁠Pool from a file

One UA per line, # lines are comments. Takes precedence over USER_AGENT_POOL if both are set.

USER_AGENT_ROTATION=random
USER_AGENT_POOL_FILE=/etc/web-loader/user-agents.txt

Sample user-agents.txt:

# Desktop Chrome
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

# Desktop Firefox
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0

⁠Explicit UA per request

curl -X POST http://localhost:14786/load \
  -H "Content-Type: application/json" \
  -H "x-user-agent: MyBot/1.0 (+https://example.com/bot)" \
  -d '{"url": "https://httpbin.org/user-agent"}'

⁠Force rotation on a single request (even when rotation is off globally)

curl -X POST http://localhost:14786/load \
  -H "Content-Type: application/json" \
  -H "x-user-agent: rotate" \
  -d '{"url": "https://httpbin.org/user-agent"}'

⁠Force the configured default (bypass rotation for one request)

curl -X POST http://localhost:14786/load \
  -H "Content-Type: application/json" \
  -H "x-user-agent: default" \
  -d '{"url": "https://httpbin.org/user-agent"}'

⁠Other Use Cases

While built for OpenWebUI, this works for:

• RAG Pipelines - Clean content for embeddings and retrieval
• Content Archiving - Save readable versions of web pages
• Web Scraping - Extract data from JavaScript-rendered pages
• Screenshot Services - Programmatic page captures
• Search Indexing - Extract text content for indexing

⁠Changelog

⁠v0.1.4

User Agent Rotation & Browser Log Control - Added configurable user agents and a way to silence warnings.

• New USER_AGENT_ROTATION env var with strategies off (default), round_robin, random - rotates per request
• Provide the pool via USER_AGENT_POOL (inline, |- or newline-separated) or USER_AGENT_POOL_FILE (path to a file, one UA per line, # comments supported). The file takes precedence when both are set
• DEFAULT_USER_AGENT overrides the hardcoded default used when rotation is off and no header is set
• x-user-agent header now accepts special values: rotate forces rotation even when USER_AGENT_ROTATION=off, and default forces the configured default
• Precedence: explicit header → rotation (if enabled) → configured default. Empty pool safely falls back to the default with a warning at startup
• New BROWSER_LOG_LEVEL env var (default error) silences chromiumoxide's noisy WS Invalid message warnings emitted when Chromium sends CDP events the driver doesn't yet model. Accepts off, error, warn, info, debug, trace - operates independently of RUST_LOG

⁠v0.1.3

Chromium Egress Proxy Support - Chromium now honors HTTPS_PROXY/HTTP_PROXY/NO_PROXY from the environment so the browser's outbound traffic can be routed through an egress proxy.

• On launch, if HTTPS_PROXY (or HTTP_PROXY as fallback) is set, Chromium is started with --proxy-server=<url>
• If NO_PROXY is set, its value is translated to Chrome's bypass-list syntax and passed via --proxy-bypass-list=<list> (commas → semicolons, *.domain → .domain)
• When no proxy env vars are set, behavior is unchanged - dev/local runs need no configuration
• The Rust HTTP client (reqwest) already honors these vars natively, so direct HTTP fetches and browser fetches now share the same egress path

⁠v0.1.2

Screenshot Delivery Fix - Screenshot URLs returned by the API are now actually reachable.

• Fixed issue where /load responses advertised a screenshot_url that returned 404 when fetched
• Saved screenshots are now served directly from the configured SCREENSHOT_DIR
• Safe by design: path-traversal attempts (e.g. /screenshots/../etc/passwd) return 404
• Respects the same API key authentication as the rest of the API when one is configured

⁠v0.1.1

Browser Pool Resilience - Fixed critical issue where dead browser connections would cause requests to hang indefinitely.

• Added automatic browser health detection with 5-second timeout on page creation
• Implemented connection error detection for Ws(AlreadyClosed) and related WebSocket errors
• Auto-recovery: dead browsers are now automatically recreated on connection failure
• Request-level retry logic (up to 3 retries) for transient connection errors
• Health endpoint now exposes healthy status and recreation_count for monitoring

Health response now includes:

{
  "status": "ok",
  "version": "0.1.4",
  "browser_pool": {
    "available": 10,
    "total": 10,
    "healthy": true,
    "recreation_count": 1
  }
}

Monitor recreation_count increasing to track browser recovery events.

⁠Links

• GitHub Repository⁠

⁠License

MIT