tamanois/web-fetcher repository overview

⁠web-fetcher

A production-ready web page fetching microservice powered by Crawl4AI⁠ and Playwright. Fetches any webpage and returns cleaned Markdown or HTML — including links and images — with anti-bot detection bypass.

⁠Quick Start

docker run -p 8000:8000 tamanois/web-fetcher:latest

Then:

curl "http://localhost:8000/fetch?url=https://example.com"

⁠API Reference

⁠GET /fetch

Fetches a web page and returns its content.

⁠GET /health

Lightweight health check — does not touch the browser crawler.

{
  "status": "ok",
  "crawler_age_seconds": 142.3,
  "crawler_max_age_seconds": 900
}

⁠Configuration

All settings are controlled via environment variables:

⁠Docker Compose (recommended)

services:
  web_fetcher:
    image: tamanois/web-fetcher:latest
    ports:
      - "8000:8000"
    restart: unless-stopped
    environment:
      - WEB_FETCHER_MAX_CONCURRENCY=10
      - WEB_FETCHER_REQUEST_TIMEOUT=30
      - WEB_FETCHER_CRAWLER_MAX_AGE=900
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 15s
    deploy:
      resources:
        limits:
          memory: 2G
        reservations:
          memory: 512M

  autoheal:
    image: willfarrell/autoheal
    restart: unless-stopped
    environment:
      - AUTOHEAL_CONTAINER_LABEL=all
      - AUTOHEAL_START_PERIOD=60
      - AUTOHEAL_INTERVAL=5
      - AUTOHEAL_DEFAULT_STOP_TIMEOUT=30
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

⁠Resilience Features

• Request timeouts — hung browser requests are cancelled after 30s, releasing resources
• Crawler recycling — the browser instance is automatically recreated at a configurable interval to prevent memory leaks
• Concurrency limiting — semaphore-based gate prevents resource exhaustion under heavy load
• Health check — lightweight endpoint that bypasses the crawler; pair with autoheal to auto-restart on failure
• Anti-bot bypass — uses randomized user agents and stealth mode via Crawl4AI's UndetectedAdapter

⁠Resource Requirements

• Memory: Minimum 512 MB, recommended 2 GB (Chromium-based crawling)
• Disk: ~1 GB (includes Playwright browsers)
• Startup time: 15–60s on first run (Playwright browser download)

⁠Tech Stack

• FastAPI⁠ — async web framework
• Crawl4AI⁠ — LLM-friendly web crawling
• Playwright⁠ — headless browser automation
• Starlette⁠ — ASGI middleware