superbizon007/web-scraper-agent repository overview

⁠web_scraper_agent

Lightweight, always-stealth web-scraping agent. Driven by a text LLM in a ReAct loop; same tools also callable directly via A2A and MCP for deterministic single-tool usage.

⁠Architecture

                    ┌──────────────────────────────────────┐
                    │  HTTPS :443  (TLS, optional API_KEY) │
                    └────────────────────┬─────────────────┘
                                         │
                  ┌──────────────────────┼──────────────────────┐
                  │                      │                      │
            A2A direct             A2A freeform               MCP /mcp
            (skill_id JSON)        (NL task → ReAct)          (mount_mcp)
                  │                      │                      │
                  ▼                      ▼                      ▼
            ┌────────────────────────────────────────────────────────┐
            │  Tool registry (app/agent/tools.py)                    │
            │  fetch_url · render_page · fill_form · click_and_wait  │
            │  scroll_to · extract · download_file · switch_engine   │
            │  finish                                                │
            └────────┬─────────────────────────┬─────────────────────┘
                     │                         │
              curl_cffi path             Playwright path
              (chrome131 JA3)            (rebrowser-Chromium / FF)
                     │                         │
                     │                  ┌──────┴───────┐
                     │                  │ ghost-cursor │
                     │                  │ humanized KB │
                     │                  └──────┬───────┘
                     │                         │
                     └──── cookie sync ────────┘
                                  │
                              storage_state.{engine}.json
                              (S3 + /tmp fallback, debounced 10s)
                                  │
                       optional tun2socks → SOCKS proxy
                       optional Xvfb + x11vnc + noVNC (HEADED)

• TLS / HTTP fingerprint: curl_cffi⁠ with impersonate=chrome131 — defeats Cloudflare/PerimeterX/DataDome at the protocol layer.
• Browser engines: rebrowser-playwright⁠ (CDP-leak-patched Chromium, default), vanilla Playwright Firefox (fallback for Chrome-specific bot rules), and opt-in cloakbrowser⁠ (stealth Chromium fork with binary-level fingerprint patches for Cloudflare Turnstile / reCAPTCHA v3 / FingerprintJS-style gates). LLM picks per task via switch_engine. The registry of supported engines lives in app/browser/engines.py. CloakBrowser is gated by CLOAK_ENABLED; its binary downloads from CloakHQ at first launch (BINARY-LICENSE.md prohibits redistribution in a public Docker image). Camoufox is deferred (PyYAML CLoader incompatibility on Python 3.12).
• Behavioral: python-ghost-cursor⁠ Bezier mouse trajectories + inline humanized keyboard (typo simulation, Gaussian inter-key delays, RNG seeded per task_id).
• State persistence: storage_state (cookies + localStorage) is debounced-saved to S3 every 10s during a task; restored on next task with the same task_id. Local /tmp/state fallback.
• ReAct loop: LangGraph state machine — call_llm → execute_tool → loop. Default 25 steps max, 25k chars per HTML observation.
• Routing: optional SOCKS proxy via tun2socks (transparent, fwmark 100 + tun0). Set SOCKS_PROXY=....
• HEADED mode: HEADED=true boots Xvfb + x11vnc + noVNC on :6901 for live-watch debugging via the dashboard.

⁠Source layout

app/
├── server/
│   ├── config.py     pydantic-settings
│   ├── a2a.py        WebScraperAgentExecutor (direct dispatch + ReAct)
│   ├── main.py       FastAPI + lifespan + mount_mcp(...)
│   └── __main__.py   uvicorn entry
├── agent/
│   ├── prompts.py    SYSTEM_PROMPT for the ReAct loop
│   ├── tools.py      9 @tool functions + SCRAPER_TOOLS / TOOLS_BY_ID
│   └── graph.py      LangGraph StateGraph
├── browser/
│   ├── context.py    rebrowser-Chromium + vanilla-Firefox runtimes
│   ├── state.py      storage_state save/load (S3 + /tmp), debounce
│   ├── cursor.py     ghost-cursor wrapper
│   ├── typing.py     humanized keyboard
│   ├── cookies.py    curl_cffi ↔ Playwright cookie sync
│   └── detect.py     anti-bot phrase scanner
└── http/
    └── session.py    per-task curl_cffi AsyncSession pool

⁠Tools

⁠Surfaces

Same tools, three callable shapes:

1. A2A direct dispatch — POST {"skill_id": "fetch_url", "params": {...}} as the message text. Tool runs synchronously, no LLM cost.
2. A2A freeform — POST a natural-language task. The ReAct loop runs; the LLM picks tools.
3. MCP — mount_mcp(...) exposes every tool as an MCP tool at /mcp. Frontier-LLM clients (Claude Desktop, Cursor, Windsurf) use their own LLM and call tools deterministically.

⁠Build

The image is multi-arch (linux/amd64,linux/arm64) and ships from superbizon007/web-scraper-agent.

⁠Local build (dev)

Native arch only — required on Apple Silicon because QEMU emulation breaks Chromium's new_page under linux/amd64.

cd apps/web_scraper_agent
docker build \
  --build-context a2a_pkg=../../packages/a2a_agent \
  -f Dockerfile.ubuntu \
  -t superbizon007/web-scraper-agent:latest \
  .

Import into a local k3d cluster:

k3d image import superbizon007/web-scraper-agent:latest -c forfetch-local

⁠Multi-arch publish

cd apps/web_scraper_agent
./docker_hub_push.sh

Reads version from pyproject.toml, runs docker buildx build --platform linux/amd64,linux/arm64 --push against both :VERSION and :latest tags.

⁠Run

⁠Locally with Docker

docker run -d --rm \
  --name web-scraper \
  --shm-size=2g \
  -p 9443:443 \
  -e API_KEY=test123 \
  -e LLM_API_KEY=$OPENAI_API_KEY \
  superbizon007/web-scraper-agent:latest

Smoke test (skill_id direct dispatch):

curl -sk https://localhost:9443/ \
  -H "Authorization: Bearer test123" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":"1","method":"message/send","params":{
        "id":"task-1",
        "message":{"role":"user","messageId":"m1",
          "parts":[{"type":"text","text":"{\"skill_id\":\"fetch_url\",\"params\":{\"url\":\"https://example.com\"}}"}]
        }}}'

Freeform (ReAct loop) — same shape, but the message text is a NL task instead of a JSON skill_id envelope.

⁠HEADED mode (live-watch via VNC)

docker run -d --rm \
  --name web-scraper-headed \
  --shm-size=2g \
  -p 9443:443 -p 6901:6901 \
  -e HEADED=true \
  -e API_KEY=test123 \
  -e LLM_API_KEY=$OPENAI_API_KEY \
  superbizon007/web-scraper-agent:latest

Open http://localhost:6901/vnc.html for the noVNC client.

⁠With SOCKS routing

docker run -d --rm \
  --name web-scraper-proxied \
  --cap-add=NET_ADMIN --device=/dev/net/tun \
  --shm-size=2g \
  -p 9443:443 \
  -e SOCKS_PROXY=socks5://user:[email protected]:1080 \
  -e API_KEY=test123 \
  -e LLM_API_KEY=$OPENAI_API_KEY \
  superbizon007/web-scraper-agent:latest

⁠With CloakBrowser engine

The cloak engine is available by default — the ReAct LLM picks it via switch_engine when blocked_signals indicate a Turnstile / reCAPTCHA v3 / FingerprintJS gate. The stealth-Chromium binary is not baked into the image (its BINARY-LICENSE.md prohibits redistribution in a public Docker artifact); the entrypoint downloads it from CloakHQ on first boot (~200 MB, ~30 s on a fast link). Mount a volume at /cache/cloakbrowser so the download survives pod restarts:

docker run -d --rm \
  --name web-scraper-cloak \
  --shm-size=2g \
  -p 9443:443 \
  -e API_KEY=test123 \
  -e LLM_API_KEY=$OPENAI_API_KEY \
  -v cloakbrowser-cache:/cache/cloakbrowser \
  superbizon007/web-scraper-agent:latest

To force cloak as the default engine for every task: pass -e DEFAULT_ENGINE=cloak. To opt out entirely (no outbound traffic to CloakHQ at any point): pass -e CLOAK_ENABLED=false. Startup fail-fasts if DEFAULT_ENGINE=cloak and CLOAK_ENABLED=false.

NET_ADMIN + /dev/net/tun are required for tun2socks to set up the tun0 device and iptables mangle rules.

⁠In-cluster (k8s)

Core seeds and launches the agent automatically — see apps/agents_mcp/app/seed.py for the published agent definition. The agent registers as web_scraper_agent with capability tag {"stealth": true}.

⁠Environment

⁠LLM (text)

⁠Auth + transport

⁠Stealth knobs

⁠SOCKS proxy (entrypoint, not server)

⁠S3 (artifacts + storage_state)

⁠ReAct loop bounds

⁠HIL (human-in-the-loop, async)

⁠Trajectory + identity

⁠Fingerprint regression test

tests/test_fingerprint_check.py is a manual / nightly smoke that catches the day TLS impersonation or webdriver patches silently break. Six checks:

1. JA3 doesn't match Python's stdlib default; UA doesn't leak python-*.
2. Cloudflare basic mode (https://www.cloudflare.com/) doesn't return a "Just a moment" interstitial via fetch_url.
3. Sannysoft loads on Chromium (screenshot artifact = audit trail).
4. navigator.webdriver not flagged on Chromium.
5. Sannysoft loads on Firefox.
6. navigator.webdriver not flagged on Firefox.

Run against a live container:

docker run -d --rm --name web-scraper-smoke --shm-size=2g \
  -p 9443:443 -e API_KEY=test123 \
  superbizon007/web-scraper-agent:latest

AGENT_URL=https://localhost:9443 API_KEY=test123 \
  python3 tests/test_fingerprint_check.py
# exit 0 = green; exit 1 = at least one regression

Re-run after every curl_cffi, rebrowser-playwright, or base-image bump.

⁠Status

All five plan phases shipped:

See specs/PLAN_WEB_SCRAPER_AGENT.md for the full plan.

⁠MCP tools

The agent image exposes an MCP (Model Context Protocol) server on the same port as A2A at /mcp. Any MCP client — Claude Desktop, Cursor, Windsurf, ChatGPT Connectors, OpenAI Agents SDK — can list and invoke these tools using the pod's API_KEY as Bearer.

In production, Core proxies https://web-scraper-agent.agents.forfetch.ai/mcp → the worker pod's /mcp.

Input schemas are authored in apps/agents_mcp/app/seed.py. Skills without an explicit schema advertise a single {task: str} freeform parameter.