superbizon007/hil-agent repository overview

⁠Human-in-the-Loop Agent

An A2A agent that routes questions from automated agents to a human operator via messaging platforms. When an execution agent (browser, terminal, Android, etc.) encounters a situation it cannot resolve on its own — a CAPTCHA, a 2FA code, an ambiguous decision — it calls this agent and waits for a human reply.

Supports four notification channels (any combination):

• Telegram — bot polling, inline keyboard buttons
• Discord — bot polling, message component buttons
• Slack — webhook mode, interactive blocks
• WhatsApp — Meta Cloud API, reply buttons

⁠How it works

Execution agent                  hil_agent                  Human operator
──────────────                   ─────────                  ──────────────
ask_human {question, screenshot}
  → A2A message/stream ────────▶ parse message
                                 create Future
                                 notify all channels ──────▶ Telegram / Discord / Slack / WhatsApp
                                 poll (reminder every 10 min)
                                                   ◀──────── operator taps Done / Skip / reply
                                 Future resolved
  ◀─────────────────── completed (answer text)
resume graph with answer

1. Execution agent emits ask_human with an optional screenshot and a question.
2. hil_agent registers a pending asyncio.Future keyed by task ID, then broadcasts to every configured channel.
3. SSE working events are streamed back to the caller as reminders are sent.
4. The human responds via any channel button or free-text reply.
5. The first response wins. The Future resolves, the task completes, and the answer is returned to the caller.
6. If nobody responds within HIL_HARD_TIMEOUT (default 24 h), the task fails with TimeoutError.

The message format from execution agents is:

data:image/jpeg;base64,<base64_screenshot>\n\n<question text>

If no screenshot is attached, the message is plain text.

⁠Project structure

app/
  channels/
    base.py           Channel ABC, HumanRequest dataclass, OnResponseCallback type
    telegram.py       TelegramChannel — aiogram, long-polling
    discord.py        DiscordChannel  — discord.py, long-polling
    slack.py          SlackChannel    — slack-bolt async, webhook (POST /slack/events)
    whatsapp.py       WhatsAppChannel — httpx direct Meta Cloud API, webhook (GET/POST /webhook/whatsapp)
  dispatcher.py       Dispatcher — pending Future registry + reminder loop
  server/
    config.py         Settings (pydantic-settings, all channel credentials)
    a2a.py            HumanRelayExecutor + _parse_message + setup_a2a_routes()
    main.py           FastAPI app with lifespan (builds channels, starts dispatcher)
    __main__.py       Entry: python3 -m app.server
tests/
  test_dispatcher.py  Unit tests — Dispatcher, reminder loop, timeout logic
  test_a2a.py         Unit tests for _parse_message + integration + ASGI HITL flow tests

⁠Configuration

All settings are read from environment variables (or a .env file). At least one channel must be configured, otherwise the agent starts but cannot reach any human.

⁠Server

⁠Telegram

Create a bot via @BotFather⁠, add it to a group or use a direct chat, then get the chat ID with getUpdates.

⁠Discord

Create a bot at discord.com/developers⁠, grant it Send Messages + Read Message History permissions, invite it to your server, then copy the target channel ID.

⁠Slack

Create a Slack app with chat:write and chat:write.public bot token scopes. Enable Interactivity and set the request URL to https://<your-host>/slack/events.

⁠WhatsApp

Requires a Meta Business account with a WhatsApp Cloud API app. Set the webhook URL to https://<your-host>/webhook/whatsapp and subscribe to messages.

⁠Running

⁠Docker Compose

cd apps/hil_agent
cp .env.example .env
# Fill in at least one channel's credentials in .env
docker compose up

⁠Docker (standalone)

Build from this folder. The shared a2a_agent workspace package (living outside this directory at ../../packages/a2a_agent) is provided to the build via a named build context:

cd apps/hil_agent
docker build \
  --build-context a2a_pkg=../../packages/a2a_agent \
  -f Dockerfile \
  -t hil_agent \
  .

docker run -d \
  -p 8090:8090 \
  -e TELEGRAM_BOT_TOKEN=... \
  -e TELEGRAM_CHAT_ID=... \
  -e API_KEY=mysecret \
  hil_agent

⁠Local

# Install the a2a_agent workspace package first
pip install -e ../../packages/a2a_agent
pip install -e .

TELEGRAM_BOT_TOKEN=... TELEGRAM_CHAT_ID=... python3 -m app.server

⁠A2A interface

The agent exposes a standard Google A2A⁠ JSON-RPC endpoint at http://<host>:8090/.

Agent card: GET http://<host>:8090/.well-known/agent-card.json

Skill: ask_human

⁠Message format

Send a message/send or message/stream request. The message text can be:

Plain question (no screenshot):

What should I enter in the CAPTCHA field?

Question with screenshot (data URL prefix + double newline separator):

data:image/jpeg;base64,/9j/4AAQSkZJRgAB...

The page is asking for a CAPTCHA. What do I type?

⁠Example: message/stream request

{
  "jsonrpc": "2.0",
  "id": "req-1",
  "method": "message/stream",
  "params": {
    "message": {
      "role": "user",
      "messageId": "msg-1",
      "parts": [
        {"kind": "text", "text": "2FA code requested for account [email protected] — please check your phone"}
      ]
    }
  }
}

SSE events while waiting:

data: {"jsonrpc":"2.0","id":"req-1","result":{"kind":"status-update","taskId":"task-abc","status":{"state":"working","message":{"parts":[{"kind":"text","text":"Notification sent to all configured channels"}]}},"final":false}}

data: {"jsonrpc":"2.0","id":"req-1","result":{"kind":"status-update","taskId":"task-abc","status":{"state":"working","message":{"parts":[{"kind":"text","text":"Reminder #1 sent"}]}},"final":false}}

Final event (after human responds):

data: {"jsonrpc":"2.0","id":"req-1","result":{"kind":"status-update","taskId":"task-abc","status":{"state":"completed"},"final":true}}

⁠Using ask_human from an execution agent

All execution agents (chrome, terminal, android, etc.) support the ask_human action natively. Set two environment variables on the execution agent:

HUMAN_RELAY_AGENT_URL=http://hil-agent:8090/
HUMAN_RELAY_AGENT_API_KEY=mysecret   # only if API_KEY is set on hil_agent

The execution agent will automatically call hil_agent when it emits ask_human and pause the graph until the human responds.

⁠Notification format

Each channel posts a message with three reply options:

For Slack and WhatsApp, button IDs are shortened to 16–18 hex characters (platform limits) and resolved back to the full task ID internally.

⁠Reminder and timeout behaviour

⁠Tests

cd apps/hil_agent
pip install -e ".[test]"

# Unit tests only (no server needed)
pytest tests/test_dispatcher.py tests/test_a2a.py::TestParseMessage -v

# Unit tests + in-process ASGI HITL flow tests (no server needed, all deps required)
pytest -v

# Integration tests against a running server
AGENT_HOST=localhost AGENT_PORT=8090 pytest -v -m integration

⁠MCP tools

The agent image exposes an MCP (Model Context Protocol) server on the same port as A2A at /mcp. Any MCP client — Claude Desktop, Cursor, Windsurf, ChatGPT Connectors, OpenAI Agents SDK — can list and invoke these tools using the pod's API_KEY as Bearer.

In production, Core proxies https://hil-agent.agents.forfetch.ai/mcp → the worker pod's /mcp.

Input schemas are authored in apps/agents_mcp/app/seed.py. Skills without an explicit schema advertise a single {task: str} freeform parameter.