superbizon007/android-agent repository overview

⁠Android Agent

A Docker image that runs a VLM-powered Android device automation agent, exposed via A2A and OpenAI-compatible APIs. Live device view streams to the browser through an on-pod scrcpy bridge: the FastAPI process spawns scrcpy-server.jar on the device over ADB, pipes the H.264 elementary stream to a WebSocket client, and renders it with the browser's native WebCodecs VideoDecoder — no on-device APK, no MediaProjection consent, no WebRTC.

Implements an agent using the ReAct (Reasoning + Acting) pattern for intelligent Android device automation via ADB.

⁠Overview

The container bundles:

• ADB (Android Debug Bridge) — connects to and controls the Android device
• scrcpy bridge — app/server/scrcpy_bridge.py pushes scrcpy-server-v2.4.jar (baked at /opt/scrcpy-server.jar) to /data/local/tmp, runs it via app_process, opens a TCP tunnel through adb reverse, and forwards the H.264 packets to the browser. Browser-side input events come back over the same WebSocket and dispatch through the shared ADBClient (same code path as the VLM tools) — see PLAN_SCRCPY_STREAMING.md
• WebSocket /stream + static /stream.html — JWT-authed WebSocket endpoint for the bridge plus a self-contained viewer page using the browser's WebCodecs API
• Python agent server — FastAPI service that accepts tasks and autonomously controls the Android device using a Vision Language Model (VLM) via LangGraph

The agent observes the device through screenshots, reasons about the next action, and executes it — repeating until the task is complete or the step budget is exhausted. The streaming path is independent of the VLM control loop; it adds a real-time human-watchable view + a click-to-tap path on top.

⁠Features

⁠Architecture

                                              ┌──────────────────────────┐
Android device (USB or network ADB)           │ User's browser (Chrome)  │
┌──────────────────────────────┐               │   stream.html?token=…    │
│ scrcpy-server-v2.4.jar       │   H.264       │  • WebCodecs VideoDecoder│
│  (pushed via adb push +      │   Annex-B     │    → <canvas>            │
│   spawned via app_process)   │◄──────────────┤  • pointer/key events    │
└──────────────┬───────────────┘  TCP via      │    → JSON over same WS   │
               │                  adb reverse  └──────────┬───────────────┘
               │                                          │
               │                                          │  WS /stream
               │                                          │  ?token=<viewer_jwt>
               ▼                                          ▼
        ┌──────────────────────────────────────────────────────┐
        │ Docker container                                     │
        │                                                      │
        │  ADB ─► ADBClient ─► LangGraph (graph.py)            │
        │                       screenshot → VLM →             │
        │                       execute_action (tap/swipe/…)   │
        │                                                      │
        │  FastAPI on :8080 plain (LOCAL_TLS=false, prod)      │
        │       or :443 self-signed (LOCAL_TLS=true, dev)      │
        │    ├── A2A      (POST /)                             │
        │    ├── OpenAI   (/v1/…)                              │
        │    ├── /stream  (JWT-authed WS — scrcpy bridge)      │
        │    └── /stream.html (static WebCodecs viewer)        │
        └──────────────────────────────────────────────────────┘

The bridge spawns a fresh scrcpy server per WebSocket connection and binds an ephemeral local TCP port, so multiple viewer sessions on one pod don't collide.

⁠Quick Start

⁠Quickest path

cd apps/android_agent
./scripts/run-local.sh ADB_SERIAL [VLM_API_KEY]

Builds the image, mints the device + viewer JWT pair, runs the container with LOCAL_TLS=false, and prints the viewer URL: http://localhost:8765/stream.html?token=<viewer_jwt>. Open it in Chrome — no consent dialog, no APK install, video starts immediately.

ADB_SERIAL accepts any of:

• 7a4e6acb — USB device (the Mac's adb daemon is proxied into the container)
• 192.168.1.10:5555 — network ADB
• emulator-5554 — local emulator

For USB / emulator serials, run this once on the host before starting the container so the container's adb can reach your daemon:

adb kill-server && adb -a -P 5037 nodaemon server &

Optional env (read by the script before invoking docker):

Example:

VLM_URL=https://my-vlm.example.com/v1 VLM_MODEL=gpt-4o-mini \
TRAJ_HOST_DIR=$PWD/agent-logs \
./scripts/run-local.sh 192.168.1.10:5555 sk-...

⁠Prerequisites

The Android device must have ADB debugging enabled:

• USB: connect directly and accept the RSA key prompt on the device
• Network: enable Wireless debugging in Developer Options (Android 11+), then connect via adb connect HOST:PORT
• Emulator / cloud Android: any image that boots scrcpy-compatible video encoders (most do; emulated CPU images may lack hardware H.264)

The on-device side is just scrcpy-server-v2.4.jar running under app_process — no APK install, no system-app signing, no Device Owner enrollment.

⁠Build

cd apps/android_agent
docker build \
  --build-context a2a_pkg=../../packages/a2a_agent \
  -f Dockerfile.ubuntu \
  -t android_agent:ubuntu .

The image bakes vendor/scrcpy-server-v2.4.jar to /opt/scrcpy-server.jar. Single-stage build — no Android SDK needed at build time.

⁠Run — network ADB

docker run -d \
  --name android_agent \
  -p 8765:8080 \
  -e ADB_SERIAL=192.168.1.10:5555 \
  -e VLM_URL=https://api.openai.com/v1 \
  -e VLM_API_KEY=sk-... \
  -e VLM_MODEL=gpt-4o \
  -e LOCAL_TLS=false \
  -e STREAM_HMAC_SECRET=$(openssl rand -hex 32) \
  -e RUN_ID=run-local-1 \
  -e POD_PUBLIC_HOST=localhost:8765 \
  android_agent:ubuntu --wait

⁠Run — host ADB server (USB device or emulator)

When the device is connected on the host (adb devices shows it), forward the host ADB server into the container instead of using network ADB.

Step 1 — restart host ADB server to listen on all interfaces:

adb kill-server
adb -a -P 5037 nodaemon server &
adb devices   # verify your device is still listed

Step 2 — run the container (macOS / Windows Docker Desktop):

docker run -d \
  --name android_agent \
  -p 8765:8080 \
  -e ADB_SERIAL=emulator-5554 \
  -e ANDROID_ADB_SERVER_ADDRESS=host.docker.internal \
  -e ANDROID_ADB_SERVER_PORT=5037 \
  -e VLM_API_KEY=sk-... \
  -e LOCAL_TLS=false \
  -e STREAM_HMAC_SECRET=$(openssl rand -hex 32) \
  -e RUN_ID=run-local-1 \
  -e POD_PUBLIC_HOST=localhost:8765 \
  android_agent:ubuntu --wait

Linux — use --network host instead:

docker run -d \
  --network host \
  -e ADB_SERIAL=emulator-5554 \
  -e VLM_API_KEY=sk-... \
  -e LOCAL_TLS=false \
  -e STREAM_HMAC_SECRET=$(openssl rand -hex 32) \
  -e RUN_ID=run-local-1 \
  -e POD_PUBLIC_HOST=localhost:8080 \
  android_agent:ubuntu --wait

ANDROID_ADB_SERVER_ADDRESS and ANDROID_ADB_SERVER_PORT are standard ADB client environment variables. The adb binary inside the container reads them automatically and connects to the host's ADB server, which proxies all communication to the device.

⁠Watching the device

Mint a viewer-role JWT (same STREAM_HMAC_SECRET, same RUN_ID, role:"viewer") and open http://localhost:8765/stream.html?token=<viewer_jwt> in Chrome.

A minimal token-minting helper (Python ≥ 3.8, no deps):

python3 - <<'PY'
import jwt, time, os
print(jwt.encode(
    {"run_id": "run-local-1", "role": "viewer", "ice": [],
     "iat": int(time.time()), "exp": int(time.time()) + 14400},
    os.environ["STREAM_HMAC_SECRET"], algorithm="HS256",
))
PY

The bridge validates token signature, role, run_id, and expiry on connect. Close codes: 4401 (auth), 4503 (streaming not configured on this pod). See tests/test_stream.py and tests/test_stream_auth.py for boundary coverage.

⁠Dev image (VNC only, no API server)

For testing ADB connectivity manually with a noVNC viewer (no agent, no scrcpy bridge):

docker build \
  --build-context a2a_pkg=../../packages/a2a_agent \
  -f Dockerfile.dev \
  -t android_agent:dev .

docker run -d \
  --name android_agent_dev \
  -p 5901:5901 \
  -p 6901:6901 \
  -e ADB_SERIAL=192.168.1.10:5555 \
  android_agent:dev --wait

Open http://localhost:6901/?password=vncpassword.

⁠Trust the certificate (one-time, LOCAL_TLS=true only)

When LOCAL_TLS=false, the pod serves plain HTTP and TLS terminates at the per-pod Ingress with a real (Let's Encrypt or wildcard) cert — no manual trust step needed. The block below applies only to local dev.

macOS:

curl -k https://localhost/ca.crt -o android-agent-ca.crt
sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain android-agent-ca.crt

Linux (Debian/Ubuntu):

curl -k https://localhost/ca.crt -o /usr/local/share/ca-certificates/android-agent-ca.crt
sudo update-ca-certificates

Windows:

curl -k https://localhost/ca.crt -o android-agent-ca.crt
certutil -addstore Root android-agent-ca.crt

⁠API

⁠A2A — send a task

curl -sk -X POST https://localhost/ \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": "1",
    "method": "message/send",
    "params": {
      "message": {
        "role": "user",
        "messageId": "m1",
        "parts": [{"kind": "text", "text": "Open Settings and show me the Android version"}]
      }
    }
  }'

⁠OpenAI-compatible

curl -sk -X POST https://localhost/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "android-agent",
    "messages": [{"role": "user", "content": "Open the YouTube app and search for cats"}]
  }'

⁠Agent card

curl -sk https://localhost/.well-known/agent-card.json

⁠Streaming (A2A)

curl -sk -N -X POST https://localhost/ \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": "1",
    "method": "message/stream",
    "params": {
      "message": {
        "role": "user",
        "messageId": "m1",
        "parts": [{"kind": "text", "text": "Take a screenshot and describe what you see"}]
      }
    }
  }'

⁠Environment Variables

⁠Core

⁠UI Locator model (optional)

The locator is a secondary vision model (e.g. UI-TARS⁠) specialised in resolving natural-language element descriptions to exact pixel coordinates. When configured, the agent describes elements by name (query) instead of estimating coordinates, and the locator finds the exact tap target.

⁠Shared workspace (optional)

Used when the agent runs as part of a multi-agent pipeline. Agents read/write shared artifacts and post their status to a blackboard visible to the orchestrator.

⁠Agent identity (optional)

⁠Streaming

These wire the JWT-authed scrcpy bridge on /stream and the static viewer page on /stream.html. See PLAN_SCRCPY_STREAMING.md for the full design.

⁠Agent Tools

⁠Trajectory Recording

When TRAJECTORY_DIR is set, each agent run creates a timestamped subdirectory:

$TRAJECTORY_DIR/
  20260317_141523/              # run timestamp (YYYYMMDD_HHMMSS)
    summary.md                  # task, result, and final plan summary
    turn_0000/
      screenshot.png            # device state at start of step
      llm_0_request.json        # messages sent to the VLM
      llm_0_response.json       # raw VLM reply + parsed action/args/reasoning
      locator.json              # locator model request + response (when LOCATOR_URL is set)
    turn_0001/
      ...

Example:

docker run -d \
  -e VLM_URL=... -e VLM_API_KEY=... \
  -e ADB_SERIAL=192.168.1.10:5555 \
  -e TRAJECTORY_DIR=/data/trajectory \
  -v /tmp/agent-logs:/data/trajectory \
  android_agent:ubuntu --wait

⁠License

MIT

⁠MCP tools

The agent image exposes an MCP (Model Context Protocol) server on the same port as A2A at /mcp. Any MCP client — Claude Desktop, Cursor, Windsurf, ChatGPT Connectors, OpenAI Agents SDK — can list and invoke these tools using the pod's API_KEY as Bearer.

In production, Core proxies https://android-agent.agents.forfetch.ai/mcp → the worker pod's /mcp.

Input schemas are authored in apps/agents_mcp/app/seed.py. Skills without an explicit schema advertise a single {task: str} freeform parameter.