joshxt/agixt-ui repository overview

⁠AGiXT UI

The shared AGiXT v2 UI lives here. The same HTML/JS/CSS frontend is embedded by the native Tauri desktop app and served by the browser wrapper image.

The native app is written in Rust (Tauri 2). It runs as a tray-anchored sidebar popover with rich activity grouping, voice input, file attachments, and direct desktop control so the agent can drive the machine it's running on.

The chat layer talks two AGiXT protocols at once:

• /v1/chat/completions (SSE) for the active turn — streams text deltas, AGiXT-native remote_command.request tool calls, and activity.stream thinking deltas. Implementation in src-tauri/src/chat_stream.rs⁠.
• /v1/conversation/{id}/stream (WebSocket) for the persisted conversation log — same envelope the web client consumes, so [ACTIVITY] / [SUBACTIVITY] markers render as collapsible thinking/action groups. Implementation in src/chat.js⁠.

Markdown bodies render with first-class support for inline images, audio, and video. Workspace media URLs (/outputs/..., /workspace/...) get the user's JWT auto-appended as ?auth=<jwt> so they actually load through the backend's workspace file handler.

⁠Layout

ui/
├── README.md
├── Dockerfile                 # Browser wrapper for serving the shared UI
├── web-runtime.js             # Browser shim for the Tauri APIs used by src/
├── nginx.conf.template        # Proxies AGiXT API/WebSocket traffic in web mode
├── extensions/desktop/        # Bundled desktop extension UI manifests/scripts
├── src/                       # Frontend (HTML/JS/CSS, no build step)
│   ├── index.html             # Sidebar shell
│   ├── toggle.html            # Floating chat-bubble window
│   ├── styles.css
│   ├── app.js                 # Boot, settings, agent/convo switchers, composer
│   ├── auth.js                # OAuth / password / magic-link / register / paste-token
│   ├── chat.js                # WebSocket + SSE chat client + activity grouping
│   ├── markdown.js            # Markdown → HTML (with media + workspace JWT rewrite)
│   ├── audio.js               # TTS audio queue
│   ├── client-actions.js      # Browser-side dispatch into Rust IPC tools
│   ├── context-menu.js        # Custom right-click menu (no Inspect)
│   ├── frontend-log.js        # Forwards renderer console to Rust tracing
│   ├── dock.js                # Sidebar/popover dock-mode controller
│   └── assets/brands/         # Per-brand SVG logos
└── src-tauri/                 # Rust backend
    ├── Cargo.toml
    ├── tauri.conf.json
    ├── capabilities/default.json
    ├── icons/
    └── src/
        ├── main.rs
        ├── lib.rs                  # Tauri setup, IPC handlers (~58 commands)
        ├── config.rs               # SQLite-backed settings + service brands
        ├── chat_stream.rs          # SSE parser for /v1/chat/completions
        ├── api.rs                  # AGiXT REST helpers (auth, conversations, workspace)
        ├── automation.rs           # Screenshot / mouse / keyboard
        ├── filesystem.rs           # fs_read/write/append/edit/list/stat/...
        ├── terminal.rs             # PTY sessions + shell_run + sudo helpers
        ├── client_tool_specs.rs    # OpenAI-shaped tool definitions for /v1/chat/completions
        └── client_tools_prompt.rs  # Prompt fragment describing the tool surface

⁠Browser wrapper

The browser image serves the same src/ frontend with Nginx and injects web-runtime.js before the desktop scripts. The runtime provides the small window.__TAURI__ surface the frontend expects, backed by browser storage and AGiXT REST/SSE/WebSocket endpoints.

Build from the WorkConductor repository root:

docker build -f ui/Dockerfile -t agixt-client-web .

Run against an AGiXT API reachable from the container as http://agixt-api:7437:

docker run --rm -p 8080:80 agixt-client-web

For local development against the host backend:

docker run --rm -p 8080:80 \
  --add-host=host.docker.internal:host-gateway \
  -e AGIXT_API_UPSTREAM=http://host.docker.internal:7437 \
  agixt-client-web

Open http://localhost:8080. The browser talks to same-origin /v1/...; Nginx proxies that to AGIXT_API_UPSTREAM, including WebSocket streams.

Desktop-only capabilities such as screenshots, shell commands, native file paths, sudo, local installation, and app updates are disabled in web mode. Conversation chat, auth, agent selection, workspace upload/download, voice recording through the browser, and agent settings use the AGiXT API directly.

⁠Local state

Settings persist to a small SQLite DB:

Stored: server URL, web URL, JWT, service brand slug, selected agent + company, current conversation id + name, voice toggle, sidebar visibility, dock mode, dock position, "allow client commands" flag.

⁠Build prerequisites

⁠All platforms

• Rust 1.77+ (rustup default stable)
• cargo install tauri-cli@^2 (only needed if you want cargo tauri dev / cargo tauri build; otherwise plain cargo run / cargo build --release from src-tauri/ works).

⁠Linux (Ubuntu / Debian)

sudo apt-get install -y \
  libwebkit2gtk-4.1-dev \
  libsoup-3.0-dev \
  libjavascriptcoregtk-4.1-dev \
  librsvg2-dev \
  libxdo-dev \
  libgtk-3-dev \
  libayatana-appindicator3-dev

⁠macOS

Xcode command-line tools (xcode-select --install). Grant Screen Recording, Accessibility, and Microphone permissions to the built app the first time you use the desktop-control tools or the mic.

⁠Windows

WebView2 is bundled with modern Windows; no extra setup.

⁠Windows signing for release downloads

Windows release artifacts should be Authenticode signed before upload. The desktop-prewarm.yml workflow builds the Windows binary without bundling, signs agixt-desktop.exe, bundles the installer, signs the installer, and verifies the selected artifact with Get-AuthenticodeSignature when signing is available.

Use one signing path:

• Azure Artifact Signing: set repository variables AGIXT_WINDOWS_ARTIFACT_SIGNING_ENDPOINT, AGIXT_WINDOWS_ARTIFACT_SIGNING_ACCOUNT, and AGIXT_WINDOWS_ARTIFACT_SIGNING_CERTIFICATE_PROFILE; set Azure auth secrets AZURE_TENANT_ID, AZURE_CLIENT_ID, and AZURE_CLIENT_SECRET.
• PFX/Authenticode certificate: set secrets WINDOWS_CERTIFICATE_BASE64 and WINDOWS_CERTIFICATE_PASSWORD, or set WINDOWS_CERTIFICATE_THUMBPRINT when the cert is already installed on the Windows runner.

AGIXT_WINDOWS_SIGNING_REQUIRED defaults to false so Windows builds can continue while Azure identity validation or credentials are still pending. Set it to true after signing is configured to prevent unsigned Windows artifacts from being uploaded.

⁠Run

cd src-tauri
cargo run               # debug build

Release build:

cd src-tauri
cargo tauri build       # if tauri-cli is installed (recommended)
# or
cargo build --release   # produces target/release/agixt-desktop

When AGiXT is started with agixt start or agixt restart, the CLI checks for an installed AGiXT Desktop app on machines with a graphical desktop session. If it is missing, the CLI downloads the prebuilt app for the current OS from https://d.devxt.com/desktop/{os} and launches it instead of building locally. It skips CI/headless terminal-only environments and can be disabled with AGIXT_DESKTOP_INSTALL=false. Set AGIXT_DESKTOP_DOWNLOAD_BASE_URL to point at a different download server.

Set RUST_LOG=info,agixt_desktop_lib=debug for verbose logging. Frontend console.log is forwarded to the same Rust tracing log via the frontend_log IPC, so you only need to watch one stream.

⁠Development note: rebuild for frontend changes

tauri.conf.json sets frontendDist: "../src" and devUrl: null, which means Tauri embeds the HTML/JS/CSS into the Rust binary at build time. Editing src/*.js or src/styles.css and just re-launching the binary will not pick up the changes — cargo build first to re-embed, then relaunch. The sidebar logs chat.js loaded (<version-tag>) on every webview load so you can confirm the new bundle is live.

The webkit2gtk webview also caches at ~/.local/share/systems.xt.agixt.desktop/{WebKitCache,CacheStorage}; clear those if a stale CSS/JS asset persists past a rebuild.

⁠First run / sign-in

The auth screen offers four ways to sign in, plus account creation. No copy-pasting JWTs unless you specifically want to.

1. Pick a service brand. The dropdown lists AGiXT.com, NurseXT.com, XT.Systems, BoltRemote.com, Local, and Custom. Picking one fills the API + Web URL fields with that brand's defaults; the topbar logo swaps to the brand's mark; the brand persists across restarts (OAuth users included — the brand is written before the browser opens).
   • Local pins to http://localhost:7437, probes /health and /openapi.json, and shows a connect state if AGiXT is already up. If nothing is running, the login screen can install locally: git clone/git pull Josh-XT/AGiXT, pip install -e ., pip install ezlocalai, pre-configure .env, then run python -m agixt.cli restart.
   • The local installer recommends ezLocalai defaults from hardware: <12GB VRAM → unsloth/Qwen3.5-4B-GGUF, 13GB+ below the 24GB high tier → unsloth/Qwen3.5-9B-GGUF, 24GB+ VRAM → unsloth/Qwen3.6-35B-A3B-GGUF. CPU-only systems use the 4B model; constrained RAM gets a smaller max-token default and a performance warning. First setup can take 30+ minutes because Docker images and models may download.
   • Custom keeps the API + Web URL fields editable for any other self-hosted AGiXT.
2. Sign in any of:
   • Email + password (with optional MFA prompt)
   • OAuth — buttons for whatever providers AGiXT exposes for the selected brand. Click → system browser opens → after consent the server redirects to agixt://login?token=<jwt> which the desktop deep-link handler validates and unlocks the chat.
   • Magic link — type your email, click the button, click the link in your inbox. The link's ?jwt= redirects through the same agixt:// deep link.
   • Paste a JWT or magic-link URL — fallback for bespoke flows.
3. Register a new account from the second tab (first/last name, email, password meeting AGiXT's complexity rules).

After auth: the sidebar refreshes companies + agents, picks the user's default agent, opens (or restores) a conversation, opens the /v1/conversation/{id}/stream WebSocket, and is ready to chat.

A global shortcut Ctrl+Shift+Space (or Cmd+Shift+Space on macOS) toggles the sidebar in/out. The tray icon does the same.

⁠The chat surface

• Topbar — brand logo on the left (with a small connection-state dot overlay when disconnected/error), then a stacked chip pair: the agent switcher (Agent @ Company) on top with settings + close buttons, and the conversation switcher (with chat-bubble icon) below it with the new-conversation button. The conversation chip is fully searchable via a small search field at the top of its dropdown.
• Composer — paperclip (file attachment), text area with a dynamic Ask {agent_name}… placeholder, mic, and send/stop. The input area is right-clickable for cut/copy/paste/select-all.
• Stop button swaps in for send while the agent is generating. Click stops both sides: detaches the SSE listener locally so further events are dropped, halts the local tool-execution recursion via a sticky turnStopped flag, and POSTs to /v1/conversation/{id}/stop (with /v1/conversations/stop as a belt-and-suspenders fallback) — same flow the web client uses.

⁠Activity rendering

Mirrors the web's useConversationWebSocketStable.ts groupMessages():

• [SUBACTIVITY] markers attach to the parent [ACTIVITY] (by UUID if referenced, else the most recent activity). Orphans get a synthetic Thinking parent.
• Consecutive Thinking activities merge into one block instead of opening a new one for each.
• Each tool call (CLIENT_TOOL + its REMOTE / untagged follow-ups — request queued, completed, received result, uploaded, processed) collapses into one ▸ Called <tool> disclosure. Click to expand for the request payload and full transport detail.
• Once the assistant's text reply lands, the activity block is relabeled Activities, its spinner stops, and it auto-collapses. Still openable on click.
• Live thinking from activity.stream events accumulates as a rolling cumulative subactivity inside the open Thinking block (each event is a token-level delta, not a replacement — see AGiXT/agixt/Interactions.py:4480).
• Workspace image URLs auto-rewrite to ?auth=<jwt> regardless of host, so AGiXT-served screenshots and uploads render inline.

⁠Right-click

Custom menu (the default webkit2gtk Inspect menu is suppressed):

• On a message bubble: Copy message, Copy selection, and Copy link address when right-clicking a link.
• In the composer: Cut, Copy, Paste, Select all.
• Anywhere else: nothing — just dismisses any open menu.

⁠Hover timestamps

Each message has a small timestamp pinned to its bubble that fades in on hover; the tooltip shows the full date down to milliseconds so it's easy to gauge response latency.

⁠Voice input (mic button)

Tap to record, tap again to stop. The captured audio (audio/webm;codecs=opus if the platform supports it, else next best) is POSTed as multipart to ${server}/v1/audio/transcriptions with model={agent_name} — the OpenAI-compatible endpoint AGiXT exposes — and the returned text is fed through the normal compose path. Esc cancels an in-progress recording without sending.

⁠File attachments (paperclip)

The paperclip opens an OS file picker (tauri-plugin-dialog). Selected paths show as removable chips above the textarea. They aren't uploaded: on send, a context block is prepended to the prompt:

…signaling the agent to reach for fs_read, shell_run, workspace_upload, etc. through the IPC tool surface.

⁠Client-side desktop control

When the model emits a tool call (either OpenAI-shaped via /v1/chat/completions, or AGiXT-native as remote_command.request / [SUBACTIVITY][CLIENT_TOOL]), the client dispatches it to a Rust IPC handler. After execution the result is posted back via POST /v1/conversation/{id}/remote-command-result (mirroring the kids app pattern) and the next chat-completion round receives the corresponding role:"tool" message.

Disable client-side commands at any time from the settings modal — every IPC handler checks allow_client_commands before doing anything.

⁠Vision / direct input

⁠Shell

shell_run is the one-shot equivalent of terminal_exec for a fresh shell, plus an optional sudo ladder for password-gated commands.

⁠Background terminal sessions

Each terminal is a portable-pty shell with its own dedicated reader thread filling a 1 MiB ring buffer, so the agent can run multiple long-lived shells concurrently and poll their output incrementally using the monotonic next_offset returned by terminal_exec / terminal_read. PTY size is adjustable on the fly via terminal_resize.

⁠Local filesystem

Paths support ~, $VAR, and %VAR% expansion so the agent can use the same paths a human would type. Writes are atomic (write-tempfile- then-rename in the same directory) and fs_edit mirrors the AGiXT Edit tool semantics (unique-substring find/replace; require replace_all for ambiguous matches).

⁠Workspace bridge — local disk ↔ AGiXT cloud workspace

Files in the AGiXT conversation workspace travel with the conversation across devices. Use the bridge to pull a file off the user's disk, work on it in the agent's sandbox, then push it back; or vice versa.

⁠Notes

• Cross-platform: every client tool is implemented in pure Rust against cross-platform crates (enigo, screenshots, portable-pty, std::fs) so the same binary builds and runs on Linux, macOS, and Windows. The Tauri 2 webview shell, the SQLite settings DB, and the AGiXT REST/SSE/WS clients all behave identically across the three.
• Cloud-resident state: JWT, agent/company selection, conversation history, persisted activities, and any files the agent stages all live on the AGiXT server. The local SQLite DB only caches the JWT, brand, and last-selected ids so re-auth isn't needed on relaunch. Sign in on a different device and you pick up where you left off.
• Vision math parity: the normalized / target_width / screen_width / monitor_offset_* math in automation::resolve_coords is bit-for-bit identical to rust_endpoint_agent/src/terminal.rs. A model trained against one works against the other.
• Atomic writes: fs_write and fs_edit write to a temp file in the destination directory and rename — no partial writes on crash.
• WebSocket keepalive: the chat WS sends a ping envelope every 30s to survive Cloudflare; reconnect with exponential backoff up to 30s.
• Voice replies: when the assistant message contains an audio URL AGiXT emitted, audio.js plays them back-to-back from a single <audio> element. Browser autoplay policies are largely a non-issue inside the Tauri webview because the user clicked something to start the session.
• Linux input: cross-platform input automation comes through enigo 0.2 + a local screenshots 0.8 patch - the same API rust_endpoint_agent uses. On X11 a brief 2 ms warp delay is inserted before each click. Wayland portal fallbacks aren't ported because this client always runs in the user's session, never as a system service.