antonmilev979/js-alert-ai-mlflow repository overview

⁠js-alert-ai-mlflow

MLflow 3 tracking server providing LLM observability for the Joule AI Alert Monitoring system. It captures distributed traces of every AI agent run, records token consumption, and calculates per-call costs — giving operators a full audit trail of the AI pipeline.

This component is optional and disabled by default. The backend operates normally without it; enabling it adds observability without changing any pipeline behaviour.

⁠Role in the architecture

FastAPI backend (AI agent runs) ↓ LangChain autolog + UsageRecorder js-alert-ai-mlflow ← you are here [Trace store · token metrics · cost recording] PostgreSQL (trace metadata) + S3 (artifacts) ↓ MLflow UI (browser, admin-only deep-link)

⁠What it tracks

• Full agent execution traces — hierarchical spans for every LangChain tool call, chain invocation, and LLM API call across all pipeline workers (enrichment, summarisation, dependency analysis, Q&A).
• Token metrics — input tokens, output tokens, and total tokens per LLM call.
• Cost breakdown — monetised cost per call (input cost + output cost) attributed to the component, cluster, and model tier that generated it.
• Model metadata — model name, provider on every span.

⁠Tech stack

• Python 3.11 slim
• MLflow 3.15.1
• psycopg2-binary — PostgreSQL driver for the backend store
• boto3 — S3-compatible artifact storage