chiefj/alert-engine repository overview

.
├── build/
│   └── Dockerfile
├── contracts/
│   └── events/
│       ├── alerts.budget.yaml
│       └── alerts.cost.anomaly.yaml
├── database/
│   └── migrations/
│       ├── 0001_init.rollback.sql
│       ├── 0001_init.sql
│       └── schema.lock
├── deploy/
│   └── docker/
│       └── docker-compose.yaml
├── feature-registry.yaml
├── pyproject.toml
├── README.md
├── scripts/
│   ├── migrate.py
│   ├── migrate.sh
│   ├── run.sh
│   └── test.sh
├── src/
│   ├── handlers/
│   │   ├── alerts_budget/
│   │   │   └── handler.py
│   │   └── alerts_cost_anomaly/
│   │       └── handler.py
│   ├── infra/
│   │   └── adapters/
│   │       ├── metrics/
│   │       │   └── prometheus_adapter.py
│   │       ├── pagerduty/
│   │       │   └── pagerduty_adapter.py
│   │       ├── postgres/
│   │       │   └── postgres_adapter.py
│   │       ├── redis/
│   │       │   └── redis_adapter.py
│   │       └── slack/
│   │           └── slack_adapter.py
│   ├── shared/
│   │   └── ports/
│   │       ├── db_port.py
│   │       ├── metrics_port.py
│   │       ├── pagerduty_port.py
│   │       ├── redis_port.py
│   │       └── slack_port.py
│   └── worker/
│       ├── config.py
│       ├── index.py
│       └── registry.py
├── tests/
│   └── unit/
│       ├── test_budget_handler.py
│       ├── test_cost_anomaly_handler.py
│       ├── test_failures.py
│       └── test_tracing.py
├── worker-registry.yaml
└── service_owners.yaml

[alerts.budget Event Consumed]
└── Rate Limit Check (Redis: rate_limit:budget_alert:{user_id}:{model})
    │
    │   ► RATIONALE: Dedups repeat alerts within a 15-minute (900s) window.
    │     If the rate limit key exists, processing is suppressed to prevent
    │     alert fatigue.
    │
    ├── Key Exists (Suppressed)
    │   └── Skip processing
    │
    └── Key Acquired (Proceed)
        ├── 1. Parse Delivery Latency (timestamp_utc vs now)
        ├── 2. Write to PostgreSQL (alert_history table)
        ├── 3. Emit Prometheus histogram latency metric
        └── 4. Notification Dispatch:
            ├── event_type = 'blocked'
            │   └── Slack message to #llm-cost-alerts channel
            └── event_type = 'warning_80pct'
                └── Lookup owner in service_owners.yaml & send Slack DM

[alerts.cost.anomaly Event Consumed]
└── Check is_cold_start (payload is_cold_start OR sample_count < 7)
    │
    ├── is_cold_start = TRUE
    │   ├── 1. Write to PostgreSQL (alert_history table)
    │   └── 2. Log message only (Do NOT notify Slack/PagerDuty)
    │
    └── is_cold_start = FALSE
        └── Check Burn Ratio (current_cost / ewma_value)
            │
            ├── burn_ratio <= 3.0 (Normal Deviation)
            │   └── Skip notification
            │
            └── burn_ratio > 3.0 (Cost Spike detected)
                └── Rate Limit Check (Redis: rate_limit:cost_anomaly:{service}:{model})
                    │
                    │   ► RATIONALE: Suppresses repeat anomaly alerts within a
                    │     1-hour (3600s) window.
                    │
                    ├── Key Exists (Suppressed)
                    │   └── Skip notifications
                    │
                    └── Key Acquired (Proceed)
                        ├── 1. Write to PostgreSQL (alert_history table)
                        ├── 2. Emit Prometheus latency metrics (critical severity)
                        ├── 3. Send PagerDuty incident alert
                        └── 4. Send Slack alert with drill-down (top contributor cluster)

[Step 1: Core Infra] --------> [Step 2: Database Migration] ----> [Step 3: Verification] ----> [Step 4: Run Worker]
  • Postgres (5432)              • ./scripts/migrate.sh           • ./scripts/test.sh         • ./scripts/run.sh
  • Redis (6379)                   (Initializes DB schema           (Runs unit test             (Starts Kafka poll
  • Kafka (9092)                    and creates indexes)             assertions)                 consumer loop)

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

./scripts/migrate.sh

./scripts/test.sh

./scripts/run.sh

docker build -f build/Dockerfile -t chiefj/alert-engine:latest .

docker compose -f deploy/docker/docker-compose.yaml up -d

docker run -d \
  --name alert-engine \
  -e KAFKA_BOOTSTRAP_SERVERS="kafka-host:9092" \
  -e REDIS_URL="redis://redis-host:6379/0" \
  -e POSTGRES_DSN="postgresql://postgres:postgres@db-host:5432/alert_db" \
  -e SLACK_WEBHOOK_URL="https://hooks.slack.com/services/T00/B00/X00" \
  -e PAGERDUTY_ROUTING_KEY="pagerduty-key" \
  -e OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4317" \
  -p 8001:8001 \
  -p 9464:9464 \
  chiefj/alert-engine:latest

{
  "user_id": "user_12345",
  "model": "gpt-4-turbo",
  "event_type": "warning_80pct",
  "timestamp_utc": "2026-05-28T10:45:00Z"
}

{
  "service": "recommendation-service",
  "model": "claude-3-opus",
  "hour_of_week": 42,
  "current_cost": 150.50,
  "ewma_value": 30.10,
  "threshold_value": 90.30,
  "sample_count": 14,
  "timestamp": "2026-05-28T10:45:00Z",
  "is_cold_start": false,
  "cluster_drilldown": [
    {
      "cluster_id": "k8s-us-east-1",
      "cost": 120.40
    }
  ]
}

service_owners:
  recommendation-service: "@john-doe"
  payment-processor: "@jane-smith"
  default: "@on-call-infra"

from worker.config import load_config

# Automatically loads and validates config from environment variables
config = load_config()
print(f"Connecting to Kafka at {config.kafka_bootstrap_servers}")

from src.infra.adapters.postgres.postgres_adapter import PostgresAdapter
from src.infra.adapters.redis.redis_adapter import RedisAdapter
from src.infra.adapters.slack.slack_adapter import SlackAdapter
from src.infra.adapters.metrics.prometheus_adapter import PrometheusAdapter
from src.handlers.alerts_budget.handler import BudgetAlertHandler

# Initialize the adapter ports
db_port = PostgresAdapter("postgresql://postgres:postgres@localhost:5432/alert_db")
redis_port = RedisAdapter("redis://localhost:6379/0")
slack_port = SlackAdapter("https://hooks.slack.com/services/T00/B00/X00")
metrics_port = PrometheusAdapter()

# Initialize the handler
budget_handler = BudgetAlertHandler(
    db_port=db_port,
    redis_port=redis_port,
    slack_port=slack_port,
    metrics_port=metrics_port,
    service_owners_path="service_owners.yaml"
)

# Process budget alert payload manually
budget_handler.handle({
    "user_id": "user_12345",
    "model": "gpt-4",
    "event_type": "blocked",
    "service": "recommendation-service",
    "timestamp_utc": "2026-05-28T10:45:00Z"
})