odevsa/monolai repository overview

⁠Monolai

Lightweight, private, self-hosted local AI model manager and OpenAI-compatible runtime built with Rust (Axum) and SvelteKit.

• Multi-Arch: linux/amd64, linux/arm64
• Features: GGUF model support, OpenAI-compatible API (/v1/chat/completions), hardware acceleration (CUDA, Vulkan, CPU), automated runtime management (llama.cpp), built-in web UI with real-time telemetry via SSE, SQLite storage.

⁠Quick Start (Docker Compose)

services:
  monolai:
    image: odevsa/monolai:latest
    container_name: monolai
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      - HOST=0.0.0.0
      - PORT=8080
      - MODELS_DIR=/app/models
      - RUNTIMES_DIR=/app/runtimes
      - DATA_DIR=/app/data
    volumes:
      - ./.monolai/models:/app/models
      - ./.monolai/runtimes:/app/runtimes
      - ./.monolai/data:/app/data

Run: docker compose up -d and open http://localhost:8080.

⁠Quick Start (Docker CLI)

docker run -d \
  --name monolai \
  -p 8080:8080 \
  -v ~/.local/share/monolai/models:/app/models \
  -v ~/.local/share/monolai/runtimes:/app/runtimes \
  -v ~/.local/share/monolai/data:/app/data \
  --restart unless-stopped \
  odevsa/monolai:latest

⁠Environment Variables

⁠Community App Stores

• Umbrel OS: Store URL https://github.com/odevsa/umbrel
• CasaOS: Custom install image odevsa/marknote:latest (Port 8080)

⁠Links

• GitHub: https://github.com/odevsa/monolai⁠