joshxt/ezlocalai repository overview

⁠ezlocalai

ezlocalai is an easy set up artificial intelligence server that allows you to easily run multimodal artificial intelligence from your computer. It is designed to be as easy as possible to get started with running local models. It automatically handles downloading the model of your choice and configuring the server based on your CPU, RAM, and GPU specifications. It also includes OpenAI Style⁠ endpoints for easy integration with other applications using ezlocalai as an OpenAI API proxy with any model. Additional functionality is built in for voice cloning text to speech and a voice to text for easy voice communication as well as image generation and video generation entirely offline after the initial setup.

⁠Prerequisites

• Python 3.10+⁠
• Docker Desktop⁠ (Windows or Mac)
• CUDA Toolkit (May Need 12.4)⁠ (NVIDIA GPU only)
• ROCm⁠ (AMD GPU only - Linux)

⁠Quick Start (Recommended)

Install the CLI and start ezlocalai with a single command:

pip install ezlocalai
ezlocalai start

It will take several minutes to download the models on the first run. Once running, access the API at http://localhost:8091⁠.

⁠CLI Commands

# Start with defaults (auto-detects GPU, uses Qwen3-VL-4B)
ezlocalai start

# Start with a specific model
ezlocalai start --model unsloth/gemma-3-4b-it-GGUF

# Start with custom options
ezlocalai start --model unsloth/Qwen3-VL-4B-Instruct-GGUF \
                --uri http://localhost:8091 \
                --api-key my-secret-key \
                --ngrok <your-ngrok-token>

# Other commands
ezlocalai stop      # Stop the container
ezlocalai restart   # Restart the container
ezlocalai status    # Check if running and show configuration
ezlocalai logs      # Show container logs (use -f to follow)
ezlocalai update    # Pull/rebuild latest images

# Send prompts directly from the CLI
ezlocalai prompt "Hello, world!"
ezlocalai prompt "What's in this image?" -image ./photo.jpg
ezlocalai prompt "Explain quantum computing" -m unsloth/Qwen3-VL-4B-Instruct-GGUF -temp 0.7

⁠CLI Options

⁠Prompt Command Options

For additional options (Whisper, image model, etc.), edit ~/.ezlocalai/.env:

⁠Data Persistence

All data is stored in ~/.ezlocalai/:

Models persist across container updates - you won't re-download them when updating the CLI or rebuilding the CUDA image.

⁠Benchmarks

⁠ASUS GX10 / NVIDIA GB10

Build the ARM64 image on the GX10 with a CUDA 13 capable host driver and the NVIDIA Container Toolkit:

docker compose -f docker-compose-gb10.yml build
docker compose -f docker-compose-gb10.yml up -d

The GB10 image uses CUDA 13 and native sm_121 kernels, a matched CUDA torch/torchaudio pair, and a source-built CUDA/cuDNN CTranslate2 for Whisper. It also builds the native Qwen TTS and stable-diffusion.cpp image backends. The ordinary PyPI ARM64 CTranslate2 wheel used previously lacked CUDA support. See NVIDIA's GB10 compilation guidance⁠ and CTranslate2's build options⁠. The first build compiles several native runtimes; subsequent builds reuse layers.

Resident CUDA embedding instances run inference warmup before joining the pool. EMBEDDING_KEEP_LOADED=true retains them; EMBEDDING_WARMUP=false opts out of the warmup. CPU and transient instances skip it. This moves first-use CUDA work into startup rather than the first embedding request. Each instance retains its full configured context, and input is never truncated for speed.

The GB10 profile starts with two text slots and two resident instances each for embedding, STT and TTS; image and video remain one generation each. The measured 1/2/4-replica sweep found little throughput benefit from four versus two, while two left about 19 GiB more available RAM. Larger counts can be configured for a different workload. Existing .env values override compose defaults; to apply the tested auxiliary pool sizes while preserving other settings:

python3 scripts/configure_gb10_pools.py --env-file .env --replicas 2
docker compose -f docker-compose-gb10.yml up -d

This creates a private .env.pre-gb10-pools backup. /v1/resources includes model_pools with each replica's actual device, so a single CPU fallback cannot hide behind another CUDA instance in the same pool. GB10 defaults to QWEN_TTS_DISABLE_CUDA_GRAPHS=true to avoid the observed libmtmd audio-decoder failure during repeated GPU requests; the setting applies only to TTS child processes. Text and embedding CUDA graph settings are unchanged.

Compare direct worker timings immediately after startup and again while warm:

python scripts/benchmark_worker.py --base-url http://GX10:8091 \
  --text-model openbmb/MiniCPM5-2B-GGUF --output gx10.json
# Optionally add --speech-file sample.wav --tts to exercise both voice models.

The benchmark reads EZLOCALAI_API_KEY from the environment or .env, tests short through approximately 8,500-token inputs and concurrent embeddings, checks embedding dimensions/norms, and records wall times separately from LLM native timings. Compare identical models, quantization, context and concurrency; exclude image builds and other GPU work from timing runs. /v1/resources records loaded models and device placement in each result file.

See the GX10 measurements and remaining limitations⁠ for first-request embeddings, speech/media checks and the CUDA graph comparison.

For Docker access without sudo, add the host account to the Docker group once:

sudo usermod -aG docker josh
# Log out and back in, or start a shell with refreshed group membership:
newgrp docker
docker compose -f docker-compose-gb10.yml ps

This follows Docker's Linux post-install instructions⁠. Docker group membership grants root-level Docker access; do not loosen socket permissions.

⁠RTX 4090

Performance tested on Intel i9-12900KS + RTX 4090 (24GB):

Both models pre-calibrate at startup and hot-swap in ~1 second.

⁠Distributed Fallback / Multi-Machine Setup

ezlocalai supports a distributed fallback system where multiple instances can fall back to each other when local resources (VRAM/RAM) are exhausted, or fall back to any OpenAI-compatible API. This enables:

• Load balancing: When one machine is busy, requests automatically route to another
• Redundancy: If one server is overloaded, the fallback handles requests
• Resource optimization: Each machine handles what it can, forwarding the rest
• Hybrid deployment: Mix local ezlocalai instances with cloud APIs

⁠Configuration

Set these environment variables in your .env file or pass them to the container:

# Fallback server URL - can be another ezlocalai instance OR any OpenAI-compatible API
FALLBACK_SERVER=http://192.168.1.100:8091  # Another ezlocalai instance
# Or use a cloud provider:
# FALLBACK_SERVER=https://api.openai.com/v1

# Authentication for the fallback server
FALLBACK_API_KEY=your-api-key

# Optional: Override model for OpenAI-compatible fallback (pass-through by default)
# If not set, the originally requested model is passed through to the fallback server
# FALLBACK_MODEL=gpt-4o-mini

# Combined memory threshold (VRAM + RAM) in GB - fallback triggers when below this
# Models can offload to system RAM, so combined memory is more accurate than VRAM alone
FALLBACK_MEMORY_THRESHOLD=8.0

The system automatically detects whether FALLBACK_SERVER points to another ezlocalai instance or an OpenAI-compatible API by checking for the /v1/resources endpoint. If it's another ezlocalai server, full endpoint forwarding is used (preserving the original request). Otherwise, it falls back to standard OpenAI API calls, passing through the originally requested model (or using FALLBACK_MODEL if set as an override).

⁠Example: Two-Machine Setup

Machine A (Primary with RTX 4090):

EZLOCALAI_URL=http://0.0.0.0:8091
EZLOCALAI_API_KEY=shared-key
FALLBACK_SERVER=http://machine-b:8091
FALLBACK_API_KEY=shared-key

Machine B (Fallback with RTX 3080):

EZLOCALAI_URL=http://0.0.0.0:8091
EZLOCALAI_API_KEY=shared-key
FALLBACK_SERVER=http://machine-a:8091
FALLBACK_API_KEY=shared-key

Both machines fall back to each other - creating a resilient two-node cluster.

⁠Example: Local + Cloud Hybrid

Run a local ezlocalai with OpenAI as the fallback:

EZLOCALAI_URL=http://0.0.0.0:8091
FALLBACK_SERVER=https://api.openai.com/v1
FALLBACK_API_KEY=sk-your-openai-key
FALLBACK_MODEL=gpt-4o-mini

⁠Monitoring Fallback Status

Check the fallback status via API:

# Get resource status including fallback info
curl http://localhost:8091/v1/resources

# Check fallback availability and models
curl http://localhost:8091/v1/fallback/status

⁠What Gets Forwarded

When fallback is triggered to another ezlocalai instance, these endpoints are automatically forwarded:

• /v1/chat/completions - Chat completions (including streaming)
• /v1/completions - Text completions
• /v1/embeddings - Text embeddings
• /v1/audio/transcriptions - Speech-to-text
• /v1/audio/speech - Text-to-speech
• /v1/audio/music - Music generation
• /v1/images/generations - Image generation
• /v1/images/edits - Image editing (image + text to image)
• /v1/videos/generations - Video generation

For OpenAI-compatible APIs, only chat completions and embeddings are forwarded.

⁠On-Demand LLM Residency

DEFAULT_MODEL accepts a comma-separated model list. With LLM_MODEL_RESIDENCY=auto, ezlocalai estimates each configured model's GPU footprint at startup. Models remain resident together when they fit; when models assigned to the same GPU exceed its usable VRAM, only the first model is loaded at startup. Requesting another configured model unloads the idle resident model, loads the requested model, and leaves it resident until a different model is requested.

DEFAULT_MODEL=unsloth/Qwen3.8-27B-GGUF,unsloth/Qwen3.6-35B-A3B-MTP-GGUF
LLM_MODEL_RESIDENCY=auto
LLM_MODEL_RESIDENCY_MARGIN_GB=1.5

Use LLM_MODEL_RESIDENCY=resident to force all configured models to load together or LLM_MODEL_RESIDENCY=swap to force one-at-a-time loading. Swap mode serializes local text work so an active model is never unloaded mid-generation. The worker continues to advertise every configured model, but its heartbeat marks all swap-dependent model slots occupied while the resident model is in use.

Model-only load/unload timings are exposed under model_lifecycle in GET /v1/resources. To alternate configured models and print those timings:

python benchmark_model_lifecycle.py \
  --models unsloth/Qwen3.8-27B-GGUF,unsloth/Qwen3.6-35B-A3B-MTP-GGUF \
  --rounds 2

⁠Qwen3.8-27B Performance Tuning

Router failures are retained independently of worker registration in ROUTER_ERROR_FILE (default /data/router-errors.json, on the router's data volume). ROUTER_ERROR_ARCHIVE_MAX bounds the archive (default 2,000 events). The dashboard and /v1/router/errors show only errors from the past 24 hours, capped at the newest 100 archived/live events (50 in the HTML table). Worker error badges count all retained events in that 24-hour window, before the table limit. Errors expire on the next dashboard refresh, including after a router or worker restart. The API's total_errors remains cumulative; recent_error_count and recent_errors in dashboard/error responses use the 24-hour window. The archive survives pruning, deregistration and router restarts; it cannot recover errors already discarded by an older router. Protect the data volume as error messages may include upstream diagnostics. If a crashed process registers with a new worker ID, recent failures are restored by its stable label and URL so it cannot bypass the circuit-breaker cooldown. A transfer interruption is not proof of an OOM or native crash: correlate its timestamp with the affected worker's container exit status and logs. Tunnel interruptions now preserve the underlying error; LLM failover retries only before assistant output starts, otherwise it emits an explicit stream error rather than silently completing or duplicating output.

Qwen3.8-27B automatically uses MTP, through xllamacpp 2026.10.11549, with one inference slot per instance, three draft tokens and a 0.1 draft probability threshold. DFlash2 remains opt-in with LLM_SPECULATIVE_TYPE=dflash2; only that backend downloads the revision-pinned Inco Q4_K_M draft⁠ (about 1.1 GB of weights, plus its runtime buffers) into the shared HF cache. The target model and sampling settings are unchanged: draft tokens are verified by the target, not accepted unconditionally. Physical prompt batches are hardware- and context-aware: 24 GB cards use an ubatch of 1024 through 200K context and 512 above 200K, while 32 GB cards use 1024. Other MTP model families retain their conservative defaults.

The host-RAM prompt cache is context-aware, but its requested capacity is capped to the smaller of 25% of total RAM or available RAM minus 4 GiB. This prevents a cache that grows during long-context traffic from OOM-killing a memory-constrained worker. The effective and requested sizes are reported in /v1/resources under model_lifecycle.loaded_llm_runtime. Tune the guard with LLM_PROMPT_CACHE_RAM_MARGIN_MIB and LLM_PROMPT_CACHE_MAX_RAM_FRACTION. LLM_PROMPT_CACHE_ALLOW_UNSAFE=true restores uncapped behavior, including for explicit LLM_PROMPT_CACHE_MIB values, and should only be used after verifying host-RAM headroom under long prompts.

The CUDA and GB10 images build xllamacpp from source to target specific GPU architectures. The DFlash large-image cache fix is included upstream in the pinned llama.cpp revision; see native build details⁠. Test an idle worker's vision stream with python benchmark_vision.py --url http://localhost:8091 --stream (set EZLOCALAI_API_KEY if authentication is enabled).

All values remain operator-overridable:

LLM_SPECULATIVE_TYPE=auto  # auto, dflash2, mtp, none
MTP_SPEC_DRAFT_N_MAX=auto  # Qwen3.8-27B: T4=2; other cards=3
MTP_SPEC_DRAFT_P_MIN=auto  # Qwen3.8-27B: 0.1
KV_CACHE_TYPE=auto        # q4_0; q8_0 remains an explicit precision opt-in
DFLASH_SPEC_DRAFT_N_MAX=auto  # T4: 2; 3090: 3; others: 4; explicit 1..7 overrides
DFLASH_SPEC_DRAFT_P_MIN=0.0
LLM_BATCH_SIZE=auto
LLM_UBATCH_SIZE=auto

GPU-aware defaults for Qwen3.8-27B (explicit settings take precedence):

These are starting profiles, not measured optima for every workload. To tune a mixed fleet from one configuration, use card-specific overrides such as MTP_SPEC_DRAFT_N_MAX_3090=3, MTP_SPEC_DRAFT_N_MAX_5090=4, or KV_CACHE_TYPE_5090=q8_0. The same suffixes work for T4, A100, and H100 (for example MTP_SPEC_DRAFT_N_MAX_T4=2, KV_CACHE_TYPE_A100=q8_0, and DFLASH_SPEC_DRAFT_N_MAX_H100=4). Card-specific overrides beat global overrides. KV_CACHE_TYPE=auto selects the table; an existing explicit KV_CACHE_TYPE=q4_0 still keeps Q4 on every card unless overridden per card. Other model families and GPU types retain Q4 by default; Jetson keeps its explicit F16 setting. The memory planner uses the resolved cache precision. The local Q3 / 3090 Ti comparison⁠ found mixed DFlash gains, including regressions on short thinking requests. The MTP three-token baseline is shared across larger cards; T4 starts with two. Higher values require on-card validation, not extrapolation from free VRAM. Batch/ubatch remain hardware- and context-aware as described above; explicit operator values are preserved.

Deployment: remove an explicit LLM_SPECULATIVE_TYPE=dflash2 override or set it to auto/mtp, then rebuild/restart each worker. Explicit MTP_SPEC_DRAFT_* and KV_CACHE_TYPE* values still win; set them to auto (or remove them) to adopt these defaults. The router does not choose the native decoding backend. DFlash's optional starting lengths are 2 on T4, 3 on 3090, and 4 on the other cards.

The Colab notebook probes GPU and host memory inside its isolated server environment. For Qwen3.8-27B Q3_K_XL, its starting settings are:

With MTP or DFlash enabled, N_PARALLEL=3 loads three independent copies of the model, each with native n_parallel=1 and the full LLM_MAX_TOKENS context. Requests reserve an instance through completion or stream cancellation; additional requests queue until one becomes free. The API exposes one public model name. Updated routers use the advertised independent instances even with ROUTER_BUSY_SLOT_FALLBACK=false. Both the worker and router need this update. Without speculative decoding, N_PARALLEL retains native shared-slot behavior. For speculative decoding, N_PARALLEL=0 still selects one instance.

The A100/H100 notebook default assumes approximately 22 GiB per Qwen3.8-27B Q3_K_XL instance and reserves 4 GiB of headroom: at least 70 GiB free selects three; 48–70 GiB selects two; smaller allocations select one. This gives full 80 GB A100 and H100 runtimes three instances while keeping 40 GB and MIG allocations memory-gated. Other model/quant choices retain one instance. Override with inference_overrides = {"N_PARALLEL": "1"}. The residency planner counts every copy; an overcommitted configuration falls back to swapping with one usable slot. Replicas increase concurrency and share GPU compute and bandwidth; they do not promise faster individual requests.

These Colab profiles are conservative starting points, not benchmarks on those GPUs. Context follows free, visible VRAM: below 20 GiB uses 8,192 tokens, 20–32 GiB uses 230,000, and at least 32 GiB uses 262,144. The 230K setting reflects the reported Qwen3.8-27B Q3_K_XL workload on 24 GB cards; 40 GB cards use the full context. MIG partitions and busy devices therefore do not inherit full-card memory budgets. Batch sizes also respond to free memory; T4 drops to 256 / 64 below 12 GiB free. The 27B model plus its vision projector is a tight fit on T4; CPU offload can still be necessary and depends on available system RAM. Keep other services disabled. The notebook prints its selection; inference_overrides can set explicit context, batch, cache, or speculative-decoding values. Context and host-cache limits here apply only to the notebook; existing server context/cache settings are unchanged.

For this 27B model, target KV at 262,144 tokens is approximately 4.5 GiB with Q4 versus 8.5 GiB with Q8 (excluding recurrent state, weights, draft and compute buffers). Q8 therefore needs about 4 GiB extra. It is a precision upgrade, not a guaranteed speed improvement. Keep Q4 on 24 GB cards at long context. A smaller main-model quant also does not guarantee faster prefill or decode: kernel choice, acceptance, cache traffic and prompt content matter.

An isolated sweep on an idle worker can be run with:

python benchmark_speculative.py --context 220000 --tokens 256 \
  --draft-lengths 3,4,5,7 --prompt-chars 0,120000,360000 --repeats 2 \
  --kv-cache q4_0

The sweep includes no-speculation and MTP controls, real code prefixes, actual prompt-token counts, prefill/decode timing, and greedy output hashes. Use the same immutable --prompt-file for all runs/cards. Allocated context alone is not a long-context benchmark: compare actual prompt lengths. Do not run this alongside the resident server on the same GPU. Greedy hash checks are a smoke test, not a proof of quality or of production-sampling throughput. Use --sampling-profile thinking or --sampling-profile instruct for the actual ezlocalai sampling settings; non-greedy runs do not compare output hashes. For a focused follow-up, --backends mtp,dflash2 skips the no-speculation control.

Single-GPU NVIDIA workers can also A/B test llama.cpp's experimental concurrent CUDA-stream optimization with GGML_CUDA_GRAPH_OPT=1. It primarily targets token-generation throughput rather than prompt processing. Results vary by GPU and model, so leave it disabled unless a representative decode benchmark shows a repeatable improvement.

Existing MTP_SPEC_DRAFT_* settings apply only with the MTP backend; they do not tune DFlash2. Set LLM_SPECULATIVE_TYPE=dflash2 for an A/B comparison or none to disable speculation. Other model families never receive the 27B draft. DFLASH_MODEL_FILE selects another quant from the same pinned repository; DFLASH_MODEL_PATH can point to an already downloaded compatible draft.

Benchmark representative prompts on each card: more speculation is not always faster, and a draft that forces target layers onto CPU can erase the benefit. Explicit LLM_UBATCH_SIZE values are attempted first; the resilient loader retries smaller physical batches if model initialization runs out of GPU memory.

For long-context prefill comparisons, use benchmark_prefill.py inside the CUDA image with the serving worker stopped. Freeze the input file between runs (for example, git show <commit>:Pipes.py > /tmp/prefill-source.py):

python benchmark_prefill.py --prompt-file /tmp/prefill-source.py \
  --prompt-chars 720000 --context 220000 --backend mtp \
  --batch 4096 --ubatch 512 --output /tmp/prefill.json

The report separates a cold prompt, an identical repeat, a short follow-up, and a larger appended message. Compare actual prompt tokens, evaluated tokens, cached tokens, and native prefill time; a 160K context with a four-token cache miss is a different workload from 160K uncached tokens. The source hash helps keep comparisons reproducible. Worker `/v1/r