iterateai/lifeboat repository overview

⁠Lifeboat

High-performance LLM inference server - 2x the concurrent capacity on the same GPU, with confidential computing built in.
_{Built by iterate.ai⁠, Dream big, build fast.}

⁠What it is

Lifeboat is an LLM inference server built for the concurrency and long-running context that agentic workloads demand. It runs on infrastructure you already own - roughly 2x the concurrent capacity on the same GPU, no per-token cloud costs, and data stays inside your perimeter.

The gain comes from adaptive KV-cache management, fairness-aware scheduling and memory tiering - not from quantizing model weights. Weights stay BF16; only KV-cache values use FP8 E4M3.

It ships as a single container with a management UI, an OpenAI-compatible API (plus Anthropic /v1/messages), a token-aware load balancer, multi-node cluster management, and optional GPU confidential computing for protecting data in use.

It also runs where there is no GPU at all - the Lite image (~720 MB against 17 GB) serves GGUF models on CPU or Vulkan, and a native desktop app runs it with no Docker at all. Same console, same API, same load balancer.

_{Baseline is a stock build of the upstream tensor engine, not Lifeboat with its optimizations off. Qwen3-30B-A3B (MoE, ~57 GB BF16) on one RTX PRO 6000 Blackwell (96 GB), 4,096-token prompts / 512-token completions, prefix caching disabled both sides. KV pool size dominates and is unrecorded here: given headroom the two converge. Harness and the raw result files these figures are read out of.⁠}

⁠pip install - no Docker, no GPU

pip install lifeboat
lifeboat engine install
lifeboat up          # console on http://127.0.0.1:8001

The quickest way to evaluate Lifeboat, and how to run it on small hardware: console, model registry, load balancer and the full OpenAI-compatible API, with no container and no accelerator. lifeboat doctor reports the board's cores, memory, GPU and the largest model it can serve before you download anything. Python 3.10-3.13 on macOS (Apple Silicon), Linux x86-64/ARM64 and Windows x64 - PyPI⁠, quickstart⁠.

Edge and embedded hardware is a first-class target. The Linux engines link glibc 2.27 and compile every CPU variant in, selected at runtime - so one artifact runs from a 2018 distribution to a current one, and from an Atom to a Zen 5. Nothing is -march=native.

Measured against other servers, same hardware and same model file, out of the box: 1.94x llama.cpp on a 64-core server, 2.01x ollama on a Jetson, 1.48x on a Mac. Harness and raw results⁠.

It carries the GGUF engine only: the 2x concurrency, FP8 KV cache, speculative decoding and safetensors weights stay with the container, as do Intel Macs (no wheel).

⁠Desktop app - no Docker

A native app with a tray icon, for running models on a laptop or workstation. It bundles the GGUF engine and the same control plane, console and API as the container. GPU offload uses Metal on Apple Silicon and Vulkan on Windows and Linux, which covers NVIDIA, AMD and Intel with one download.

The macOS and Windows builds are code-signed (macOS also notarized); the Linux builds are not, and ship a checksum file instead.

⁠Downloads - github.com/IterateAI/lifeboat-releases/releases⁠

Exact filenames and a SHA256SUMS for every build are on the releases page.

Linux builds need glibc 2.31+ (Debian 11+, Ubuntu 20.04+, RHEL 9+). They are built in an old-glibc container and verified on a clean Debian 11 before release, not on the machine that made them.

⁠Installing

macOS - open the .dmg and drag Lifeboat to Applications. Signed with a Developer ID, notarized by Apple and stapled, so there is no security warning and it works offline.

Windows - run the setup.exe; per-user, so no administrator prompt.

Linux - sudo apt install ./lifeboat-desktop_2.2.52_amd64.deb, or unpack the tarball with sudo tar -C / -xzf Lifeboat-2.2.52-linux-x86_64.tar.gz. Use apt install ./file.deb rather than dpkg -i: the tray binding and the Vulkan loader are recommended packages and dpkg skips them. You get /opt/lifeboat plus lifeboat-core (server and CLI, runs headless) and lifeboat-tray on PATH.

Per-platform requirements, GPU support, upgrading and uninstalling: docs/desktop.md⁠.

⁠Verifying a download

macOS and Windows artifacts are code-signed. Linux has no equivalent - verify with sha256sum -c SHA256SUMS-linux.txt --ignore-missing.

⁠What it does

The tray menu has Open Console, Show Log and Quit - no Dock icon, by design. The console at http://127.0.0.1:30800⁠ asks you to create the administrator account on first visit.

It refuses a safetensors download on macOS or Windows before it starts rather than after an hour - that format needs the tensor engine (Linux, CUDA or ROCm only). See Which model format⁠.

⁠Quick start

Lifeboat ships three images. Pick by what the host has - see Tags⁠:

# OPTIONAL. Without it, the console asks you to create the administrator
# account on first visit - Lifeboat ships with no credentials at all.
sudo mkdir -p /etc/lifeboat
sudo install -m 600 /dev/stdin /etc/lifeboat/admin-pass <<< 'CHANGE-ME'

⁠NVIDIA GPU

docker run --device nvidia.com/gpu=all --name lifeboat -p 8001:8001 \
  -v lifeboat-data:/opt/lifeboat/.lifeboat \
  -v /path/to/models:/models \
  -v /etc/lifeboat/admin-pass:/run/secrets/admin-pass:ro \
  -e LIFEBOAT_ADMIN_PASSWORD_FILE=/run/secrets/admin-pass \
  -e TMPDIR=/opt/lifeboat/.lifeboat/tmp \
  --security-opt no-new-privileges \
  --cap-drop ALL \
  --cap-add NET_BIND_SERVICE --cap-add SETUID --cap-add SETGID --cap-add CHOWN \
  --read-only --tmpfs /tmp:size=2G --tmpfs /run:size=10M \
  -dt --restart=always \
  iterateai/lifeboat:latest

⁠AMD GPU

There is no container toolkit and no --gpus flag on ROCm - the GPU is passed through as two device nodes, /dev/kfd (the compute node) and /dev/dri (the render nodes):

docker run --name lifeboat -p 8001:8001 \
  -v lifeboat-data:/opt/lifeboat/.lifeboat \
  -v /path/to/models:/models \
  -v /etc/lifeboat/admin-pass:/run/secrets/admin-pass:ro \
  -e LIFEBOAT_ADMIN_PASSWORD_FILE=/run/secrets/admin-pass \
  -e TMPDIR=/opt/lifeboat/.lifeboat/tmp \
  --cap-drop ALL \
  --cap-add NET_BIND_SERVICE --cap-add SETUID --cap-add SETGID --cap-add CHOWN \
  --read-only --tmpfs /tmp:size=2G --tmpfs /run:size=10M \
  --device=/dev/kfd --device=/dev/dri \
  --ipc=host \
  -dt --restart=always \
  iterateai/lifeboat:latest-rocm

Confirm the GPU is visible from inside the container:

docker exec lifeboat rocm-smi

⁠No GPU, integrated GPU, or Graviton - Lite

No accelerator runtime, so no toolkit, no device mapping and no driver floor. Serves GGUF on the CPU, or on a Vulkan-capable GPU (Intel Arc/iGPU, AMD Radeon, older NVIDIA) if you pass the render node through:

docker run --name lifeboat -p 8001:8001 \
  -v lifeboat-data:/opt/lifeboat/.lifeboat \
  -v /path/to/models:/models \
  -e TMPDIR=/opt/lifeboat/.lifeboat/tmp \
  --cap-drop ALL --cap-add NET_BIND_SERVICE \
  --cap-add SETUID --cap-add SETGID --cap-add CHOWN \
  --read-only --tmpfs /tmp:size=2G --tmpfs /run:size=10M \
  --device=/dev/dri -dt --restart=always \
  iterateai/lifeboat:lite

Drop --device=/dev/dri on a host with no GPU: Docker refuses to start a container whose devices: entry does not exist.

Sizing. Decode speed is bounded by memory bandwidth, not core count, so no setting changes it. On 8 GB, 1.5B-4B at 4-bit is comfortable and 14B will not fit. GET /api/hardware/profile sizes the actual host, reading a container memory limit from the cgroup rather than /proc.

⁠Any image

Then open http://localhost:8001⁠ and add a model from the dashboard.

• Dashboard - http://localhost:8001
• OpenAI-compatible API - http://localhost:8001/v1/chat/completions
• Anthropic-compatible API - http://localhost:8001/v1/messages
• Health / version - http://localhost:8001/api/version
• Prometheus metrics - http://localhost:8001/metrics

⁠Docker Compose

One compose file per target. The AMD file sets the device nodes, groups and ipc: host; the NVIDIA file sets up CDI; the Lite file defaults GPU passthrough to a harmless self-mapping:

# NVIDIA
curl -fsSLO https://raw.githubusercontent.com/IterateAI/lifeboat-releases/main/install/docker-compose.yaml

# AMD / ROCm
curl -fsSL -o docker-compose.yaml \
  https://raw.githubusercontent.com/IterateAI/lifeboat-releases/main/install/docker-compose.rocm.yaml

# CPU / Vulkan (Lite)
curl -fsSL -o docker-compose.yaml \
  https://raw.githubusercontent.com/IterateAI/lifeboat-releases/main/install/docker-compose.cpu.yaml

docker compose up -d
docker compose logs -f lifeboat

Or let the installer choose - it preflights the host, detects the accelerator (Lite when there is no GPU), downloads the matching compose file and writes .env:

curl -fsSL https://raw.githubusercontent.com/IterateAI/lifeboat-releases/main/install/get-lifeboat.sh | bash

# preflight only, change nothing
curl -fsSL https://raw.githubusercontent.com/IterateAI/lifeboat-releases/main/install/get-lifeboat.sh | bash -s -- --check

⁠Supported platforms

NVIDIA - :latest

AMD - :latest-rocm

CPU / Vulkan - :lite

Host, any image

Some capabilities are generation-dependent and configured automatically: FP8 needs CDNA3+ on AMD, FP4 needs CDNA4 or Blackwell, and GPU confidential computing is NVIDIA-only (auto-disabled on AMD, with the reason reported).

The GPU images ship two engines - a high-throughput tensor engine and a GGUF engine - and Lifeboat picks per model, including GGUF vision (mmproj) and quantized architectures the tensor engine cannot load. 168+ architectures, plus a Transformers fallback. Lite, the desktop app and pip carry GGUF only.

⁠Which model format

Safetensors needs the tensor engine - Linux with CUDA or ROCm only. The other shapes say so before a download starts, not after an hour of bandwidth.

⁠Licensing

There are three separate free routes, and they are not the same thing:

• 24-hour grace period, in the product, no key and no signup. It starts on first boot so you can pull, launch a model and run traffic before talking to anyone.
• Free tier, for non-commercial and evaluation use. No card. Up to 2 concurrent inference servers on a single node; confidential computing is not included. Perpetual - it does not expire. Request a key from iterate.ai/lifeboat⁠.
• 7-day free trial on your first subscription. It is available on the first purchase for an account with no existing licenses; seats added later are paid seats. No card is required to start it - nothing is due, so none is collected. If the trial ends without one the subscription pauses rather than cancelling, so you can add a card and pick up where you left off.

After the grace period, starting an inference server requires a license - the dashboard, model management and audit log keep working, and the activation dialog is one click from wherever you hit the limit.

# Activate non-interactively during install
curl -fsSL https://raw.githubusercontent.com/IterateAI/lifeboat-releases/main/install/get-lifeboat.sh \
  | bash -s -- --license-key LB-XXXX-XXXX-XXXX-XXXX

Air-gapped sites install first, read the Cluster ID off the License page, and request a signed offline .lic bound to that ID, which they upload through the activation dialog. One license binds to one cluster.

All licensing - free tier, trial, purchase, upgrades and offline files - goes through iterate.ai/lifeboat⁠. That is the only address you need.

The license is a runtime gate, not a pull gate - activation happens in the running container, never at docker pull. No credentials are needed to pull this image.

⁠Kubernetes

The chart is published as a rolling tarball, so no repository checkout is needed. It deploys the rolling image, so re-running helm upgrade picks up the newest build.

helm install lifeboat \
  https://raw.githubusercontent.com/IterateAI/lifeboat-releases/main/helm/lifeboat-latest.tgz \
  --namespace lifeboat --create-namespace \
  --set admin.password='CHANGE-ME' \
  --set gpu.count=1          # add --set gpu.vendor=amd|lite for AMD / CPU

Or add it as a Helm repo so helm upgrade tracks new chart versions:

helm repo add lifeboat https://raw.githubusercontent.com/IterateAI/lifeboat-releases/main/helm
helm repo update
helm install lifeboat lifeboat/lifeboat \
  --namespace lifeboat --create-namespace \
  --set admin.password='CHANGE-ME'

helm show values <chart> lists everything. admin.password is required (or admin.existingSecret) - the chart refuses to render without it. There is deliberately no license flag: activation is a runtime API call, so activate in the dashboard once the pod is up.

gpu.vendor takes nvidia (default), amd or lite, and switches the image, the requested device resource, hostIPC, the seccomp profile and the pod's supplemental groups together - plus it gates OFF the NVIDIA-only runtime class and CDI annotation, which would otherwise make an AMD or CPU-only pod fail admission for a GPU it never asked for.

NVIDIA clusters require nvidia-device-plugin >= 0.15 with deviceListStrategy=cdi-annotations, or the NVIDIA GPU Operator with CDI enabled. AMD clusters require the AMD GPU device plugin (or the AMD GPU Operator). lite requires neither.

⁠Tags

Newest container image: 2.2.53. The two ship on their own cadences, so the numbers are not always the same.

Version tags are immutable; latest, latest-rocm and lite roll forward.

The AMD images are tagged separately on purpose: tensor-engine kernels are compiled per CDNA generation (the FP8 number format differs between CDNA3 and CDNA4), so one image targets one family - pick the tag that names your card. The GGUF engine covers all three, so only the tensor-engine path is generation-specific, and the container warns at startup if it is running on a GPU it was not built for.

⁠What Lifeboat reports back

Stated plainly, because a customer should not have to discover it with tcpdump.

A licensed install sends a heartbeat every 6 hours: the license key, the cluster ID, a pod count, the version, and a short hardware description (CPU model, GPU model, memory) sent once - after that the heartbeat carries only a hash of it, so re-sending is unnecessary. Steady state is about 160 bytes. That heartbeat is also how a license binding is verified, which is a contractual obligation rather than telemetry.

An unlicensed install sends a smaller message once a day, so an evaluation is visible to support. It carries no key.

No model names, no prompts, no completions, no token counts, no request metadata. Enforced in code: the snapshot builder omits those fields, and a test walks every key of the real payload against a forbidden list, so adding one breaks the build.

• LIFEBOAT_TELEMETRY=off disables the hardware and census reporting.
• An offline / air-gapped license sends nothing at all, ever - it never contacts a license server, by construction rather than by configuration.

⁠Links

• Desktop app downloads - https://github.com/IterateAI/lifeboat-releases/releases⁠
• Python package - https://pypi.org/project/lifeboat/⁠
• Product page & licensing - https://iterate.ai/lifeboat⁠
• iterate.ai - https://iterate.ai⁠
• Support - [email protected]⁠

_{Lifeboat includes third-party open-source software; the applicable license texts and attribution notices ship with the product. Interplay®, Generate® and AgentOne™ are trademarks of iterate.ai, protected by issued U.S. patents and others pending.}