irfanuruchi/multi-gpu-ai-orchestrator repository overview

⁠Multi-GPU AI Orchestrator

Multi-GPU AI Orchestrator (MGAI) is a Docker-native control plane for running local language models across CPUs and accelerators.

It handles:

• CPU and GPU hardware discovery
• persistent hardware inventory
• model discovery
• built-in model search and downloads
• Runtime Pack installation and management
• automatic runtime selection
• automatic device placement
• automatic CPU fallback
• single-GPU execution
• heterogeneous multi-GPU execution
• weighted multi-GPU model sharding
• persistent RuntimeInstance state
• model lifecycle management
• runtime recovery and reconciliation
• built-in web UI
• OpenAI-compatible chat API

The MGAI Docker image is intentionally lightweight.

It contains the control plane, not the full inference stack.

Inference engines such as llama.cpp CUDA or llama.cpp CPU are installed separately from inside MGAI through the Runtimes section.

⁠v1.0.0

Stable release:

docker pull irfanuruchi/multi-gpu-ai-orchestrator:v1.0.0

Commit-specific release:

docker pull irfanuruchi/multi-gpu-ai-orchestrator:v1.0.0-cc69a5879a9c

Release digest:

sha256:080a06d2e66b6c228476e5838aa7963c38b26b46ba8c30b5b0665aec723589d3

Published platform:

linux/amd64

Source release:

v1.0.0

Source commit:

cc69a5879a9c3cbeefb9c3b94d7e540362a4f1a2

⁠Important: Runtime Packs are installed separately

The MGAI image does not bundle inference runtimes.

A fresh installation contains the MGAI control plane only.

After starting MGAI, open:

Runtimes

in the web interface.

MGAI detects the hardware available on the machine and presents Runtime Packs that can be installed and used on that system.

For example:

CPU-only machine
    |
    +--> llama.cpp CPU

or:

NVIDIA machine
    |
    +--> llama.cpp CPU
    +--> llama.cpp CUDA
    +--> other compatible NVIDIA runtimes

Runtimes that are not applicable to the detected machine are hidden from the normal installation flow.

You choose the runtime you want and click Install.

Only after a compatible Runtime Pack is installed and reaches:

ready

can MGAI use that backend to launch a model.

A normal first setup looks like:

Install MGAI
    |
    v
Open web UI
    |
    v
Hardware detected
    |
    v
Open Runtimes
    |
    v
Install a compatible Runtime Pack
    |
    v
Runtime becomes ready
    |
    v
Use an existing model
or download one through MGAI
    |
    v
Launch
    |
    v
MGAI creates RuntimeInstance

This avoids downloading several gigabytes of CUDA, ROCm, SYCL, MLX, vLLM, or other runtime software that a particular machine may never use.

⁠Main features

MGAI includes:

• CPU and GPU hardware detection
• persistent hardware inventory
• automatic execution planning
• automatic runtime selection
• automatic device placement
• automatic CPU fallback
• single-GPU execution
• heterogeneous NVIDIA multi-GPU execution
• weighted multi-GPU model sharding
• Runtime Pack installation
• Runtime Pack lifecycle management
• local model discovery
• built-in model search/download/install workflow
• support for existing local model libraries
• persistent RuntimeInstance state
• start, stop, restart, and recovery
• runtime reconciliation
• built-in web UI
• OpenAI-compatible chat API
• Docker-based persistent deployment

⁠Models

MGAI supports both model workflows at the same time.

You can:

• point MGAI at models you already have
• search for and download additional models through MGAI

Existing models are configured using:

MGAI_MODEL_DIR

Models downloaded through MGAI are stored in:

MGAI_MODEL_DOWNLOAD_DIR

Example:

~/mgai/models/
├── qwen-model.gguf
├── llama-model.gguf
└── existing-model.gguf

~/mgai/downloads/
├── model-downloaded-through-mgai.gguf
└── another-managed-model.gguf

Both locations can be used by the same MGAI installation.

You do not need to move an existing model library into MGAI-managed storage.

You can keep your current models and use MGAI's built-in downloader for additional models.

Models stay outside the MGAI control-plane image.

⁠Hugging Face access

Public Hugging Face models can usually be downloaded without authentication.

Some repositories are gated, private, access-controlled, or subject to model-specific terms. For those models, MGAI can use:

HF_TOKEN

Set the token in your shell:

export HF_TOKEN="hf_your_token_here"

Then add:

-e HF_TOKEN \

to the appropriate docker run command.

For gated models, you may also need to accept the model terms or request access on Hugging Face before the token can download the repository.

A token is not needed for models you already have locally.

Do not publish or commit real long-lived token values.

⁠Runtime Packs

Runtime Packs are the actual inference environments used to execute models.

Examples include:

llamacpp-cpu
llamacpp-cuda

They are separate from:

irfanuruchi/multi-gpu-ai-orchestrator:v1.0.0

The MGAI image provides the control plane.

Runtime Packs provide the inference engines.

Runtime Pack states can include:

not_installed
pulling / installing
ready
running
stopped
updating
error

MGAI does not silently install heavyweight inference runtimes simply because compatible hardware exists.

Runtime installation is an explicit user action.

⁠Runtime filtering

The Runtimes page is hardware-aware.

MGAI detects the current machine and shows the Runtime Packs applicable to the available hardware.

For example, an NVIDIA host can see CPU and CUDA Runtime Packs while incompatible AMD, Intel, and Apple choices are hidden from the normal installation flow.

⁠Runtime and accelerator architecture

MGAI includes architecture for CPU, NVIDIA CUDA, heterogeneous NVIDIA CUDA, Vulkan, AMD ROCm, Intel SYCL/XPU, Apple Metal/MLX, llama.cpp, vLLM, and native-worker paths.

Not all of these paths received the same level of physical validation for v1.0.0.

MGAI distinguishes between:

physically validated
implemented / preview
planned
unsupported

A runtime being present in the project does not mean it received the same v1.0 validation as the stable CPU and NVIDIA CUDA paths.

⁠Installation

MGAI can be installed directly from Docker Hub.

You do not need to clone the source repository to use it.

⁠Requirements

You need:

• Docker Engine or Docker Desktop
• Linux containers
• enough system RAM for the models you plan to run
• local storage for MGAI state and models

For NVIDIA GPU execution you additionally need:

• supported NVIDIA GPU
• working NVIDIA driver
• Docker GPU passthrough

⁠Important: Linux RuntimeInstance host connectivity

MGAI launches inference RuntimeInstances as separate Docker containers through the mounted Docker socket.

On a native Linux Docker host, the managed runtime containers must be able to resolve and reach the MGAI host. Start the MGAI control plane with:

--add-host=host.docker.internal:host-gateway
-e MGAI_UPSTREAM_HOST=host.docker.internal

The complete Linux examples below include these options.

Without the host-gateway mapping, a RuntimeInstance may be created successfully but remain in:

launching

because the managed runtime cannot complete the expected host/control-plane connectivity path.

This requirement is separate from publishing the MGAI web UI port. For example:

-p 127.0.0.1:8000:8000

controls where the MGAI web UI/API is exposed, while:

--add-host=host.docker.internal:host-gateway
-e MGAI_UPSTREAM_HOST=host.docker.internal

provides the host address used by MGAI-managed runtime containers.

⁠1. Pull MGAI

docker pull irfanuruchi/multi-gpu-ai-orchestrator:v1.0.0

⁠2. Create persistent MGAI storage

docker volume create mgai-data

This volume stores MGAI control-plane state separately from the container.

⁠3. Create model directories

Create a directory for models you already have:

mkdir -p "$HOME/mgai/models"

Create a writable directory for models downloaded through MGAI:

mkdir -p "$HOME/mgai/downloads"

Both can be used at the same time.

~/mgai/models

can hold existing models.

~/mgai/downloads

is writable storage for MGAI's built-in model download/install workflow.

If you plan to use only MGAI's downloader, the existing-model directory can remain empty.

⁠CPU / GPU-less installation

Before creating the control-plane container, remove an older container with the same name if one exists:

docker rm -f mgai 2>/dev/null || true

This does not delete the mgai-data volume or your external model directories.

Run:

docker run -d \
  --name mgai \
  --restart unless-stopped \
  --add-host=host.docker.internal:host-gateway \
  -e MGAI_UPSTREAM_HOST=host.docker.internal \
  -p 127.0.0.1:8000:8000 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v mgai-data:/app/data \
  -v "$HOME/mgai/models:$HOME/mgai/models:ro" \
  -v "$HOME/mgai/downloads:$HOME/mgai/downloads" \
  -e MGAI_MODEL_DIR="$HOME/mgai/models" \
  -e MGAI_MODEL_DOWNLOAD_DIR="$HOME/mgai/downloads" \
  -e MGAI_MODEL_SCAN_ON_START=1 \
  -e MGAI_MODEL_SCAN_RECURSIVE=1 \
  irfanuruchi/multi-gpu-ai-orchestrator:v1.0.0

The host.docker.internal mapping is important on native Linux because MGAI creates RuntimeInstances as separate Docker containers. It gives those managed containers a stable route back to the Docker host and avoids RuntimeInstances becoming stuck in launching because the upstream host cannot be reached.

If you need access to gated/private Hugging Face models, add:

-e HF_TOKEN \

after first setting:

export HF_TOKEN="hf_your_token_here"

Verify with docker ps --filter name=mgai and docker logs -f mgai, then open:

http://127.0.0.1:8000

The CPU installation does not expose NVIDIA GPUs to the MGAI control plane.

⁠Required next step

Starting the MGAI container does not install the CPU inference engine.

After opening the web UI:

Runtimes
    |
    v
Install llama.cpp CPU
    |
    v
Runtime status = ready

Once the CPU Runtime Pack is installed, models can be launched using CPU execution.

On a GPU-less machine, AUTO placement can then select that CPU runtime when appropriate.

⁠NVIDIA installation

First verify that the host sees the NVIDIA GPU:

nvidia-smi

Then verify that Docker can use the GPU:

docker run --rm \
  --gpus all \
  nvidia/cuda:12.8.1-base-ubuntu24.04 \
  nvidia-smi

If that works, remove any older MGAI control-plane container before recreating it:

docker rm -f mgai 2>/dev/null || true

This keeps the persistent mgai-data volume and external model directories intact.

Then start MGAI with NVIDIA access:

docker run -d \
  --name mgai \
  --restart unless-stopped \
  --gpus all \
  --add-host=host.docker.internal:host-gateway \
  -e MGAI_UPSTREAM_HOST=host.docker.internal \
  -p 127.0.0.1:8000:8000 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v mgai-data:/app/data \
  -v "$HOME/mgai/models:$HOME/mgai/models:ro" \
  -v "$HOME/mgai/downloads:$HOME/mgai/downloads" \
  -e NVIDIA_DRIVER_CAPABILITIES=utility \
  -e MGAI_MODEL_DIR="$HOME/mgai/models" \
  -e MGAI_MODEL_DOWNLOAD_DIR="$HOME/mgai/downloads" \
  -e MGAI_MODEL_SCAN_ON_START=1 \
  -e MGAI_MODEL_SCAN_RECURSIVE=1 \
  irfanuruchi/multi-gpu-ai-orchestrator:v1.0.0

The host-gateway mapping is required for the normal native-Linux deployment path used by MGAI-managed RuntimeInstances:

--add-host=host.docker.internal:host-gateway
-e MGAI_UPSTREAM_HOST=host.docker.internal

It allows runtime containers created through /var/run/docker.sock to resolve the Docker host consistently. Without this path, a runtime can be created but fail to finish initialization and remain in launching.

If Hugging Face authentication is required, add:

-e HF_TOKEN \

after setting:

export HF_TOKEN="hf_your_token_here"

Verify the container:

docker ps --filter name=mgai
docker logs --tail 100 mgai

Open:

http://127.0.0.1:8000

MGAI should now detect the NVIDIA devices exposed by Docker.

⁠Optional: expose MGAI on a trusted LAN interface

The default examples bind the web UI/API only to localhost:

127.0.0.1:8000

For a dedicated server that should be reachable from other machines on a trusted local network, bind port 8000 to that server's actual LAN address instead.

Example:

docker rm -f mgai 2>/dev/null || true

mkdir -p /opt/mgai/downloads

docker run -d \
  --name mgai \
  --restart unless-stopped \
  --gpus all \
  --add-host=host.docker.internal:host-gateway \
  -e MGAI_UPSTREAM_HOST=host.docker.internal \
  -p 192.168.0.161:8000:8000 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v mgai-data:/app/data \
  -v /srv/spider/Models:/srv/spider/Models:ro \
  -v /opt/mgai/downloads:/opt/mgai/downloads \
  -e NVIDIA_DRIVER_CAPABILITIES=utility \
  -e MGAI_MODEL_DIR=/srv/spider/Models \
  -e MGAI_MODEL_DOWNLOAD_DIR=/opt/mgai/downloads \
  -e MGAI_MODEL_SCAN_ON_START=1 \
  -e MGAI_MODEL_SCAN_RECURSIVE=1 \
  irfanuruchi/multi-gpu-ai-orchestrator:v1.0.0

Then open:

http://192.168.0.161:8000

Replace 192.168.0.161 and the example storage paths with values for your machine.

Do not bind the management interface to an untrusted or Internet-facing interface without an appropriate security layer. MGAI has access to the Docker socket and therefore has highly privileged control over the Docker host.

⁠Required next step

GPU detection does not mean the CUDA inference runtime is installed.

Open:

Runtimes

in MGAI.

For the stable NVIDIA v1.0 path, install:

llama.cpp CUDA

Once llama.cpp CUDA reaches ready, select a model and launch it. MGAI then creates and manages the RuntimeInstance.

Users do not need to manually create the llama.cpp CUDA container.

⁠Windows / WSL2

The published v1.0 image runs as a Linux container.

On Windows, use Docker Desktop with WSL2/Linux containers. Verify nvidia-smi and Docker GPU passthrough before using the NVIDIA path. Runtime Packs are still installed from inside MGAI.

Docker Desktop provides its own host.docker.internal integration; use networking appropriate to that Docker environment.

⁠First use

After starting MGAI:

1. Open the web UI.
2. Confirm that CPU, GPU(s), RAM, and expected hardware are detected.
3. Open Runtimes.
4. Install the Runtime Pack you want.
5. Wait until its status becomes ready.
6. Add an existing model or download one through MGAI.
7. Select the model.
8. Choose AUTO or an explicit placement target.
9. Launch.

A model cannot use a Runtime Pack that has not been installed and reached ready.

⁠If a RuntimeInstance stays in launching

A RuntimeInstance should not remain indefinitely in:

launching

On a native Linux Docker host, first confirm that the MGAI control plane itself was created with:

--add-host=host.docker.internal:host-gateway
-e MGAI_UPSTREAM_HOST=host.docker.internal

Check the running container configuration:

docker inspect mgai --format '{{json .Config.Env}}'

and:

docker inspect mgai --format '{{json .HostConfig.ExtraHosts}}'

You should see MGAI_UPSTREAM_HOST=host.docker.internal in the environment and a host.docker.internal:host-gateway host mapping.

If those options are missing, recreate only the control-plane container:

docker rm -f mgai

Then rerun the appropriate CPU or NVIDIA installation command from this README. Recreating the control-plane container does not remove the mgai-data volume or external model directories.

Also verify the following:

docker ps --filter name=mgai
docker logs --tail 200 mgai

For NVIDIA execution, verify GPU passthrough independently:

docker run --rm \
  --gpus all \
  nvidia/cuda:12.8.1-base-ubuntu24.04 \
  nvidia-smi

Confirm in the MGAI Runtimes page that the selected Runtime Pack has reached:

ready

before launching a model.

If a failed RuntimeInstance record remains after the underlying connectivity problem has been corrected, use MGAI's normal stop/restart/recovery flow rather than deleting the persistent mgai-data volume. The volume contains RuntimeInstance and other control-plane state.

⁠Built-in model download/install workflow

MGAI includes its own model discovery and download/install workflow.

You can therefore keep an existing model collection while letting MGAI manage additional downloads.

Both are available to the same MGAI installation.

For public Hugging Face models no token may be required; gated/private repositories can use HF_TOKEN after any required access approval.

⁠Automatic placement

AUTO placement chooses an execution path from installed and usable runtimes.

AUTO prefers an installed, usable GPU runtime when appropriate and can fall back to an installed CPU runtime when no usable GPU path exists.

Hardware compatibility alone is not enough.

The selected backend also needs its Runtime Pack installed and ready.

⁠Heterogeneous NVIDIA multi-GPU

MGAI can use compatible NVIDIA GPUs with different VRAM capacities for the same model.

The GPUs do not need to be identical.

For example, an RTX 5060 8 GB and RTX 3050 6 GB can participate in the same model execution, with MGAI assigning different tensor shares instead of assuming an equal split.

Actual placement depends on:

• model size
• available VRAM
• GPU capacity
• current GPU usage
• planner policy
• execution mode
• user-selected placement

Heterogeneous execution across mismatched NVIDIA GPUs was physically tested during the v1.0 release process.

The compatible CUDA Runtime Pack must still be installed first.

⁠CPU fallback

CPU is a first-class execution domain.

On a machine with no usable GPU, AUTO can select the installed llama.cpp CPU Runtime Pack.

The CPU Runtime Pack must be installed and ready.

GPU-less AUTO → CPU behavior was physically validated during the v1.0 release process.

⁠RuntimeInstance management

After a Runtime Pack is installed, MGAI handles the inference containers itself.

MGAI manages:

• RuntimeInstance creation
• runtime selection
• model placement
• device assignment
• model loading
• start
• stop
• restart
• health
• recovery
• reconciliation

⁠Persistent state

MGAI stores control-plane data in:

mgai-data

Because this is a Docker named volume, the control-plane container can be recreated while persistent state remains.

Persistent state includes hardware inventory, RuntimeInstance records, runtime state, and other control-plane data.

The control-plane container is disposable.

The state volume is separate.

⁠Start, stop, and restart MGAI

docker stop mgai
docker start mgai
docker restart mgai
docker logs -f mgai

⁠Remove the control-plane container

docker rm -f mgai

This does not automatically remove:

mgai-data

It also does not remove your external model directories.

⁠Remove persistent MGAI state

Only do this if you intentionally want to delete MGAI control-plane state:

docker volume rm mgai-data

External model directories remain separate.

⁠Updating MGAI

For a future release, remove the existing control-plane container:

docker rm -f mgai

Pull the new version:

docker pull irfanuruchi/multi-gpu-ai-orchestrator:<version>

Run the same current CPU or NVIDIA command from this README again with the new image tag.

When recreating a native Linux deployment, keep the host-connectivity options:

--add-host=host.docker.internal:host-gateway
-e MGAI_UPSTREAM_HOST=host.docker.internal

Dropping them during an update can recreate the RuntimeInstance launching connectivity problem even though the MGAI control-plane container itself starts successfully.

The existing:

mgai-data

volume can be reused.

Your model directories remain unchanged.

Runtime Packs remain separately managed from the MGAI control-plane image.

For reproducible deployments, use explicit version tags such as:

v1.0.0

instead of moving tags.

⁠Verify v1.0.0

Pull:

docker pull irfanuruchi/multi-gpu-ai-orchestrator:v1.0.0

Inspect repository digests:

docker image inspect \
  irfanuruchi/multi-gpu-ai-orchestrator:v1.0.0 \
  --format '{{json .RepoDigests}}'

Official v1.0.0 OCI digest:

sha256:080a06d2e66b6c228476e5838aa7963c38b26b46ba8c30b5b0665aec723589d3

Commit-specific image:

docker pull \
  irfanuruchi/multi-gpu-ai-orchestrator:v1.0.0-cc69a5879a9c

Both release tags were published from the same frozen release image.

⁠v1.0.0 validation

The final v1.0.0 image was physically tested end-to-end for:

• NVIDIA CUDA single-GPU inference
• heterogeneous NVIDIA multi-GPU inference
• CPU inference
• GPU-less CPU inference
• AUTO → CPU fallback
• first-run hardware inventory generation
• local model discovery
• RuntimeInstance lifecycle
• model start and stop
• runtime recovery
• persistent state across control-plane recreation
• source-independent Docker deployment
• CPU-safe public deployment
• NVIDIA public deployment
• real token generation
• physical CUDA VRAM allocation
• VRAM release after stopping the runtime

The Docker image published as:

irfanuruchi/multi-gpu-ai-orchestrator:v1.0.0

is the same frozen image used during final release acceptance.

The Linux deployment examples in this README additionally document the required host-gateway/upstream-host configuration for reliable managed RuntimeInstance startup. This is deployment configuration around the frozen image; it does not change the v1.0.0 image digest.

⁠OpenAI-compatible API

MGAI provides an OpenAI-compatible chat API.

Applications communicate with the MGAI control plane instead of depending directly on a specific Runtime Pack.

This keeps the application-facing API independent from the selected inference backend.

⁠Security

MGAI manages RuntimeInstances using:

/var/run/docker.sock

Docker socket access is highly privileged.

Only run trusted MGAI images.

Do not expose the management API directly to untrusted networks without an appropriate security layer.

The example commands bind MGAI to:

127.0.0.1:8000

so it is local-only by default.

If you intentionally replace the localhost bind with a LAN address such as:

192.168.0.161:8000

treat MGAI as a privileged management service and expose it only on a trusted network unless an appropriate authentication/reverse-proxy/security layer is placed in front of it.

Hugging Face access tokens should also be treated as credentials.

Do not publish or commit real token values.

⁠Source and Docker Hub

GitHub:

https://github.com/irfanuruchi/multi-gpu-ai-orchestrator

Docker Hub:

irfanuruchi/multi-gpu-ai-orchestrator

Stable release:

v1.0.0

Final source commit:

cc69a5879a9c3cbeefb9c3b94d7e540362a4f1a2

⁠License

Multi-GPU AI Orchestrator is licensed under the Apache License 2.0.

Third-party libraries, Runtime Pack images, model weights, base images, and other external components retain their respective licenses.