n8500x/nvidia-gpu-toolkit repository overview

⁠nvidia-gpu-toolkit

Everything a RHEL 9 VM needs to become a GPU node for containerised model inference (OCP / CRI-O, Docker, Podman), baked into one UBI 9 image so it can be carried into an air-gapped site and copied out onto the host:

Installed in the image: nvidia-ctk, nvidia-container-cli, lspci. Not installed in the image: the driver. The container toolkit injects the host's nvidia-smi / libcuda into containers at run time, so a missing nvidia-smi inside a container always means the host side needs fixing.

• Image: n8500x/nvidia-gpu-toolkit (~1 GB, driver rpm included)
• Pins: driver 615.71.09, toolkit 1.20.0-1, device plugin v0.20.0 (build args / env NVIDIA_DRIVER_VERSION, NVIDIA_CONTAINER_TOOLKIT_VERSION, K8S_DEVICE_PLUGIN_VERSION)
• Air-gap: nothing is fetched at run time. Move the image with docker save n8500x/nvidia-gpu-toolkit | gzip > ngt.tgz → docker load, or push it to the internal registry (the k8s manifests also need nvcr.io/nvidia/k8s-device-plugin:v0.20.0 mirrored there).

⁠1. Copy the kit out onto the RHEL 9 VM

docker run --rm -v "$PWD/ngt:/target" n8500x/nvidia-gpu-toolkit packages   # (podman works too)
scp -r ngt rhel9-vm:                                                          # or shared storage

⁠2. Install on the VM: driver → toolkit → runtime

cd ngt
sudo ./install.sh --runtime crio            # OCP / MicroShift / CRI-O node
sudo ./install.sh --runtime docker --verify # Docker host, ends with the documented nvidia-smi sample workload
sudo ./install.sh --runtime podman --verify # Podman via CDI
sudo reboot                                  # first driver install: lets the nvidia module replace nouveau

Offline mode is the default when rpms/ is present. What it runs, in order (each command is printed before it executes; --dry-run prints only):

1. Driver (driver installation guide, RHEL 9; skipped automatically when nvidia-smi already works, force with --driver, skip with --no-driver): hostnamectl, uname -r, then the prerequisites dnf install kernel-devel-$(uname -r) kernel-headers-$(uname -r) gcc gcc-c++ make elfutils-libelf-devel patch file kmod cpio ocl-icd opencl-filesystem libvdpau, the shipped dkms rpm, the local repository installation dnf install ./nvidia-driver-local-repo-rhel9-615.71.09-1.0-1.x86_64.rpm, dnf clean expire-cache, the driver installation dnf install nvidia-open, then post-install systemctl enable/restart nvidia-persistenced, cat /proc/driver/nvidia/version, nvidia-smi.
2. Container toolkit (install guide): dnf install of the four shipped rpms with --disablerepo='*', the documented repo file dropped in (disabled) for later online upgrades, nvidia-ctk --version.
3. Runtime: see the table below.
4. Sample workload with --verify.

See the full plan without touching a host: docker run --rm n8500x/nvidia-gpu-toolkit install [--runtime podman ...].

⁠The documented online commands (what --online runs)

Driver (guide, RHEL 9⁠):

sudo dnf install kernel-devel-$(uname -r) kernel-headers-$(uname -r) gcc gcc-c++ make
sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
sudo dnf clean expire-cache
sudo dnf module enable nvidia-driver:open-dkms
sudo dnf install nvidia-open
sudo systemctl restart nvidia-persistenced
cat /proc/driver/nvidia/version

Container toolkit (install guide⁠) and sample workload⁠:

sudo dnf install -y curl
curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
  sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.0-1
sudo dnf install -y \
  nvidia-container-toolkit-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  nvidia-container-toolkit-base-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  libnvidia-container-tools-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  libnvidia-container1-${NVIDIA_CONTAINER_TOOLKIT_VERSION}
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

The image build runs the same toolkit steps (as root, so without sudo); the only deviation is --allowerasing on the curl install because UBI ships curl-minimal, which conflicts with the full curl package.

⁠NVSwitch boards (HGX A100/H100/H200) in a VM

On an HGX baseboard the eight SXM GPUs are wired to four NVSwitch chips. The driver will not let CUDA initialise on such a GPU until the NVLink fabric has been configured by Fabric Manager, and Fabric Manager can only do that where the switches are. The switches are separate PCIe devices (Bridge [0680], id 10de:22a3 on H100/H200), so a VM that was given only the GPUs is stuck:

gpu-check (the image's default command) reports the Fabric state and the NvLinkDisable module setting in its NVLink fabric section.

Three ways out. The first two are hypervisor-side decisions; the kit covers the guest side of each:

What --nvlink-disable does on the VM, and how to undo it:

echo "options nvidia NVreg_NvLinkDisable=1" | sudo tee /etc/modprobe.d/nvidia-nvlink.conf
sudo dracut -f --kver "$(uname -r)" && sudo reboot
grep NvLinkDisable /proc/driver/nvidia/params        # NvLinkDisable: 1
# undo: sudo rm /etc/modprobe.d/nvidia-nvlink.conf && sudo dracut -f && sudo reboot

What --fabric-manager does (full passthrough only; versions must match the driver):

sudo dnf install -y nvidia-fabricmanager-615.71.09 libnvidia-nscq-615.71.09   # from the kit's local repo
sudo systemctl enable --now nvidia-fabricmanager
nvidia-smi -q | grep -A3 -E '^ *Fabric$'             # State: Completed / Status: Success
sudo journalctl -u nvidia-fabricmanager -n 30

References: Fabric Manager user guide⁠, Shared NVSwitch GPU passthrough integration guide⁠.

⁠3. OCP / Kubernetes: expose the GPU to pods

After install.sh --runtime crio (nvidia is CRI-O's default runtime) on the node, from a machine with cluster access:

oc apply -f k8s/nvidia-device-plugin.yml        # DaemonSet; needs image nvcr.io/nvidia/k8s-device-plugin:v0.20.0 (mirror it)
oc describe node <gpu-node> | grep nvidia.com/gpu   # capacity/allocatable shows the GPU
oc apply -f k8s/gpu-check-pod.yaml && oc logs -f gpu-check   # nvidia-smi from inside a pod

The device plugin DaemonSet runs privileged; on OCP grant its service account the privileged SCC (or deploy it in a namespace that allows it). Inference workloads then just request resources.limits: {nvidia.com/gpu: 1}.

For a full OpenShift cluster with RHCOS workers the supported route is the NVIDIA GPU Operator⁠ (driver, toolkit and device plugin as operator-managed containers); this kit is for RHEL 9 hosts you manage yourself (MicroShift, CRI-O/containerd nodes, Docker or Podman hosts).

⁠4. Check a GPU host from a container

docker run --rm --runtime=nvidia --gpus all n8500x/nvidia-gpu-toolkit
podman run --rm --security-opt=label=disable --device=nvidia.com/gpu=all n8500x/nvidia-gpu-toolkit

Prints PCI devices (lspci, works before any driver), injected /dev/nvidia*, nvidia-smi (driver + CUDA per GPU), the NVLink fabric state / NvLinkDisable setting, nvidia-container-cli info (needs --privileged) and CDI specs, then RESULT: GPU reachable / GPU NOT reachable (exit 1) with the reason. Anything that is not a subcommand is exec'd:

docker run --rm --gpus all n8500x/nvidia-gpu-toolkit nvidia-smi
docker run --rm n8500x/nvidia-gpu-toolkit nvidia-ctk --version
docker run --rm n8500x/nvidia-gpu-toolkit readme

The container runs as root (its jobs — copying into volumes, CDI generation, nvidia-container-cli — need it; nothing listens on a port). NVIDIA_VISIBLE_DEVICES=all / NVIDIA_DRIVER_CAPABILITIES=utility,compute are preset so --runtime=nvidia alone also injects the GPUs.

⁠Build

.\build.ps1                    # tags date+latest (downloads ~650 MB driver rpm once; cached after)
.\build.ps1 -Push              # also push both tags, records .push_tag
docker build --build-arg NVIDIA_DRIVER_VERSION=615.71.09 --build-arg NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.0-1 .

Sources: nvidia.github.io/libnvidia-container (toolkit, repo_gpgcheck=1), developer.download.nvidia.com (driver local-repo rpm, verified in a build stage: the bundled installer key's fingerprint must equal the pinned NVIDIA_DRIVER_KEY_FPR, then rpm -K on the rpm and its contents), EPEL (dkms), GitHub (device plugin manifest), UBI 9 repos. On a host with a TLS-intercepting proxy or AV, build.ps1 auto-detects the Avast root CA and passes DNF_OPTS=--setopt=sslverify=false CURL_OPTS=-k for the build only; the GPG/signature checks above still hold and no CA or insecure setting is baked into the image.

To bump: pick a driver from developer.download.nvidia.com/compute/nvidia-driver⁠ (must have a local_installers/nvidia-driver-local-repo-rhel9-<ver>-1.0-1.x86_64.rpm), a toolkit from the releases⁠, and rebuild with the build args.