Training containers for fine-tuning LLMs on AMD Instinct MI100 GPUs (gfx908, 32GB HBM2).
All images are built on btbtyler09/vllm-dev-mi100+AITER (ROCm 7.0 + PyTorch 2.9.1 + Triton 3.4.0 + CK flash_attn 2.8.3 + AITER).
| Tag | Model | Description |
|---|---|---|
qwen3.5 | Qwen 3.5 (0.8B-4B) | LoRA fine-tuning with Liger FLCE + fla + Triton flash attention |
All images require these docker flags for MI100 GPU access:
docker run -d \
--device /dev/kfd --device /dev/dri \
--group-add video --cap-add SYS_PTRACE \
--shm-size=16g \
...
Solves the three OOM bottlenecks of Qwen 3.5 training on MI100:
| Bottleneck | Cause | Solution | Savings |
|---|---|---|---|
| 248K vocab logits | seq × 248320 × 2 bytes | Liger Kernel FLCE | ~8 GB |
| GatedDeltaNet (18 layers) | PyTorch fallback upcasts to fp32 | flash-linear-attention (fla) | Several GB |
| Full attention (6 layers) | seq² × heads × 2 bytes | Triton flash attention (AMD) | ~8 GB |
Result: ~5-7 GB VRAM per GPU (down from 22+ GB OOM). Tested with Qwen3.5-0.8B at 16384 seq length on 4x MI100.
| Package | Version | Purpose |
|---|---|---|
| transformers | >=5.0.0 | Model loading, training loop |
| peft | >=0.15.0 | LoRA adapters |
| trl | >=0.28.0 | SFTTrainer |
| liger-kernel | git main | Fused Linear Cross Entropy for 248K vocab |
| flash-linear-attention | >=0.4.0 | Triton kernels for GatedDeltaNet layers |
| deepspeed | >=0.18.0 | ZeRO-2 optimizer/gradient sharding |
| accelerate | >=1.0.0 | Multi-GPU launch |
| datasets | >=3.0.0 | Dataset loading |
| bitsandbytes | >=0.43.0 | Quantization support |
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE # Use Triton flash attn (CK crashes on gfx908)
PYTORCH_ALLOC_CONF=garbage_collection_threshold:0.6 # Aggressive GC (expandable_segments unsupported on HIP 7.0)
ROCM_ARCH=gfx908
OMP_NUM_THREADS=16 # Adjust for your CPU
docker run -d --name training \
--device /dev/kfd --device /dev/dri \
--group-add video --cap-add SYS_PTRACE \
--shm-size=16g \
-v $(pwd):/workspace \
-e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
-e PYTORCH_ALLOC_CONF=garbage_collection_threshold:0.6 \
-e ROCM_ARCH=gfx908 \
-e OMP_NUM_THREADS=16 \
btbtyler09/mi100-training:qwen3.5 \
accelerate launch --num_processes 4 /workspace/your_training_script.py
chunk_gated_delta_rule kernel still >varlen_fwd has a cu_seqlens shape mismatch with transformers' _upad_input. The image includes /workspace/mi100_patches.py to fix this. In your training s>
import mi100_patches
mi100_patches.patch_flash_attention()
garbage_collection_threshold instead.Content type
Image
Digest
sha256:a094edc17…
Size
6.4 GB
Last updated
16 days ago
docker pull btbtyler09/mi100-training:2026.06-validated