powderluv/vllm_fp8_throughput repository overview

⁠(Deprecated. Please use https://hub.docker.com/repository/docker/powderluv/vllm_dev_channel/general⁠ )

⁠vllm FP8 Latency and Throughput benchmarks on AMD MI300x

⁠Overview

vLLM is a toolkit and library for large language model (LLM) inference and serving. It deploys the PagedAttention algorithm, which reduces memory consumption and increases throughput by leveraging dynamic key and value allocation in GPU memory. vLLM also incorporates many recent LLM acceleration and quantization algorithms, such as fp8 GeMM, fp8 KV cache, continuous batching, flash attention, hip graph, tensor parallel, GPTQ, AWQ, and token speculation. In addition, AMD implements high-performance custom kernels and modules in vLLM to enhance performance further.

This documentation shows some reference performance numbers and the steps to reproduce it for the popular Llama 3.1 series models from Meta with a pre-built AMD vLLM docker optimized for an AMD Instinct™ MI300X accelerator.

It includes:

· ROCm™ 6.2

· vLLM 0.4.3

· PyTorch 2.4

⁠System configuration

The performance data below was measured on a server with MI300X accelerators with the following system configuration. The performance might vary with different system configurations.

⁠Reproducing benchmark results

⁠Download the model

Download the Model View the Meta-Llama-3.1-405B model at https://huggingface.co/meta-llama/Meta-Llama-3.1-405B⁠. Ensure that you have been granted access, and apply for it if you do not have access.

If you do not already have a HuggingFace token, open your user profile (https://huggingface.co/settings/profile⁠), select "Access Tokens", press "+ Create New Token", and create a new Read token.

Install the huggingface-cli (if not already available on your system) and log in with the token you created earlier and download the model. The instructions in this document assume that the model will be stored under /data/llama-3.1. You can store the model in a different location, but then you'll need to update other commands accordingly. The model is quite large and will take some time to download; it is recommended to use tmux or screen to keep your session running without getting disconnected.

sudo pip install -U "huggingface_hub[cli]"

huggingface-cli login

Enter the token you created earlier; you do NOT need to save it as a git credential

Create the directory for Llama 3.1 models (if it doesn't already exist)

sudo mkdir -p /data/llama-3.1

sudo chmod -R a+w /data/llama-3.1

Download the model

huggingface-cli download meta-llama/Meta-Llama-3.1-405B --exclude "original/*" --local-dir /data/llama-3.1/Meta-Llama-3.1-405B

Similarly, you can download Meta-Llama-3.1-70B and Meta-Llama-3.1-8B.

⁠Model Preparation

To make it easier to run fp8 Llama 3.1 models on MI300X, the quantized checkpoints are available on AMD Huggingface space as follows

• https://huggingface.co/amd/Meta-Llama-3.1-8B-Instruct-fp8-quark-vllm⁠
• https://huggingface.co/amd/Meta-Llama-3.1-70B-Instruct-fp8-quark-vllm⁠
• https://huggingface.co/amd/Meta-Llama-3.1-405B-Instruct-fp8-quark-vllm⁠

Download the model you want to run. For 70B and 405B models, there is an extra step to use the merge.py to merge the splitted llama-*.safetensors into a single llama.safetensors for vLLM. Then move llama.safetensors and llama.json to the saved directory of Meta-Llama-3.1 models in the Step 1.

Take Meta-Llama-3.1-405B as an example,

Create the directory for llama.safetensors and llama.json

sudo mkdir -p /data/llama-3.1/Meta-Llama-3.1-405B/quantized

cp llama.json /data/llama-3.1/Meta-Llama-3.1-405B/quantized

cp llama.safetensors /data/llama-3.1/Meta-Llama-3.1-405B/quantized

For more details, please refer to the model card of Meta-Llama-3.1-70B-Instruct-fp8-quark-vllm and Meta-Llama-3.1-405B-Instruct-fp8-quark-vllm.

These FP8 quantized checkpoints were generated with AMD’s Quark Quantizer. For more information about Quark, please refer to https://quark.docs.amd.com/latest/quark_example_torch_llm_gen.html⁠

⁠System Settings

⁠NUMA balancing setting

To optimize performance, disable automatic NUMA balancing. Otherwise, the GPU might hang until the periodic balancing is finalized. For further details, refer to the AMD Instinct MI300X system optimization guide.

Disable automatic NUMA balancing

sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'

Check if NUMA balancing is disabled (returns 0 if disabled)

cat /proc/sys/kernel/numa_balancing
0

⁠LLM performance settings

Some environment variables enhance the performance of the vLLM kernels and PyTorch's tunableOp on the MI300X accelerator. The settings below are already preconfigured in the Docker image. See the AMD Instinct MI300X workload optimization guide for more information.

⁠vLLM performance settings

export HIP_FORCE_DEV_KERNARG=1
export VLLM_USE_ROCM_CUSTOM_PAGED_ATTN=1
export VLLM_USE_TRITON_FLASH_ATTN=0
export VLLM_USE_TRITON_FLASH_ATTN=0
export VLLM_INSTALL_PUNICA_KERNELS=1
export TOKENIZERS_PARALLELISM=false
export RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES=1
export NCCL_MIN_NCHANNELS=112
export VLLM_FP8_PADDING=1
export VLLM_FP8_ACT_PADDING=1
export VLLM_FP8_WEIGHT_PADDING=1
export VLLM_FP8_REDUCE_CONV=1
export VLLM_SCHED_PREFILL_KVC_FREEPCT=31.0

⁠Benchmark with AMD vLLM Docker

Download and launch the docker,

docker run -it --rm --ipc=host --network=host --group-add render \
--privileged --security-opt seccomp=unconfined \
--cap-add=CAP_SYS_ADMIN --cap-add=SYS_PTRACE \
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
-v /data/llama-3.1:/data/llm \
docker.gpuperf:5000/dcgpu-rocm/vllm_fp8_throughput:20240826b

Benchmark Meta-Llama-3.1-405B with input 128 tokens, output 128 tokens and tensor parallelism 8 as an example,

⁠Inside the container:

torchrun --standalone --nproc_per_node=8 /app/vllm/benchmarks/benchmark_throughput.py \
--model /data/llm/Meta-Llama-3.1-405B \
--quantized-weights-path /quantized/llama.safetensors \
--quantization fp8 \
--kv-cache-dtype fp8 \
--dtype float16 \
--max-num-batched-tokens 65536 \
--gpu-memory-utilization 0.99 \
--max-model-len 8192 \
--num-prompts 2000 \
--tensor-parallel-size 8 \
--input-len 128 \
--output-len 128

You can change to the other models with various input and output length and run the benchmark as well.

For more information about the parameters, please run

/app/vllm/benchmarks/benchmark_throughput.py -h

⁠MMLU_PRO_Biology Accuracy Eval

⁠fp16

vllm (pretrained=models--meta-llama--Meta-Llama-3.1-405B-Instruct/snapshots/069992c75aed59df00ec06c17177e76c63296a26,dtype=float16,tensor_parallel_size=8), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 64

⁠fp8

vllm (pretrained=models--meta-llama--Meta-Llama-3.1-405B-Instruct/snapshots/069992c75aed59df00ec06c17177e76c63296a26,dtype=float16,quantization=fp8,quantized_weights_path=/llama.safetensors,tensor_parallel_size=8), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 32

⁠Performance

⁠LLaMA3 405B

⁠LLaMA2/3 MLPerf 70B

Please refer to the MLPerf instructions for recreating the MLPerf numbers.

⁠Version

⁠Release Notes

20240906a: Legacy quantization formats required --quantization fp8_rocm as a flag instead of --quantization fp8

Updated:

torchvision: https://github.com/ROCm/vision/commit/fab848869c0f88802297bad43c0ad80f33ecabb4/⁠

vLLM: https://github.com/ROCm/vllm/commit/6f35c77845068dcc90c222fdfd1b56c3db149ad1⁠

⁠Docker Manifest

ROCm6.2 GA tip-of-tree (hipBLASLT, rocBLAS, Flash-attention, CK, Triton, MIOpen, RCCL, Apex) Python 3.9 Ubuntu 22 PyTorch 2.4 Release