exllamav2 is "a fast inference library for running LLMs locally on modern consumer-class GPUs".
1.3K
exllamav2 is "a fast inference library for running LLMs locally on modern consumer-class GPUs".
Unofficial Docker image for exllamav2-0.3.2.
Contains:
On the Docker host:
# Set the path to your downloaded Huggingface model
INPUT_MODEL=/mydrive/mymodel
# Do measurement pass (only has to be done once per model)
docker run -it --rm \
-v$INPUT_MODEL:/workspace/input-model \
--runtime=nvidia --gpus all agentlans/exllamav2 \
bash Measure.sh
# Quantize to 4 bits per weight
docker run -it --rm \
-v$INPUT_MODEL:/workspace/input-model \
--runtime=nvidia --gpus all agentlans/exllamav2 \
bash Quantize.sh 4
# Output will be written to
# /mydrive/mymodel/EXL2/Output/4bpw
Content type
Image
Digest
sha256:1b430eeba…
Size
4.5 GB
Last updated
about 1 year ago
docker pull agentlans/exllamav2