Script to convert Huggingface models to rkllm format to running on Rockchip NPU (Orange Pi 5 series)
10K+
Store your Hugging Face token to pass it into container
mkdir -p ~/.cache/huggingface
echo "hf_your_token_here" > ~/.cache/huggingface/token
Run container with command
sudo docker run --rm --privileged \
-v ~/.cache/huggingface:/root/.cache/huggingface \
thanhtantran/rkllm-cli:latest MODEL_IDS
Usage: rkllm-toolkit [OPTIONS] MODEL_IDS...
╭─ Arguments ───────────────────────────────────────────────────────────────────────────────────────────────╮
│ * model_ids MODEL_IDS... HuggingFace model IDs [default: None] [required] │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --qtypes TEXT Quantization types [default: w8a8] │
│ --hybrid-rates FLOAT Hybrid rates [default: 0.0] │
│ --optimization --no-optimization Optimization level [default: optimization] │
│ --platform [rk3588|rk3576] Target platform [default: rk3588] │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Content type
Image
Digest
sha256:fc809d88f…
Size
3.1 GB
Last updated
over 1 year ago
docker pull thanhtantran/rkllm-toolkit-cli