LLM Server run on RUST, expose OpenAPI, run on Orange Pi device with NPU like Orange Pi 5 series
10K+
Only run on Orange Pi with NPU device, like RK3588 series.
docker run --privileged --rm -it -p 8080:8080 --name llmserver-rust thanhtantran/llmserver-rust LLM_MODEL
LLM_MODEL should be like Hugging Face model like thanhtantran/gemma-3-1b-it-rk3588-1.2.0
Example:
docker run --privileged --rm -it -p 8080:8080 --name llmserver-rust thanhtantran/llmserver-rust thanhtantran/gemma-3-1b-it-rk3588-1.2.0
Content type
Image
Digest
sha256:379275043…
Size
66.1 MB
Last updated
over 1 year ago
docker pull thanhtantran/llmserver-rust