darkautism/llmserver-rs repository overview

⁠llmserver-rs

A Rust-based, OpenAI-style API server for large language models (LLMs) that can run on the Orange Pi 5.

⁠Description

This project provides a Rust implementation of an API server that mimics the functionality of OpenAI's LLM API. It allows users to interact with large language models directly from their applications, leveraging the performance and safety of Rust.

⁠Features

• OpenAI-style API: Compatible with OpenAI's API endpoints for easy integration.
• Rust Language: Utilizes Rust for its performance, safety, and concurrency features.
• Hardware Compatibility: Specifically designed to run on the Orange Pi 5, powered by the rk3588 chip.

⁠Installation

To install and run llmserver-rs, follow these steps:

1. Clone the Repository:

git clone https://github.com/darkautism/llmserver-rs

Build the Project:

cd llmserver-rs
cargo build --release

Run the Server:

./target/release/llmserver kautism/DeepSeek-R1-Distill-Qwen-1.5B-RK3588S-RKLLM1.1.4

⁠Install on cluster

You need to find out which sbc in your cluster is cpu rk3588

microk8s kubectl get nodes
NAME                STATUS   ROLES    AGE     VERSION
kautism-desktop     Ready    <none>   16d     v1.32.2
kautism-orangepi5   Ready    <none>   6d16h   v1.32.2

Label your node

microk8s kubectl label nodes <node-name> cpu=rk3588

Apply your yaml, if you don't know how to write it, you can copy k8s/* as template

microk8s kubectl apply -f simple.yaml

⁠Support module

This llmserver now only support these modules

⁠Usage

You can access the online documentation at http://localhost:8080/swagger-ui/⁠, which includes request examples and curl demo code.

The API server provides the following endpoints:

• /v1/completions: Generate text completions based on a given prompt.
• /v1/chat/completions: Generate chat completions for conversational AI.

⁠License

This project is licensed under the MIT License.

⁠Acknowledgements

OpenAI⁠ for their pioneering work in LLM APIs.

Orange Pi 5⁠ for providing the hardware platform.