kwaailab/kwaainet-api repository overview

⁠KwaaiNet API - OpenAI Compliant RESTful API

KwaaiNet is a project focused on distributed inference and training of AI models. Currently, it supports text generation models only. The goal of this project is to decentralize GPU/CPU requirements for AI, enabling efficient workflows.

KwaaiNet is open-source and, like many other open-source projects, it builds on the contributions of existing technologies. It leverages the following:

• Petals⁠
• Hivemind⁠
• libp2p⁠

⁠KwaaiNet API

KwaaiNet API provides an OpenAI-compliant RESTful API for the Petals distributed inference SDK in Python. This makes it easy to integrate with other applications, as the OpenAI API format is becoming the de facto standard for LLM inference.

The source code for KwaaiNet API is available on GitHub:

🔗 KwaaiNet API Repository⁠

⁠System Requirements

⁠Disk Space

The KwaaiNet API requires sufficient disk space to host the tokenizer for the model being served. It is recommended to map a volume to /root/.cache to persist downloaded models between reboots.

⁠Environment Variables

⁠Optional Variables:

• KWAAINET_MODEL: The Hugging Face model path for inference.

  • defaults: unsloth/Llama-3.1-8B-Instruct

• INITIAL_PEERS: The libp2p multi-address of the bootstrap peers to enable routing requests within the distributed network.

  • default: kwaainet bootstrap peers.

• HUGGING_FACE_HUB_TOKEN: Required if using gated models from Hugging Face to enable model downloads.

⁠Example Configuration

If you are using a gated Hugging Face model, ensure that you set the necessary environment variables before running the API:

export KWAAINET_MODEL=deepseek-ai/DeepSeek-R1-Distill-Llama-8B
export HUGGING_FACE_HUB_TOKEN=your_huggingface_token

This ensures that the API connects to the decentralized swarm and serves inference requests seamlessly.

⁠API Endpoint

The KwaaiNet API listens for requests on port 8000 by default. This port can be mapped to 443 to serve requests over HTTPS, for example, at https://api.kwaai.ai.

Note: Simply opening port 443 does not enable SSL. A valid SSL certificate must be applied to secure the connection. A common approach is to use tools like ngrok, NGINX reverse proxy, or other SSL termination solutions to facilitate HTTPS.

⁠Running KwaaiNet API with Docker

⁠Pull the Image

docker pull kwaailab/kwaainet-api:latest

⁠Run the Container with Port Mapping

docker run -d --name kwaainet-api \
  -v /root/.cache:/root/.cache \
  -p 443:8000 \
  kwaailab/kwaainet-api:latest

⁠Running with Docker Compose

Create a docker-compose.yml file:

version: '3'
services:
  kwaainet-api:
    image: kwaailab/kwaainet-api:latest
    container_name: kwaainet-api
    volumes:
      - /root/.cache:/root/.cache
    ports:
      - "443:8000"
    restart: unless-stopped

Start the container using:

docker-compose up -d

For further details and contributions, please refer to the repository documentation and official guidelines.