kwaailab/kwaainet-node repository overview

⁠KwaaiNet Node - Distributed Model Hosting

KwaaiNet is a project focused on distributed inference and training of AI models. Currently, it supports text generation models only. The goal of this project is to decentralize GPU/CPU requirements for AI, enabling efficient workflows.

KwaaiNet is open-source and, like many other open-source projects, it builds on the contributions of existing technologies. It leverages the following:

• Petals⁠
• Hivemind⁠
• libp2p⁠

⁠KwaaiNet Node

KwaaiNet Node allows users to host models in a distributed fashion by contributing their CPU or GPU resources. The model architecture consists of transformer blocks distributed across nodes, with tokens forwarded between them for inference. This approach is similar to how Nvidia utilizes NVLink or Intel uses PCI Express, but KwaaiNet achieves this over TCP/IP on WAN.

Although there is some latency due to internet-based routing, the primary goal is to establish a community-hosted DePIN architecture to reduce reliance on private corporations.

⁠Decentralized Network

KwaaiNet Node is built on libp2p and follows a BitTorrent-like architecture, where nodes can freely join and leave the network. As long as at least one instance of each required transformer block remains online, inference remains possible.

Nodes can function in two modes:

• Relay Node: Behind a NAT/firewall, requiring connection via a Direct Node.
• Direct Node: Has a public IP, allowing it to serve as a connectivity bridge.

⁠Persistent Storage Recommendation

Since nodes download model shards, it is recommended to map a persistent volume to /root/.cache. This prevents repeated downloads upon reboot, reducing bandwidth usage and startup times.

⁠Optional Environment Variables

• ANNOUNCE_ADDR (Optional): A libp2p multi-address (maddr) used to advertise your public DNS or IP if acting as a direct node.

  • Example: /dns/directnode.kwaai.ai/tcp/8080

• INITIAL_PEERS (Required): libp2p multi-address to connect the node to KwaaiNet. Defaults to kwaainet peers.

  • Example: /dns/bootstrap.kwaai.ai/tcp/8080/p2p/xxxxxxxxxxxxxxxxxxxxxx

• KWAAINET_MODEL (Required): The Hugging Face model name for hosting inference.

  • Defaults to unsloth/Llama-3.1-8B-Instruct

• KWAAINET_BLOCKS (Optional, Default = 1): Number of blocks to host; specify more to contribute additional workload.

• PUBLIC_NAME (Optional): A URL or tag to publicly credit your contribution.

  • Example: balaji@kwaainet

⁠Peer-to-Peer Communication & Port 8080

Nodes can operate in relay mode or direct mode:

• Relay Mode: If behind NAT or a firewall, the node can operate without exposing port 8080.
• Direct Mode: If the node has a public IP, exposing TCP port 8080 is required for peer-to-peer communication to establish direct connections.

If running in direct mode, ensure port 8080 is accessible for stable communication with other nodes.

⁠Running KwaaiNet Node with Docker

⁠Pull the Image

docker pull kwaailab/kwaainet-node:latest

⁠Run the Container

For CPU-based inference in relay mode:

docker run -d --name kwaainet-node \
  -e PUBLIC_NAME="balaji@kwaai" \
  -v /root/.cache:/root/.cache \
  kwaailab/kwaainet-node:latest

For CPU-based inference in direct mode (exposing port 8080):

docker run -d --name kwaainet-node \
  -e PUBLIC_NAME="balaji@kwaai" \
  -v /root/.cache:/root/.cache \
  -p 8080:8080 \
  kwaailab/kwaainet-node:latest

For GPU-based inference in direct mode:

docker run -d --gpus all --name kwaainet-node \
  -e PUBLIC_NAME="balaji@kwaai" \
  -v /root/.cache:/root/.cache \
  -p 8080:8080 \
  kwaailab/kwaainet-node:latest

⁠Running with Docker Compose

Create a docker-compose.yml file:

version: '3'
services:
  kwaainet-node:
    image: kwaailab/kwaainet-node:latest
    container_name: kwaainet-node
    environment:
      - PUBLIC_NAME=balaji@kwaai
    volumes:
      - /root/.cache:/root/.cache
    ports:
      - "8080:8080"
    restart: unless-stopped

Start the container using:

docker-compose up -d

KwaaiNet Nodes play a crucial role in decentralizing AI inference by distributing computational workloads across a peer-to-peer network. For more details, visit the repository and documentation.