garenleeasa/ftllm repository overview

fastllm is a high-performance LLMs inference library implemented in C++ with no backend dependencies (e.g. PyTorch).

It enables hybrid inference of MOE models, achieving 20+ tps on consumer-grade single GPUs (e.g., 4090) for DeepSeek R1 671B INT4 model inference.