exllamav2 is "a fast inference library for running LLMs locally on modern consumer-class GPUs".