rdson/llamate repository overview

Mirror of https://github.com/R-Dson/llamate⁠.

⁠llamate 🌟

llamate (llama + mate) is a simple, "Ollama-like" tool for managing and running GGUF language models from your terminal.

llamate streamlines your local LLM workflow by automating downloads, configuration, and execution. It's designed for users who want a straightforward way to handle models with GPU acceleration.

⁠Key Features ✨

• Easy Model Management: Add models from Huggingface or use simple aliases (llama3:8b). All available can be found here⁠.
• Automated Setup: Downloads and manages GGUF files and required server binaries for you.
• GPU Accelerated: Runs models using a llama.cpp-based server, optimized for GPUs.
• Persistent Configuration: Set default inference parameters (context size, temp, etc.) for each model.

⁠Native Installation 🚀

⁠llama-server is having issues currently with ROCm and will not work by default.

curl -fsSL https://raw.githubusercontent.com/R-Dson/llamate/main/install.sh | bash

⁠Running with Docker 🐳

You can run llamate in a Docker container.

⁠Run the Container

Ensure you have Docker and your GPU drivers installed.

⁠For NVIDIA GPUs:

docker run --gpus all -p 11434:11434 -v ~/.config:/app rdson/llamate <command>

⁠For AMD ROCm GPUs:

⁠llama-server is having issues currently with ROCm and will not work by default.

docker run --device=/dev/kfd --device=/dev/dri --group-add=video -p 11434:11434 -v ~/.config/llamate:/app rdson/llamate <command>

⁠Usage ⚡

⁠1. Add and Download a Model

You can use a pre-configured alias or a full Huggingface repository link. All available aliases can be found on R-Dson/llamate-alias⁠.

# Add and automatically download using an alias
llamate add llama3:8b

# Or, add a specific model from Huggingface with a custom alias
# llamate add <hf_repo>:<hf_file> --alias <your-alias>
llamate add bartowski/Qwen_Qwen3-0.6B-GGUF:Qwen_Qwen3-0.6B-Q8_0.gguf --alias my-model

⁠2. Run a Model

API Server (Ollama-compatible):

# Start the server (listens on localhost:11434 by default)
llamate serve

⁠3. (Optional) Chat with your LLM

Interactive Chat:

# Basic chat function
llamate run llama3:8b

⁠Available Commands ⚡

• init: Initialize llamate
• set: Set global config or model arguments
• add: Add a new model
• list: List configured models
• remove: Remove a model
• pull: Download GGUF file for a model
• show: Show model information
• copy: Copy a model configuration
• list-aliases: List all available model aliases
• config: Model configuration commands
• serve: Run the llama-swap server
• update: Update llamate CLI components
• run: Run a model in interactive chat mode
• print: Print the llama-swap config

⁠Model Management Commands

# List configured models
llamate list

# Show detailed information about a model
llamate show llama3:8b

# Remove a model and optionally its GGUF file
llamate remove llama3:8b --delete-gguf


# To add without downloading, use --no-pull
llamate add llama3:8b --no-pull

# To disable automatic GPU configuration, use --no-gpu
llamate add llama3:8b --no-gpu

# Set model arguments during creation
llamate add llama3:8b --set "ctx-size=8192" "n-gpu-layers=99"

# To listen on all interfaces (public) use --public
llamate serve --public

# Use a custom port
llamate serve --port 9090

# Connect to a specific host or port
llamate run llama3:8b --host <IP> --port <PORT>

# Download GGUF file for a model
llamate pull my-model

# Copy a model configuration
llamate copy source-model new-model-name

⁠Configuration Commands

# List all available model aliases
llamate list-aliases

# Set global config value
llamate set ggufs_storage_path=/new/path

# Set model argument
llamate config set llama3:8b ctx-size 16384

# Get model argument
llamate config get llama3:8b ctx-size

# List all model arguments
llamate config list llama3:8b

# Remove model argument
llamate config remove llama3:8b ctx-size

⁠Utility Commands

# Re-initialize llamate
llamate init

# Update llamate components
llamate update

# Print llama-swap config
llamate print

⁠Configuration

llamate stores all its files in ~/.config/llamate/:

• bin/: Server binaries (llama-server, llama-swap).
• models/: YAML configuration for each model.
• ggufs/: Downloaded GGUF model files.

⁠TODO

• Add pre-configured optimal parameters for pre-configured alias models.
• Add support for importing model yaml files.

⁠Platform Support & Troubleshooting

• Default Support: The default llama-server binary is built for CUDA, Metal, ROCm and Vulkan (Non-CUDA builds are still testing).
• Requirements: You must have your GPU drivers (e.g., CUDA) installed.

⁠Acknowledgement

This tool is built on the following projects:

• llama.cpp⁠: For its incredible, high-performance C++ inference engine.
• llama-swap⁠: For its elegant server wrapper that enables on-the-fly model swapping and multi-model configuration.

Thank you to Ollama⁠ For popularizing a simple and powerful local LLM workflow.

This tool uses binaries compiled from llama-server-compile⁠ and llama-swap⁠.