axiilay/nanochat repository overview

⁠🚀 nanochat Training Environment

A pre-configured Docker image for nanochat⁠ - a minimal, full-stack LLM implementation designed for educational purposes and rapid experimentation.

To test the pre-trained model, you can check out our online demo⁠

⁠📦 What's Included

⁠⚡ Quick Start

# 1. Launch pod with 8×H100 GPUs
# 2. Container starts automatically with helpful info displayed

# 3. Start training (recommended: use screen session)
screen -L -Logfile speedrun.log -S speedrun bash speedrun.sh

# 4. Wait ~4 hours for complete pipeline

# 5. Chat with your trained model
python -m scripts.chat_web
# Access at http://YOUR_POD_IP:8000

⁠✨ Features

⁠Complete LLM Pipeline

Tokenization → Pretraining → Finetuning → Evaluation → Inference

• Minimal & Hackable: ~8K lines of clean, readable code
• Comprehensive Evaluation: CORE, ARC, GSM8K, HumanEval, MMLU benchmarks
• Web Interface: ChatGPT-like UI for model interaction
• Metrics Tracking: Optional Weights & Biases integration
• Educational: Perfect for learning end-to-end LLM training

⁠📊 Training Outputs

After the 4-hour pipeline completes:

⁠💻 System Requirements

⁠📚 Documentation & Support

• GitHub Repository: karpathy/nanochat⁠
• Detailed Walkthrough: See Discussions⁠
• License: MIT

⁠🔧 Advanced Usage

Enable Wandb Logging:

wandb login
WANDB_RUN=my_run bash speedrun.sh

Download Trained Model:

tar -czf nanochat_model.tar.gz -C ~/.cache/nanochat \
    tok_checkpoints/ chatsft_checkpoints/

Troubleshoot OOM:

# Reduce batch size if running out of memory
torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- --device_batch_size=16

⁠🎯 Use Cases

• Education: Learn LLM training end-to-end
• Research: Experiment with minimal, hackable codebase
• Prototyping: Rapid iteration on model architectures
• Benchmarking: Establish baselines for micro-models

Built with ❤️ by Andrej Karpathy⁠ | Powered by RunPod⁠