blazux/llm-lab repository overview

LLM-Lab is a complete environment for training language models from scratch, featuring a visual node-based GUI that makes architecture design accessible without writing code.

⁠🎯 What You Can Do

• Design Models Visually: Drag-and-drop nodes to build custom transformer architectures
• Train From Scratch: Full pretraining on datasets like FineWeb-Edu, RedPajama, The Pile
• Real-Time Monitoring: Live training metrics, loss curves, and evaluation results

⁠🚀 Quick Start

docker run --gpus all -p 8000:8000 blazux/llm-lab:latest

Open http://localhost:8000⁠ in your browser and start designing!

⁠⚡ Features

⁠Architecture Components:

• Attention: MHA, GQA, MQA, MLA (DeepSeek-V2 style)
• Position Encodings: RoPE, ALiBi, YARN, Sinusoidal
• Activations: SwiGLU, GELU, ReLU
• Normalization: RMSNorm, LayerNorm

⁠Training Features:

• Modern optimizers: AdamW, Muon, Lion, Sophia
• Learning rate schedules: Cosine, Linear, Polynomial
• Automatic eval metrics and checkpointing
• Gradient clipping and accumulation

⁠Built for Testing:

• Experiment with different architectures (GPT, Llama-style, DeepSeek-style)
• Compare attention mechanisms and position encodings
• Train models from 100M to 1B+ parameters
• Resume training from checkpoints

⁠🔧 Technology Stack

• Frontend: React + TypeScript + ReactFlow
• Backend: PyTorch + FastAPI
• Training: BFloat16 mixed precision, streaming datasets
• Monitoring: Real-time SSE metrics streaming

⁠📝 Volume Mounts

Mount directories for persistent storage:

docker run -d -p 8000:8000 \ 
--gpus all \
-v $(pwd)/checkpoints:/app/checkpoints \
-v $(pwd)/cache:/app/cache \
--name llm-lab \
blazux/llm-lab

License: Apache-2.0 Source: https://github.com/blazux/LLM-Lab⁠