samso9th/toonstore repository overview

⁠ToonStore

⚠️ NOT PRODUCTION READY - v0.1 Alpha ⚠️

A blazingly fast embedded database with Redis-compatible protocol, built in Rust.

ToonStore is a high-performance key-value store that gives you the speed of embedded databases (5.28M ops/sec) with the convenience of Redis compatibility. Use it as an embedded library for maximum performance, or run it as a Redis-compatible server accessible from any language.

⁠🎯 What is the TOON Format?

TOON (Token-Oriented Object Notation) is a compact, human-readable data format specifically designed for the age of AI and LLMs. In a world where every token counts—both for cost and context window limits—TOON provides 40-60% token savings compared to JSON while maintaining higher LLM comprehension accuracy.

⁠Why TOON in the AI Era?

• 🤖 LLM-Optimized: Achieves 74% accuracy vs JSON's 70% in LLM comprehension benchmarks
• 💰 Cost-Efficient: ~40% fewer tokens = 40% lower API costs for AI applications
• 📊 Schema-Aware: Explicit [N] lengths and {fields} headers help LLMs parse data reliably
• 🔄 JSON-Compatible: Encodes the same objects, arrays, and primitives with lossless round-trips
• 👁️ Human-Readable: YAML-like readability with CSV-style compactness

⁠TOON vs JSON Example

JSON (22,250 tokens):

{
  "metrics": [
    {"date": "2025-01-01", "views": 5715, "clicks": 211, "conversions": 28, "revenue": 7976.46},
    {"date": "2025-01-02", "views": 7103, "clicks": 393, "conversions": 28, "revenue": 8360.53}
  ]
}

TOON (9,120 tokens - 59% reduction):

metrics[2]{date,views,clicks,conversions,revenue}:
  2025-01-01,5715,211,28,7976.46
  2025-01-02,7103,393,28,8360.53

⁠Learn More About TOON

ToonStore uses the TOON format for efficient data storage. To learn more about the format specification:

• TOON Format Repository⁠ - Official spec and implementations
• TOON Specification⁠ - Complete technical specification
• Benchmarks⁠ - Token efficiency & accuracy comparisons

⁠🎯 What is ToonStore?

ToonStore is a persistent key-value database that combines the token-efficient TOON format with extreme performance:

• 🚀 Extreme performance - 5.28M ops/sec for cached reads, 66x faster than network databases
• 💾 TOON format storage - Token-efficient data format perfect for AI/LLM applications
• 💰 Cost-efficient - ~40% fewer tokens means lower storage and transmission costs
• 🔌 Redis compatibility - Works with existing Redis clients (Node.js, Python, Go, etc.)
• 📦 Embedded mode - Use directly in Rust applications for maximum speed
• 🌐 Network mode - Run as a server, connect from any language

⁠⚡ Why ToonStore?

⁠The Problem

• Redis is fast but volatile (RAM-only by default) and complex to persist data
• PostgreSQL/MySQL are reliable but slower for key-value workloads
• RocksDB/LevelDB are fast but lack network access and use inefficient storage formats
• Traditional formats (JSON, XML) waste tokens and storage space in the AI era

⁠The Solution: ToonStore

ToonStore combines the best of all worlds with TOON format storage:

⁠🎁 Key Benefits

⁠1. Blazingly Fast

• 5.28M operations/second in embedded mode (cached reads)
• 215k ops/sec for storage operations (66x faster than network)
• 32M deletions/second (320x faster than Redis)

⁠2. TOON Format Efficiency

• ~40% fewer tokens compared to JSON storage
• Perfect for AI/LLM applications - lower costs, faster processing
• Human-readable - easy to inspect and debug stored data
• Schema-aware - built-in structure validation

⁠3. Data Persistence

• All data stored on disk using efficient TOON format
• Survives restarts and crashes
• Memory-mapped I/O for fast disk access

⁠4. Redis Compatible

• Use any Redis client library (50+ languages supported)
• Familiar commands: GET, SET, DEL, EXISTS, KEYS
• Drop-in replacement for Redis in many use cases

⁠5. Dual Mode Operation

Network Mode:

// Connect from Node.js, Python, Go, etc.
const redis = require('redis');
const client = redis.createClient({ url: 'redis://localhost:6379' });
await client.set('key', 'value');

Embedded Mode:

// Direct Rust integration (66x faster!)
let cache = ToonCache::new("./data", 10000)?;
let id = cache.put(b"data")?;
let data = cache.get(id)?;

⁠6. Built-in LRU Cache

• Automatic caching of hot data in RAM
• 10,000 item default capacity (configurable)
• No manual cache management needed

⁠7. Easy Deployment

• Single binary, no dependencies
• Docker images available
• Works on Linux, Windows, macOS
• Cross-platform (amd64 and arm64)

⁠📦 How is Data Saved?

ToonStore saves all data persistently on disk using the efficient TOON format:

⁠Data Storage Format

Data is stored in TOON (Token-Oriented Object Notation) format, which provides:

• ~40% fewer tokens compared to JSON
• Human-readable format for easy inspection
• LLM-optimized for better AI comprehension
• Persistent - survives restarts and crashes

⁠Storage Location

By default, ToonStore saves data to:

./data/        # Data directory
├── toon.db    # Main database file (TOON format)
└── index.db   # Index mappings

⁠Example: JSON vs TOON

When you save this data:

{
  "id": "user:1",
  "name": "John Doe",
  "email": "[email protected]",
  "age": 30
}

ToonStore converts and stores it in TOON format:

user:1{id,name,email,age}:
  user:1,John Doe,[email protected],30

This format is:

• ✅ Persistent - Written to disk immediately
• ✅ Efficient - Smaller file size (~40% reduction)
• ✅ Fast - Quick to parse and serialize
• ✅ Cached - Hot data kept in memory (LRU cache)

⁠Data Persistence Guarantees

ToonStore provides:

1. Automatic Persistence

   • Every SET operation writes to disk
   • No manual save/flush required
   • Data survives process restarts

2. Memory-Mapped I/O

   • Fast disk access via OS-level caching
   • Automatic synchronization
   • Efficient memory usage

3. Built-in LRU Cache

   • 10,000 item default capacity
   • 5.28M ops/sec for cached reads
   • Automatic cache invalidation

⁠Configuring Storage

# Specify data directory
tstd --data /path/to/data

# Configure cache size
tstd --capacity 50000

⁠Checking Your Data

# View stored data files
ls -lh ./data/

# Check database size
redis-cli DBSIZE

# Get server info
redis-cli INFO

⁠Why TOON Format?

In the age of AI and LLMs, the TOON format provides significant advantages:

1. Cost Savings - ~40% fewer tokens = 40% lower API costs
2. Context Efficiency - More data fits in LLM context windows
3. Better Comprehension - 74% accuracy vs JSON's 70% in LLM benchmarks
4. Schema-Aware - Explicit structure helps LLMs parse reliably

Learn more about TOON: TOON Format Repository⁠

⁠📊 Performance Comparison

ToonStore is designed for speed:

Key Insight: Embedded mode is 66x faster than network mode (no TCP overhead)

See BENCHMARKS.md⁠ for detailed benchmarks and methodology.

⁠🚀 Quick Start

⁠Option 1: Docker (Easiest - Recommended)

# Pull from Docker Hub
docker pull samso9th/toonstore:latest

# Or pull from GitHub Container Registry
docker pull ghcr.io/kalama-tech/toonstoredb:latest

# Run ToonStore
docker run -d \
  --name toonstore \
  -p 6379:6379 \
  -v toonstore_data:/data \
  samso9th/toonstore:latest

# Test connection
redis-cli -h 127.0.0.1 -p 6379 PING
# Output: PONG

# Use it
redis-cli -h 127.0.0.1 -p 6379
127.0.0.1:6379> SET mykey "Hello World"
OK
127.0.0.1:6379> GET mykey
"Hello World"

With Docker Compose:

# Download docker-compose.yml
curl -O https://raw.githubusercontent.com/Kalama-Tech/toonstoredb/main/docker-compose.yml

# Start
docker-compose up -d

# Stop
docker-compose down

⁠Option 2: Pre-built Binary (Coming Soon)

# Linux/macOS
curl -L https://github.com/Kalama-Tech/toonstoredb/releases/latest/download/tstd -o tstd
chmod +x tstd
./tstd --bind 0.0.0.0:6379

# Windows
# Download from: https://github.com/Kalama-Tech/toonstoredb/releases

⁠Option 3: Build from Source

# Clone repository
git clone https://github.com/Kalama-Tech/toonstoredb.git
cd toonstoredb

# Build (requires Rust 1.70+)
cargo build --release

# Run
./target/release/tstd --bind 0.0.0.0:6379

⁠Option 4: Embedded Library (Rust Only)

# Cargo.toml
[dependencies]
tooncache = { git = "https://github.com/Kalama-Tech/toonstoredb" }

use tooncache::ToonCache;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Open database
    let cache = ToonCache::new("./data", 10000)?;
    
    // Store data
    let id = cache.put(b"Hello, World!")?;
    
    // Retrieve data
    let data = cache.get(id)?;
    println!("Retrieved: {:?}", String::from_utf8(data)?);
    
    Ok(())
}

⁠🔌 Connect from Your Application

ToonStore is Redis-compatible, so you can use any Redis client library:

⁠Node.js

const redis = require('redis');
const client = redis.createClient({ url: 'toonstore://localhost:6379' });

await client.connect();
await client.set('user:1', 'John Doe');
const user = await client.get('user:1');
console.log(user); // "John Doe"

⁠Python

import redis

client = redis.from_url('toonstore://localhost:6379')
client.set('user:1', 'John Doe')
user = client.get('user:1')
print(user)  # b'John Doe'

⁠Go

import "github.com/redis/go-redis/v9"

client := redis.NewClient(&redis.Options{
    Addr: "localhost:6379",
})

client.Set(ctx, "user:1", "John Doe", 0)
user, _ := client.Get(ctx, "user:1").Result()
fmt.Println(user)  // "John Doe"

See docs/connecting-from-apps.md⁠ for more examples.

⁠✨ Features

⁠Core Features (v0.1)

• ✅ Dual Mode: Network (Redis-compatible) or Embedded (5.28M ops/sec)
• ✅ TOON Format: Token-efficient storage (~40% savings vs JSON) designed for AI/LLMs
• ✅ LRU Cache: Automatic caching with 5.28M ops/sec cached reads
• ✅ RESP Protocol: Works with any Redis client library
• ✅ Memory-Mapped I/O: Fast disk access with OS-level caching
• ✅ Cross-Platform: Linux, Windows, macOS
• ✅ Docker Ready: Official images on Docker Hub

⁠Commands Supported (v0.1)

PING, ECHO          - Connection testing
GET, SET, DEL       - Core operations  
EXISTS, KEYS        - Key inspection
DBSIZE, FLUSHDB     - Database management
INFO                - Server statistics

⁠📚 Documentation

⁠Getting Started

• Quick Start Guide⁠ - Your first ToonStore app
• Installation⁠ - All installation methods
• Docker Deployment⁠ - Complete Docker guide
• Connection Guide⁠ - Network vs Embedded mode
• Architecture⁠ - 3-layer architecture & connection strings

⁠Usage Guides

• Rust API Reference⁠ - Embedded library usage
• RESP Server Guide⁠ - Network server setup
• Configuration⁠ - Server & cache tuning

⁠Advanced

• TOON Format⁠ - Storage format specification
• TOON Format⁠ - Storage format specification
• Performance Tuning⁠ - Optimization guide
• Benchmarks⁠ - Detailed performance data

⁠Deployment

• Docker Guide⁠ - Container deployment
• Production Checklist⁠ - Before going live
• Monitoring⁠ - Health checks & metrics

⁠🏗️ Architecture

┌────────────────────────────────────────────────────────────────┐
│                      Application                                │
└─────────────────┬──────────────────────┬───────────────────────┘
                  │                      │
        Network Mode (tstd)    Embedded Mode (library)
        redis://host:port      ToonCache::new()
        ~70k ops/sec           5.28M ops/sec
                  │                      │
                  └──────────┬───────────┘
                             ↓
              ┌──────────────────────────────────┐
              │  tooncache (LRU Cache)           │
              │  - 5.28M ops/sec (cached reads)  │
              │  - Configurable capacity         │
              └──────────────┬───────────────────┘
                             ↓
              ┌──────────────────────────────────┐
              │  toonstoredb (Storage Engine)    │
              │  - 215k ops/sec (storage reads)  │
              │  - TOON format parser            │
              │  - Memory-mapped files           │
              └──────────────────────────────────┘

⁠🎯 Use Cases

⁠✅ Ideal For:

• 🤖 AI/LLM applications - Token-efficient storage for embeddings, prompts, and context
• 🚀 High-performance caching (5.28M ops/sec!)
• 📦 Embedded databases in Rust applications
• 🔄 Redis replacement with better performance and efficiency
• 💾 Key-value storage with persistence and TOON format
• ⚡ In-process caching with disk backup
• 💰 Cost-sensitive applications - Reduce storage and transmission costs by ~40%

⁠❌ Not Suitable For (v0.1):

• 🔐 ACID transactions
• 🔗 Complex queries / JOINs
• 🌐 Multi-node clustering
• 🔒 Strong consistency guarantees

⁠📦 Installation

⁠From Source (Recommended for v0.1)

# Clone repository
git clone https://github.com/Kalama-Tech/toonstoredb
cd toonstoredb

# Build release
cargo build --release

# Run server
./target/release/tstd --bind 0.0.0.0:6379

⁠Using Cargo (Coming Soon)

# Install server binary
cargo install tstd

# Add to your Rust project
cargo add tooncache

⁠Docker

# Pull image
docker pull ghcr.io/yourusername/toonstore:latest

# Run server
docker run -d \
  -p 6379:6379 \
  -v $(pwd)/data:/data \
  ghcr.io/yourusername/toonstore:latest

See DOCKER_SETUP_GUIDE.md⁠ for complete Docker setup.

⁠🔌 Language Support

⁠Network Mode (Any Language via Redis Protocol)

Python

import redis
client = redis.from_url('toonstore://localhost:6379')
client.set('key', 'value')

Node.js

const Redis = require('ioredis');
const client = new Redis('toonstore://localhost:6379');
await client.set('key', 'value');

Go

import "github.com/redis/go-redis/v9"
client := redis.NewClient(&redis.Options{Addr: "localhost:6379"})
client.Set(ctx, "key", "value", 0)

⁠Embedded Mode (Direct Library)

Rust

use tooncache::ToonCache;
let cache = ToonCache::new("./data", 10000)?;

Python (Coming Week 4)

import toonstore
db = toonstore.ToonCache("./data", capacity=10000)

⁠🛠️ Configuration

⁠Server Options

tstd \
  --bind 0.0.0.0:6379 \       # Bind address
  --data ./data \              # Data directory
  --capacity 10000             # Cache capacity

⁠Environment Variables

RUST_LOG=info    # Logging level (info, debug, trace)

⁠Embedded Configuration

let cache = ToonCache::new(
    "./data",    // Data directory
    10000        // Cache capacity
)?;

⁠📈 Monitoring

⁠Check Server Status

# Connect with redis-cli
redis-cli -h 127.0.0.1 -p 6379

# Get statistics
127.0.0.1:6379> INFO
# Server
toonstore_version:0.1.0

# Stats
total_keys:1000
cache_size:850
cache_capacity:10000
cache_hits:95000
cache_misses:5000
cache_hit_ratio:0.95

# Check database size
127.0.0.1:6379> DBSIZE
(integer) 1000

⁠Health Check

# Docker health check
tstd --health

# Or via TCP
redis-cli PING
PONG

⁠🐛 Troubleshooting

⁠Server won't start

# Check if port is already in use
netstat -an | grep 6379

# Try different port
tstd --bind 127.0.0.1:6380

⁠Connection refused

# Check server is running
ps aux | grep tstd

# Check firewall
sudo ufw allow 6379/tcp

⁠Low performance

# Increase cache capacity
tstd --capacity 50000

# Check cache hit ratio
redis-cli INFO | grep cache_hit_ratio

# Use embedded mode for maximum performance

See docs/troubleshooting.md⁠ for more solutions.

⁠🗺️ Roadmap

⁠v0.1 (Current - Weeks 1-3) ✅

• Storage engine (toonstoredb)
• LRU cache (tooncache)
• RESP server (tstd)
• Basic benchmarks
• Docker support

⁠v0.2 (Week 4) 🚧

• Python bindings (PyO3)
• npm package (Neon)
• Complete documentation
• PyPI + npm publish

⁠v0.3 (Future)

• WAL for durability
• Transactions
• Replication
• More RESP commands
• Clustering support

⁠🤝 Contributing

ToonStore is in active development. We welcome contributions!

⁠Development Setup

# Clone
git clone https://github.com/Kalama-Tech/toonstoredb
cd toonstoredb

# Build
cargo build

# Run tests
cargo test

# Run benchmarks
cargo bench

⁠Guidelines

• Write tests for new features
• Run cargo fmt and cargo clippy
• Update documentation
• Follow existing code style

See CONTRIBUTING.md⁠ for details.

⁠📜 License

ToonStore is licensed under the MIT License⁠.

⁠🙏 Acknowledgments

• Built with Rust⁠
• TOON format inspired by toondb⁠
• RESP protocol compatible with Redis⁠

⁠📞 Support & Community

• Issues: GitHub Issues⁠
• Discussions: GitHub Discussions⁠
• Documentation: docs/⁠

⁠⚡ Quick Links

• Getting Started⁠
• API Documentation⁠
• Benchmarks⁠
• Connection Guide⁠
• Docker Setup⁠
• Architecture⁠

Built with ❤️ in Rust | Performance: 5.28M ops/sec | License: MIT