bigsk1/voice-chat-ai

By bigsk1

•Updated 2 days ago

Speak with AI - Run locally using Ollama, OpenAI or xAI - Speech uses XTTS, OpenAI or ElevenLabs

Image

Machine learning & AI

10K+

Overview Tags

bigsk1/voice-chat-ai repository overview

⁠Voice Chat AI 🎙️

Voice Chat AI allows you to interact with AI characters using speech. Choose between various characters, each with unique personalities and voices. Run locally with Ollama, or use cloud providers like OpenAI, Anthropic, or xAI.

Mix and match: Use ElevenLabs voices with Ollama models, OpenAI TTS with Anthropic chat, or run completely locally with Spark-TTS voice cloning.

WebRTC Real-Time: Experience instant conversations with OpenAI's Realtime API - interrupt the AI and get immediate responses.

Full Documentation: https://github.com/bigsk1/voice-chat-ai⁠

⁠✨ Features

Multiple LLM Providers: OpenAI, xAI, Anthropic, or Ollama (local)
Flexible TTS Options:
- Spark-TTS (local zero-shot voice cloning)
- OpenAI TTS (cloud, emotional voices)
- ElevenLabs (premium cloud voices)
- Kokoro TTS (fast, local)
OpenAI Realtime API: Real-time speech-to-speech conversations
Enhanced Mode: Emotional, expressive AI responses
Local or Cloud: Run entirely locally or mix local/cloud services
50+ Characters: Pre-built personalities, games, and story modes
Interactive Experiences: 15+ game types, escape rooms, trivia, word puzzles
WebUI + CLI: Full web interface or terminal mode
Mood Analysis: Sentiment-aware AI responses

⁠🐳 Docker Quick Start

⁠CPU Version (Recommended)

Windows with WSL2:

docker run -d \
  -e "PULSE_SERVER=/mnt/wslg/PulseServer" \
  -v \\wsl$\Ubuntu\mnt\wslg:/mnt/wslg/ \
  -v ./elevenlabs_voices.json:/app/elevenlabs_voices.json \
  --env-file .env \
  --name voice-chat-ai \
  -p 8000:8000 \
  bigsk1/voice-chat-ai:latest

Linux/WSL2 Ubuntu:

docker run -d \
  -e "PULSE_SERVER=/mnt/wslg/PulseServer" \
  -v /mnt/wslg/:/mnt/wslg/ \
  -v ./elevenlabs_voices.json:/app/elevenlabs_voices.json \
  --env-file .env \
  --name voice-chat-ai \
  -p 8000:8000 \
  bigsk1/voice-chat-ai:latest

Native Linux (PulseAudio):

docker run -d \
  -e PULSE_SERVER=unix:/tmp/pulse/native \
  -v ~/.config/pulse/cookie:/root/.config/pulse/cookie:ro \
  -v /run/user/$(id -u)/pulse:/tmp/pulse:ro \
  -v ./elevenlabs_voices.json:/app/elevenlabs_voices.json \
  --env-file .env \
  --name voice-chat-ai \
  -p 8000:8000 \
  bigsk1/voice-chat-ai:latest

⁠CUDA Version (GPU)

Requirements: NVIDIA GPU, nvidia-container-toolkit, CUDA drivers

Windows with WSL2:

docker run -d --gpus all \
  -e "PULSE_SERVER=/mnt/wslg/PulseServer" \
  -v \\wsl$\Ubuntu\mnt\wslg:/mnt/wslg/ \
  -v ./elevenlabs_voices.json:/app/elevenlabs_voices.json \
  --env-file .env \
  --name voice-chat-ai-cuda \
  -p 8000:8000 \
  bigsk1/voice-chat-ai:cuda

Linux/WSL2 Ubuntu:

docker run -d --gpus all \
  -e "PULSE_SERVER=/mnt/wslg/PulseServer" \
  -v /mnt/wslg/:/mnt/wslg/ \
  -v ./elevenlabs_voices.json:/app/elevenlabs_voices.json \
  --env-file .env \
  --name voice-chat-ai-cuda \
  -p 8000:8000 \
  bigsk1/voice-chat-ai:cuda

⁠Docker Compose

CPU:

services:
  voice-chat-ai:
    image: bigsk1/voice-chat-ai:latest
    container_name: voice-chat-ai
    ports:
      - "8000:8000"
    env_file:
      - .env
    environment:
      - PULSE_SERVER=/mnt/wslg/PulseServer
    volumes:
      - /mnt/wslg/:/mnt/wslg/
      - ./elevenlabs_voices.json:/app/elevenlabs_voices.json
    restart: unless-stopped

CUDA:

services:
  voice-chat-ai-cuda:
    image: bigsk1/voice-chat-ai:cuda
    container_name: voice-chat-ai-cuda
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ports:
      - "8000:8000"
    env_file:
      - .env
    environment:
      - PULSE_SERVER=/mnt/wslg/PulseServer
    volumes:
      - /mnt/wslg/:/mnt/wslg/
      - ./elevenlabs_voices.json:/app/elevenlabs_voices.json
    restart: unless-stopped

⁠⚙️ Configuration

Create a .env file with your settings:

# LLM Provider (ollama, openai, anthropic, xai)
MODEL_PROVIDER=ollama
OLLAMA_MODEL=llama3:latest
OLLAMA_BASE_URL=http://localhost:11434

# OpenAI (optional)
OPENAI_API_KEY=sk-...
OPENAI_MODEL=gpt-4o-mini

# Anthropic (optional)
ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_MODEL=claude-3-5-sonnet-20241022

# xAI (optional)
XAI_API_KEY=xai-...
XAI_MODEL=grok-2-latest

# TTS Provider (sparktts, openai, elevenlabs, kokoro)
TTS_PROVIDER=openai

# ElevenLabs (optional)
ELEVENLABS_API_KEY=...

# Transcription (openai or local)
TRANSCRIPTION_PROVIDER=openai
FASTER_WHISPER_LOCAL=false

# Default character
CHARACTER_NAME=samantha

Access: http://localhost:8000⁠

⁠🎮 Interactive Modes

⁠Story Mode

Experience immersive narrative adventures with AI characters. Each story has unique plot lines, choices, and outcomes.

⁠Game Mode

Play interactive games with specialized game master characters:

Word Puzzles: Anagrams, rhymes, word chains
Trivia: General knowledge, science, history
Escape Rooms: Solve puzzles to escape
Detective Mysteries: Gather clues and solve cases
And more: 15+ game types

⁠🚀 Installing Spark-TTS (Optional)

For local zero-shot voice cloning inside the container:

# Enter container
docker exec -it voice-chat-ai bash

# Run setup script (GPU recommended)
python setup_sparktts.py

# Restart container
exit
docker restart voice-chat-ai

Set TTS_PROVIDER=sparktts in your .env file.

Model size: ~2GB download (Hugging Face)

⁠📋 Available Tags

latest - Latest stable release (CPU, Python 3.11-slim)
cuda - CUDA 12.4 support for GPU acceleration
v1.0.0 - Specific version tags
main - Development branch (may be unstable)

⁠🔧 Environment Variables

Variable	Options	Default	Description
`MODEL_PROVIDER`	`ollama`, `openai`, `anthropic`, `xai`	`ollama`	LLM provider
`TTS_PROVIDER`	`sparktts`, `openai`, `elevenlabs`, `kokoro`	`openai`	Text-to-speech service
`TRANSCRIPTION_PROVIDER`	`openai`, `local`	`openai`	Speech-to-text service
`CHARACTER_NAME`	Any character folder name	`samantha`	Default character
`FASTER_WHISPER_LOCAL`	`true`, `false`	`false`	Use local Whisper for transcription
`OLLAMA_BASE_URL`	URL	`http://localhost:11434`	Ollama API endpoint

Full configuration guide: https://github.com/bigsk1/voice-chat-ai#configuration⁠

⁠📚 Documentation

Installation Guide: https://github.com/bigsk1/voice-chat-ai/blob/main/INSTALL.md⁠
Requirements Files: https://github.com/bigsk1/voice-chat-ai/blob/main/README_REQUIREMENTS.md⁠
Spark-TTS Setup: https://github.com/bigsk1/voice-chat-ai/blob/main/docs/SPARKTTS.md⁠
Games Documentation: https://github.com/bigsk1/voice-chat-ai/blob/main/docs/games.md⁠
Stories Documentation: https://github.com/bigsk1/voice-chat-ai/blob/main/docs/stories.md⁠

⁠🐧 Native Installation (Recommended)

For better performance and audio support, we recommend native installation over Docker:

⁠With pip:

git clone https://github.com/bigsk1/voice-chat-ai.git
cd voice-chat-ai
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install PyTorch (choose CPU or CUDA)
pip install torch torchaudio torchvision --index-url https://download.pytorch.org/whl/cpu

# Install dependencies
pip install -r requirements.txt

⁠With uv (faster):

git clone https://github.com/bigsk1/voice-chat-ai.git
cd voice-chat-ai
uv sync  # Creates venv and installs everything

Start: uvicorn app.main:app --host 0.0.0.0 --port 8000

⁠🎯 Use Cases

Personal AI Assistant: Talk naturally with AI for tasks, questions, or brainstorming
Language Practice: Converse with AI characters in different languages
Entertainment: Role-play scenarios, play games, or experience stories
Education: Interactive learning with AI tutors and quiz masters
Voice Cloning: Clone any voice with a short audio sample (Spark-TTS)
Development: Build voice-enabled AI applications on this framework

⁠🛠️ Troubleshooting

⁠Audio Issues (WSL2)

If microphone doesn't work in WSL2:

# In WSL2
sudo apt-get update
sudo apt-get install pulseaudio
pulseaudio --start

⁠CUDA Not Detected

Verify GPU access in container:

docker exec -it voice-chat-ai-cuda nvidia-smi

⁠Model Download Errors

If Spark-TTS or Whisper models fail to download, check:

Internet connection
Hugging Face availability
Disk space (models can be 1-3GB each)

More troubleshooting: https://github.com/bigsk1/voice-chat-ai#troubleshooting⁠

⁠📦 What's New in v1.0.0

🚨 Breaking Changes:

Removed XTTS (replaced with Spark-TTS)
Requires Python 3.11+
New installation methods (pip, uv, conda)

✨ New Features:

Spark-TTS for faster, better local voice cloning
pyproject.toml for modern dependency management
Leaner Docker images (Python 3.11-slim base)
Improved documentation with detailed install guide

Upgrade Guide: https://github.com/bigsk1/voice-chat-ai/releases/tag/v1.0.0⁠

⁠🤝 Contributing

Contributions welcome! Please read our contributing guidelines and submit pull requests.

Areas for contribution:

New character personalities
Additional game types
TTS provider integrations
UI/UX improvements
Documentation updates

⁠📄 License

MIT License - See LICENSE⁠

⁠🔗 Links

GitHub: https://github.com/bigsk1/voice-chat-ai⁠
Issues: https://github.com/bigsk1/voice-chat-ai/issues⁠
Docker Hub: https://hub.docker.com/r/bigsk1/voice-chat-ai⁠
Author: bigsk1⁠ | bigsk1.com⁠

⁠⭐ Show Your Support

If you find this project useful, please consider:

⭐ Starring the GitHub repository
🐛 Reporting bugs or requesting features
🤝 Contributing code or documentation
📢 Sharing with others who might enjoy it

Made with ❤️ by bigsk1⁠

Tag summary

Recent tags

Content type

Image

Digest

sha256:c47d2a995…

Size

852 MB

Last updated

2 days ago

docker pull bigsk1/voice-chat-ai