FaultMaven Knowledge Base - ChromaDB-powered RAG system for documentation
2.3K
Part of FaultMaven — The AI-Powered Troubleshooting Copilot
FaultMaven Knowledge Management Microservice - Open source RAG-powered knowledge base for troubleshooting documentation.
The Knowledge Service implements a Retrieval-Augmented Generation (RAG) system for FaultMaven, allowing users to upload and search through troubleshooting documentation. Documents are chunked, embedded using BGE-M3 embeddings, and stored in a vector database for fast semantic search.
Features:
The service uses a provider pattern for vector database abstraction:
| Provider | Use Case | Scale | Configuration |
|---|---|---|---|
| ChromaDB (default) | Laptop/dev, self-hosted | ~100K documents | Embedded SQLite backend |
| Pinecone | Production, enterprise | Billions of documents | Managed cloud service |
Benefits:
# Run with persistent storage
docker run -d -p 8004:8004 \
-v ./data/chromadb:/data/chromadb \
-v ./data/sqlite:/data/sqlite \
faultmaven/fm-knowledge-service:latest
The service will be available at http://localhost:8004.
See faultmaven-deploy for complete deployment with all FaultMaven services.
# Clone repository
git clone https://github.com/FaultMaven/fm-knowledge-service.git
cd fm-knowledge-service
# Create virtual environment
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install dependencies
pip install -e .
# Run service
uvicorn knowledge_service.main:app --reload --port 8004
| Method | Endpoint | Description |
|---|---|---|
| POST | /api/v1/documents/upload | Upload document (multipart/form-data) |
| GET | /api/v1/documents | List user's documents |
| GET | /api/v1/documents/{document_id} | Get document metadata |
| DELETE | /api/v1/documents/{document_id} | Delete document |
| Method | Endpoint | Description |
|---|---|---|
| POST | /api/v1/search | Semantic search across documents |
| POST | /api/v1/search/hybrid | Hybrid full-text + vector search |
| Method | Endpoint | Description |
|---|---|---|
| GET | /health | Health check |
Configuration via environment variables:
| Variable | Description | Default |
|---|---|---|
SERVICE_NAME | Service identifier | fm-knowledge-service |
ENVIRONMENT | Deployment environment | development |
PORT | Service port | 8004 |
LOG_LEVEL | Logging level | INFO |
| Variable | Description | Default |
|---|---|---|
DATABASE_URL | Database connection string | sqlite+aiosqlite:////data/sqlite/fm_knowledge.db |
Supported databases:
sqlite+aiosqlite:////data/sqlite/fm_knowledge.dbpostgresql+asyncpg://user:pass@host:5432/faultmaven| Variable | Description | Default |
|---|---|---|
VECTOR_DB_PROVIDER | Vector database provider (chroma or pinecone) | chroma |
| Variable | Description | Default |
|---|---|---|
CHROMA_HOST | ChromaDB server host | localhost |
CHROMA_PORT | ChromaDB server port | 8007 |
CHROMADB_PATH | ChromaDB data directory (legacy) | /data/chromadb |
| Variable | Description | Default |
|---|---|---|
PINECONE_API_KEY | Pinecone API key | (required) |
PINECONE_ENVIRONMENT | Pinecone environment (e.g., us-east-1) | (required) |
PINECONE_INDEX_NAME | Pinecone index name | faultmaven-knowledge |
| Variable | Description | Default |
|---|---|---|
EMBEDDING_MODEL | Embeddings model name | BAAI/bge-m3 |
CHUNK_SIZE | Text chunk size | 1000 |
CHUNK_OVERLAP | Chunk overlap size | 200 |
MAX_UPLOAD_SIZE_MB | Maximum file size | 10 |
Development (ChromaDB):
VECTOR_DB_PROVIDER=chroma
CHROMA_HOST=localhost
CHROMA_PORT=8007
DATABASE_URL=sqlite+aiosqlite:////data/sqlite/fm_knowledge.db
Production (Pinecone + PostgreSQL):
VECTOR_DB_PROVIDER=pinecone
PINECONE_API_KEY=your-api-key
PINECONE_ENVIRONMENT=us-east-1
PINECONE_INDEX_NAME=faultmaven-production-kb
DATABASE_URL=postgresql+asyncpg://user:pass@postgres:5432/faultmaven
Upload documents via multipart/form-data:
curl -X POST http://localhost:8004/api/v1/documents/upload \
-H "X-User-ID: user_123" \
-F "file=@troubleshooting_guide.pdf" \
-F "title=Database Troubleshooting Guide" \
-F "description=Common database issues and solutions" \
-F "tags=database,performance,errors"
Response:
{
"document_id": "doc_abc123",
"user_id": "user_123",
"filename": "troubleshooting_guide.pdf",
"title": "Database Troubleshooting Guide",
"description": "Common database issues and solutions",
"file_type": "pdf",
"file_size": 245678,
"chunk_count": 42,
"tags": ["database", "performance", "errors"],
"created_at": "2025-11-16T10:30:00Z"
}
Search documents using natural language queries:
curl -X POST http://localhost:8004/api/v1/search \
-H "X-User-ID: user_123" \
-H "Content-Type: application/json" \
-d '{
"query": "How to fix database connection timeouts?",
"limit": 5,
"min_relevance": 0.7
}'
Response:
{
"results": [
{
"chunk_id": "chunk_001",
"document_id": "doc_abc123",
"document_title": "Database Troubleshooting Guide",
"content": "Connection timeouts typically occur when...",
"relevance_score": 0.92,
"metadata": {
"page": 15,
"section": "Connection Issues"
}
}
],
"query": "How to fix database connection timeouts?",
"total_results": 5
}
| Extension | Format | Processing |
|---|---|---|
.txt | Plain text | Direct chunking |
.md | Markdown | Direct chunking |
.pdf | PyPDF2 extraction | |
.doc, .docx | Word | python-docx extraction |
.rtf | Rich Text | striprtf extraction |
{
"document_id": str, # Unique identifier
"user_id": str, # Owner user ID
"filename": str, # Original filename
"title": str, # Document title
"description": str, # Optional description
"file_type": str, # File extension
"file_size": int, # Size in bytes
"chunk_count": int, # Number of chunks
"tags": List[str], # Searchable tags
"created_at": datetime, # Upload timestamp
"updated_at": datetime # Last modification
}
{
"chunk_id": str, # Unique chunk identifier
"document_id": str, # Parent document
"content": str, # Chunk text
"embedding": List[float], # BGE-M3 vector (1024-dim)
"metadata": {
"user_id": str,
"document_title": str,
"chunk_index": int,
"file_type": str,
"tags": List[str]
}
}
This service uses trusted header authentication from the FaultMaven API Gateway:
X-User-ID (required): Identifies the user making the requestX-User-Email (optional): User's email addressX-User-Roles (optional): User's rolesAll document operations are scoped to the user specified in X-User-ID. Users can only access their own documents.
Important: This service should run behind the fm-api-gateway which handles authentication and sets these headers. Never expose this service directly to the internet.
┌─────────────────┐
│ API Gateway │ (Handles authentication)
└────────┬────────┘
│ X-User-ID header
↓
┌─────────────────┐
│ Knowledge Svc │ (Document processing)
└────┬───────┬────┘
│ │
↓ ↓
┌─────────┐ ┌──────────────┐
│ SQLite │ │ ChromaDB │
│Metadata │ │Vector Store │
└─────────┘ └──────────────┘
# Run all tests
pytest
# Run with coverage
pytest --cov=knowledge_service
# Run specific test file
pytest tests/test_documents.py -v
Apache 2.0 - See LICENSE for details.
See our Contributing Guide for detailed guidelines.
Content type
Image
Digest
sha256:6773cd317…
Size
687 MB
Last updated
10 months ago
docker pull faultmaven/fm-knowledge-service