wolfgarbe/seekstorm_server repository overview

Github⁠ | Website⁠ | Benchmark⁠ | Demo⁠ | Library Docs⁠ | Server Docs⁠ | Server Readme⁠ | Roadmap⁠ | Blog⁠ | Twitter⁠

⁠docker run -ti -p "8000:80" -v "$(pwd)/seekstorm_storage:/seekstorm_index:z" wolfgarbe/seekstorm_server

⁠REST API docs⁠, see master API key in console or in docker desktop / view details / logs .

SeekStorm is an open-source, sub-millisecond lexical & vector search library & multi-tenancy server implemented in Rust.

Development started in 2015, in production⁠ since 2020, Rust port in 2023, open sourced in 2024, work in progress.

SeekStorm is open source licensed under the Apache License 2.0⁠

Blog Posts: SeekStorm is now Open Source⁠ and SeekStorm gets Faceted search, Geo proximity search, Result sorting⁠

⁠SeekStorm high-performance search library

• Full-text lexical search
• Vector search
• True real-time search, with negligible performance impact
• Incremental indexing
• Multithreaded indexing & search
• Unlimited field number, field length & index size
• Compressed document store: ZStandard
• Boolean queries: AND, OR, PHRASE, NOT
• BM25F and BM25F_Proximity ranking
• Field filtering
• Faceted search⁠: Counting & filtering of String & Numeric range facets (with Histogram/Bucket & Min/Max aggregation)
• Result sorting by any field, ascending or descending, multiple fields combined by "tie-breaking".
• Geo proximity search, filtering and sorting.
• Tokenizer with Chinese word segmentation
• KWIC snippets, highlighting
• One-way and multi-way synonyms
• Billion-scale index
• Language independent
• API keys
• RESTful API with CORS
• Index either in RAM or memory mapped files
• Cross-platform (Windows, Linux, MacOS)
• SIMD (Single Instruction, Multiple Data) hardware acceleration support,
  both for x86-64 (AMD64 and Intel 64) and AArch64 (ARM, Apple Silicon).
• Single-machine scalability: serving thousands of concurrent queries with low latency from a single commodity server without needing clusters or proprietary hardware accelerators.

⁠SeekStorm uses two separate, first-class, native index architectures, under one roof.

• Lexical search: sharded and leveled inverted index.
• Vector search: sharded and leveled IVF index for ANN or exhaustive search.
• Shared document store, shared document ID space.
• Both first-class engines are integrated at the query planner level.
• Query planner with QueryModes (Lexical, Vector, Hybrid…) and FusionTypes (RRF, …).

⁠Vector Features

• Multi-Vector indexing: both from multiple fields and from multiple chunks per field.
• Integrated inference from any text document field or Import external embeddings.
• Variable dimensions and precisions: f32, i8.
• TurboQuant (TQ) and affine Scalar Quantization (SQ).
• Multiple similarity measures: Cosine similarity, Dot product, Euclidean distance.
• Chunking that respects sentence boundaries and Unicode segmentation for multilingual text.
• K-medoid clustering with actual data points as centers.
• Field filters are active during vector search, not just as post-search filtering step.
• True real-time search
• disk-based billion-scale vector search
• Sub-millisecond search for 1 million vectors on a laptop, CPU only:
  • Sift1M recall@10=95%, 0.2 ms | recall@10=99%, 0.3 ms

⁠Query types

• OR disjunction union
• AND conjunction intersection
• "" phrase
• - NOT

⁠Result types

• TopK
• Count
• TopKCount

⁠SeekStorm multi-tenancy search server

• Index and search via RESTful API⁠.
• Ingest local data files in JSON⁠, Newline-delimited JSON⁠ (ndjson), and Concatenated JSON⁠ formats via console command.
• Ingest local PDF files via console command (single file or all files in a directory).
• Multi-tenancy index management.
• API-key management.
• Embedded web server and web UI⁠ to search and display results from any index without coding.
• Code first OpenAPI generated REST API documentation⁠
• Cross-platform: runs on Linux, Windows, and macOS (other OS untested).