hivehub/vectorizer repository overview

⁠Vectorizer

?? Docker Hub: https://hub.docker.com/r/hivehub/vectorizer⁠

A high-performance vector database and search engine built in Rust, designed for semantic search, document indexing, and AI-powered applications. The v3.x line ships a binary RPC transport (MessagePack over TCP, port 15503) as the recommended primary channel alongside REST + MCP on 15002.

v3.6.1 highlights:

• Fixed - the arm64 -fastembed image now starts. hivehub/vectorizer:3.5.0-fastembed and 3.4.0-fastembed exit 127 on linux/arm64 with error while loading shared libraries: libstdc++.so.6: the runtime stage copied an amd64 library into the arm64 image, and cross-linking succeeded because the linker reads a different path than the runtime does. The slim default image was never affected - it does not load that library. If you run the dense variant on arm64, 3.6.1 is the first working release.
• Fixed - a collection listing during startup no longer looks like data loss. The server loads collections in the background, so GET /collections could answer with a partial list whose total_collections agreed with it - a 181-collection store reporting "11 collections, total 11" seconds after boot. The response now carries loading, loaded_collections and expected_collections, and a new GET /ready answers 503 + Retry-After until the catalog is in. Gate your orchestrator on /ready; /health stays a liveness check and keeps answering 200 during warm-up on purpose.
• Fixed - the Rust SDK's REST client rejects a vectorizer:// URL up front instead of failing later inside reqwest, and points you at RpcClient::connect_url.

Full 3.6.0 and 3.6.1 notes are in the changelog linked below.

Earlier in v3.x: 3.6.0 added collection TTL with read-back and full RPC parity; 3.5.0 fixed BM25 vocabulary loss across restarts, made the WAL fsync and checksum its records, removed a bulk_update_metadata deadlock, and unblocked search during batch inserts; 3.4.0 made /data the canonical volume mount.

Full changelog: CHANGELOG.md⁠.

⁠?? Quick Start

⁠Basic Usage

docker run -d \
  --name vectorizer \
  -p 15002:15002 \
  -p 15503:15503 \
  --restart unless-stopped \
  hivehub/vectorizer:3.6.1

First boot creates an admin user and writes credentials to /data/.root_credentials inside the container (read with docker exec + cat or docker cp - the image is distroless so there's no shell). Rotate via the dashboard or /auth API as soon as you've copied them.

⁠With Persistent Data (v3.4.0+ canonical layout)

docker run -d \
  --name vectorizer \
  -p 15002:15002 \
  -p 15503:15503 \
  -v vec-data:/data \
  --restart unless-stopped \
  hivehub/vectorizer:3.6.1

The image defaults VECTORIZER_DATA_DIR=/data, so a single -v vec-data:/data mount captures the entire persistent state (.vecdb store, auth keys, JWT secret, snapshots, fastembed cache when enabled). docker compose up -d --force-recreate vectorizer is now safe - collections survive the recreate.

⁠Docker Compose

services:
  vectorizer:
    image: hivehub/vectorizer:3.6.1
    container_name: vectorizer
    # Distroless nonroot (UID 65532) refuses host-UID bind mounts on
    # Docker Desktop for Windows / macOS; flip to `user: root` if your
    # host can't align UIDs. On Linux with a named volume or a
    # UID-65532-owned bind mount, remove this.
    # user: root
    ports:
      - "15503:15503"   # VectorizerRPC (binary MessagePack over TCP - primary)
      - "15002:15002"   # REST + MCP + GraphQL + Dashboard
    volumes:
      # v3.4.0+ canonical mount - VECTORIZER_DATA_DIR defaults to /data.
      - vec-data:/data
    environment:
      - VECTORIZER_HOST=0.0.0.0
      - VECTORIZER_PORT=15002
      - VECTORIZER_AUTH_ENABLED=true
      - VECTORIZER_ADMIN_USERNAME=admin
      - VECTORIZER_ADMIN_PASSWORD=change-me-in-production
      - VECTORIZER_JWT_SECRET=change-this-to-a-random-32-char-secret
    restart: unless-stopped

volumes:
  vec-data:

⁠? Features (v3.5.0)

• ?? Single-volume container persistence - the image defaults VECTORIZER_DATA_DIR=/data so a single -v vec-data:/data mount captures collections, auth keys, JWT secret, snapshots, and the fastembed cache. Linux startup emits WARN data dir at <path> is ephemeral; recommend mounting a volume when the resolved dir has no backing mount, so a forgotten --volume flag fails loud instead of silently wiping collections on --force-recreate. New --data-dir <path> CLI flag for non-/data deployments.
• ?? Honoured embedding_provider / model contract - POST /collections and POST /embed reject unknown providers with 400 unsupported_provider { requested, available } / 400 unsupported_model { requested, available } instead of silently coercing to BM25. GET /stats lists every registered provider with dimension + default flag; mirrored as the list_providers MCP tool. CollectionConfig.embedding_provider persists which provider the collection was created with.
• ??? Backpressure-aware ingest - bounded-resource bulk-upsert: per-collection admission with 429 Too Many Requests + Retry-After on overload, BM25 vocab-build semaphore, structured queue_full MCP error, gRPC RESOURCE_EXHAUSTED. Configured via backpressure.{max_concurrent_vocab_builds,upsert_queue_high_water,upsert_queue_hard_limit} in config.yml. All five first-party SDKs honor Retry-After (1 s default, 30 s cap, 3 retries).
• ?? Stable client-id upserts - POST /insert_texts and POST /insert use the request id verbatim as Vector.id (or <id>#<chunkIndex> for chunked entries). Re-running the same payload upserts in place. Bulk POST /insert_vectors ingests pre-computed embeddings without going through the embedding pipeline.
• ? VectorizerRPC - length-prefixed MessagePack over raw TCP on port 15503, ~10� lower per-frame overhead than REST/JSON. Default binary transport across every SDK (Rust, TypeScript, Go, Python, C#).
• ?? Semantic Search - Cosine / Euclidean / Dot Product, HNSW indexing, sub-3 ms typical search, hybrid dense + sparse (BM25) with rank fusion.
• ? SIMD Acceleration - AVX2 on x86_64, NEON on aarch64, scalar fallback. CPU-feature detection at boot.
• ?? Embeddings - BM25 (default, 512-dim), TF-IDF, and FastEmbed ONNX models: all-MiniLM-L6-v2 (384-dim), all-MiniLM-L12-v2, all-mpnet-base-v2, bge-small-en-v1.5 (384), bge-base-en-v1.5 (768), bge-large-en-v1.5 (1024), plus -q int8-quantized variants (selected via embedding.model: fastembed:<id> in config.yml - use the bare id, not the Xenova/ HF path). The default hivehub/vectorizer:3.5.0 image is BM25-only (~91 MB); hivehub/vectorizer:3.5.0-fastembed is a published image with FastEmbed compiled in and all-MiniLM-L6-v2 pre-fetched - no self-build required.
• ?? Compact Storage - unified .vecdb format with 20-30% space savings, MMap support for datasets larger than RAM, automatic snapshots.
• ?? Quantization - Scalar + Product Quantization (PQ) for up to 64� memory reduction with minimal accuracy loss.
• ?? Replication & Sharding - master ? replica TCP streaming (BETA), openraft-backed consensus for HA clusters.
• ?? Document Conversion - built-in pipelines for PDF, DOCX, XLSX, PPTX, HTML, XML, images.
• ?? Qdrant Compatibility - drop-in /qdrant/collections/{name}/points/* surface for migrations.
• ?? MCP Integration - focused tool-per-action MCP 2025-03-26 server on POST /mcp (streamable HTTP).
• ??? Graph Relationships - relationship discovery + traversal, GUI-backed.
• ?? Auth enforcement - JWT + API Key with RBAC gating every data route when auth.enabled: true.
• ?? Observability - Prometheus metrics at /prometheus/metrics (now including upsert_queue_depth, upsert_in_flight, vocab_build_permits_available, upsert_rejected_total{reason}, bm25_empty_vocab_fallback_total{collection}), OpenTelemetry OTLP export, structured tracing via RUST_LOG. Operator runbook at docs/deployment/backpressure.md⁠ and importable Grafana panels at docs/grafana/backpressure-panels.json⁠.

⁠?? Tags

Older 1.x / 2.x tags remain on Docker Hub for rollback but are no longer receiving updates.

⁠?? Access Points

⁠??? Configuration

⁠Environment Variables

⁠Config Files

• Mount config.yml to /vectorizer/config.yml to override defaults (embedding model, quantization, HNSW params, auth mode, replication topology). Set embedding.model: fastembed:all-MiniLM-L6-v2 (or any other fastembed id) to register a dense provider as the default - note that the default published image is BM25-only; build with --build-arg ENABLE_FASTEMBED=1 --build-arg NO_DEFAULT_FEATURES=0 --build-arg FEATURES=fastembed to get fastembed compiled in.
• Mount workspace.yml to /vectorizer/workspace.yml + bind the source tree as /workspace:ro for monorepo indexing (the file-watcher service re-indexes on change).

⁠?? Examples

⁠REST - create / insert / search

# First-boot admin password was written to /data/.root_credentials.
# Read it (distroless image - no shell, so cat the file via exec):
#   docker exec vectorizer cat /data/.root_credentials
# Then login to get a JWT:
TOKEN=$(curl -sS -X POST http://localhost:15002/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"username":"admin","password":"<password-from-.root_credentials>"}' \
  | jq -r .access_token)

# Create a collection (dim 512 matches the default BM25 embedder)
curl -X POST http://localhost:15002/collections \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"name":"docs","dimension":512,"metric":"cosine"}'

# Insert text with automatic embedding generation
curl -X POST http://localhost:15002/insert_texts \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "collection":"docs",
    "texts":[
      {"id":"doc_1","text":"Vectorizer is a high-performance vector database","metadata":{"tag":"intro"}},
      {"id":"doc_2","text":"Rust is a systems programming language focused on safety","metadata":{"tag":"rust"}}
    ]
  }'

# Semantic search by text
curl -X POST http://localhost:15002/collections/docs/search/text \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"query":"vector database","limit":10}'

# Direct vector search
curl -X POST http://localhost:15002/search \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"collection":"docs","vector":[0.1, 0.2, 0.3, ...], "limit":10}'

⁠API Key (preferred for non-interactive clients)

# Create a long-lived key
curl -X POST http://localhost:15002/auth/keys \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"name":"my-service","permissions":["read","write","create_collection","delete_collection"]}'

# Use it via X-API-Key header
curl -H "X-API-Key: <api-key-from-response>" http://localhost:15002/collections

⁠MCP (Claude Code / Cursor / any MCP client)

{
  "mcpServers": {
    "vectorizer": {
      "type": "http",
      "url": "http://localhost:15002/mcp",
      "headers": { "X-API-Key": "<api-key>" }
    }
  }
}

⁠Graph Relationships

curl -X POST http://localhost:15002/graph/discover/docs \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"similarity_threshold":0.7,"max_per_node":10}'

⁠??? Image Details

• Base Image: dhi.io/debian-base:trixie (Docker Hardened Image - Docker-signed, Scout-native, weekly rebuilds, CIS-compliant), pinned by digest in 3.5.0 for reproducible builds. v3.0.0 shipped on gcr.io/distroless/cc-debian12:nonroot; v3.0.2 swapped to DHI to make Scout's "Approved Base Images" + "Up-to-Date Base Images" policies flip to Compliant.
• CVE posture: 0 CRITICAL / 0 HIGH. The unfixable base-image LOW/MEDIUM CVEs (no patched version in Debian trixie - e.g. tar file-smuggling, glibc regex DoS the binary never triggers, systemd code that never runs in the container) are declared not_affected in an OpenVEX attestation attached to the image, each with a written justification. A CI gate (docker-cve-gate.yml) re-scans weekly + on release and fails on any new fixable CRITICAL/HIGH.
• Default User: nonroot (UID 65532). Every COPY in the runtime stage is --chown=65532:65532 so the binary writes config.yml / workspace.yml on first boot without --user root.
• Architectures: linux/amd64, linux/arm64 (multi-arch manifest)
• Compressed Size: ~88 MB (v3.0.2+ on DHI; +21 MB vs the original distroless build because debian-base ships bash + full libssl/libcrypto/libsystemd/libreadline).
• Healthcheck: built-in HEALTHCHECK ... CMD ["/busybox", "wget", "-q", "--spider", "http://127.0.0.1:15002/health"] (a static busybox:stable-musl is COPY-ed into /busybox and used only as the healthcheck entrypoint).
• Rust Edition: 2024 (mandatory, pinned rustc ? 1.90 per async-graphql / asynk-strim floor)
• Build Flags: --package vectorizer-server --bin vectorizer --no-default-features (excludes ONNX / FastEmbed / GPU / Transmutation from the default image to keep the dependency surface small). The container binary is compiled with the dedicated release-docker Cargo profile (lto = false, codegen-units = 16, inherits release otherwise) - same opt-level=3 + strip behavior as the host cargo build --release, but ~30% faster to compile inside BuildKit at the cost of ~10-15% lower throughput on hot paths versus a host-built release binary. Operators chasing peak per-op throughput should rebuild from source with the workspace release profile.
• Supply Chain: per-arch SBOM and SLSA mode=max provenance attached as in-toto attestations to the multi-arch manifest list (Docker Scout reads from there). Inspect with docker buildx imagetools inspect hivehub/vectorizer:<tag>. OpenContainer labels carry revision, source, and license metadata.
• License: Apache-2.0

⁠?? Advanced Usage

⁠Monorepo Indexing

docker run -d \
  --name vectorizer \
  -p 15002:15002 -p 15503:15503 \
  -v vec-data:/data \
  -v $(pwd)/workspace.yml:/vectorizer/workspace.yml \
  -v $(pwd):/workspace:ro \
  --restart unless-stopped \
  hivehub/vectorizer:3.6.1

⁠Fastembed Variant (dense semantic search out of the box)

The -fastembed image is published - no self-build needed. It ships with FastEmbed compiled in and all-MiniLM-L6-v2 pre-fetched; a config file selects it as the default provider:

docker run -d \
  --name vectorizer \
  -p 15002:15002 -p 15503:15503 \
  -v vec-data:/data \
  -v $(pwd)/fastembed-config.yml:/vectorizer/config.yml:ro \
  hivehub/vectorizer:3.6.1-fastembed

Where fastembed-config.yml carries (use the bare model id - Xenova/all-MiniLM-L6-v2 is rejected):

embedding:
  model: fastembed:all-MiniLM-L6-v2

GET /stats.providers then lists both fastembed:all-MiniLM-L6-v2 (384-dim, default) and bm25 (512-dim, sparse fallback). Hybrid retrieval works out of the box. To compile fastembed into your own image instead, build with --build-arg ENABLE_FASTEMBED=1 --build-arg NO_DEFAULT_FEATURES=0 --build-arg FEATURES=fastembed.

⁠Debug Logging

docker run -d \
  --name vectorizer \
  -p 15002:15002 \
  -e RUST_LOG="vectorizer=debug,vectorizer::replication=trace,hyper=info" \
  -e RUST_BACKTRACE=1 \
  hivehub/vectorizer:3.6.1

⁠Pinning by Digest (Production)

docker pull hivehub/vectorizer@sha256:f3a642d4662ff9bf0f1acdb74b6cd56f12e0e4ea5cdeb2128714087cea40cc47

⁠?? Support

• ?? Docker Hub: https://hub.docker.com/r/hivehub/vectorizer⁠
• GitHub: https://github.com/hivellm/vectorizer⁠
• Issues: https://github.com/hivellm/vectorizer/issues⁠
• Discussions: https://github.com/hivellm/vectorizer/discussions⁠

⁠?? License

Apache-2.0 License - see LICENSE⁠.

?? Pull:

docker pull hivehub/vectorizer:3.6.1
docker pull hivehub/vectorizer:3.6.1-fastembed   # dense embeddings pre-baked
docker pull hivehub/vectorizer:latest
# Or pin earlier v3.x stable points:
docker pull hivehub/vectorizer:3.6.0
docker pull hivehub/vectorizer:3.5.0
docker pull hivehub/vectorizer:3.4.0
docker pull hivehub/vectorizer:3.3.0
docker pull hivehub/vectorizer:3.2.0

?? Repository: https://hub.docker.com/r/hivehub/vectorizer⁠