olkowa/pattern-tei-rerank repository overview

⁠pattern-tei-rerank

Offline reranker sidecar for the pattern MCP servers — Hugging Face TEI (CPU) with Alibaba-NLP/gte-reranker-modernbert-base⁠ (149M params, 8k context, English) baked into the image. No downloads, no API keys at runtime.

┌────────────────────────────────────────────────────────────────┐
│  architecture-pattern-mcp / agent-pattern-mcp                  │
│        ───────── HTTP /rerank ─────────►                       │
│                                                                │
│      ┌───────────────────────────────┐                         │
│      │     pattern-tei-rerank        │ ◄── pattern-tei-infra   │
│      │  gte-reranker-modernbert-base │                         │
│      │         :8080 (TEI, CPU)      │                         │
│      └───────────────────────────────┘                         │
└────────────────────────────────────────────────────────────────┘

⁠Architecture

This image is the reranker only. It contains no MCP server, no embedder, and no LLM client. The MCP servers (olk/architecture-pattern-mcp⁠, olk/agent-pattern-mcp⁠) reach it via the shared Docker network pattern-tei-shared at the DNS name pattern-tei-rerank (see the pattern-tei-infra stack⁠).

Runtime base is ghcr.io/huggingface/text-embeddings-inference:cpu-1.9, but the router binary is not stock: it is self-built from TEI v1.9.4 plus unmerged upstream PR #884⁠ (see Startup and --warmup-tokens⁠ below). Pinned invariants: --model-id /data/reranker-model --dtype float32 (ORT backend; sequence-classification models take no --pooling flag).

⁠What you get

• Model: Alibaba-NLP/gte-reranker-modernbert-base — English, 149M params, 8k context, Apache 2.0 (fp32 ONNX baked in, onnx/model.onnx)
• Server: TEI router v1.9.4 + PR #884 on :8080 (ORT backend, float32)
• Endpoints: POST /rerank (raw TEI) · GET /health · GET /info
• Fully offline: weights baked in at build time, runs with HF_HUB_OFFLINE=1
• Tuned defaults: MAX_CONCURRENT_REQUESTS=64, MAX_BATCH_TOKENS=16384, MAX_CLIENT_BATCH_SIZE=48, AUTO_TRUNCATE=true, WARMUP_TOKENS=0, PROMETHEUS_PORT=0 (metrics not exposed)
• Image size: ~600 MB on disk · Architecture: linux/amd64, linux/arm64

No config.json patching is needed: max_position_embeddings=8192 is below the default max_batch_tokens=16384, so TEI accepts the model config without the warmup warning that affects larger-context models.

⁠Startup and --warmup-tokens

Why this image carries a custom router build instead of stock cpu-1.9:

• Stock TEI cpu-1.9 has no --warmup-tokens / WARMUP_TOKENS flag. On CPU it always runs a synthetic warmup pass of min(max_input_length, max_batch_tokens) tokens before serving. Measured cost for this 149M model: ~78 s of an ~81 s cold start — warmup is the boot time.
• The only workaround on stock TEI is lowering MAX_BATCH_TOKENS (upstream issue #819⁠), which caps throughput to buy back startup time.
• Upstream PR #884⁠ (open/unmerged as of 2026-10-09) adds --warmup-tokens / WARMUP_TOKENS. This image builds the router from TEI v1.9.4 + PR #884 (pinned SHA 5806bdc) and sets WARMUP_TOKENS=0 (skip the synthetic pass) by default, so the container is ready in seconds (weight load only).

Semantics of WARMUP_TOKENS (this image only — stock TEI ignores the variable):

Unlike the Qwen3 embedder sidecar (capped at MAX_BATCH_TOKENS=8192 for the documented CPU warmup OOM, PR #634/#725), 16384 is safe here — the 149M reranker never hit that failure mode.

⁠Configuration

All tunables are environment variables (clap fallbacks; CLI flags win). Image defaults are overridable per container, e.g. docker run -e WARMUP_TOKENS=512 … or the compose environment: block:

The MCP consumer chunks its fused candidate pool into requests of RERANKER_MAX_BATCH_SIZE (default 48, config.json → RERANKER_MAX_BATCH_SIZE) against http://pattern-tei-rerank:8080, with keep_retrieval_score preserving the stage-1 fusion score alongside the cross-encoder logit.

⁠Prerequisites

• Docker 24+ with compose plugin (docker compose version)
• Disk: ≥ 2 GB free
• CPU-only: no GPU required

⁠Quick start

Part of the pattern-tei-infra⁠ stack (recommended — shared Docker network pattern-tei-shared, DNS name pattern-tei-rerank):

git clone https://github.com/olk/pattern-tei-infra.git && cd pattern-tei-infra
TEI_RERANK_IMAGE=olkowa/pattern-tei-rerank:latest docker compose up -d

Or standalone (no MCP consumer):

docker run -d --name pattern-tei-rerank -p 8080:8080 olkowa/pattern-tei-rerank:latest
curl -fsS http://localhost:8080/health

Nothing to configure — no env vars, no model download, no HF token. Cold start is seconds (WARMUP_TOKENS=0); expect ~80 s only if you explicitly set a large WARMUP_TOKENS.

⁠Operations

# Update
docker compose -f pattern-tei-infra/docker-compose.yml pull

# Logs
docker compose -f pattern-tei-infra/docker-compose.yml logs -f pattern-tei-rerank

# Health
docker inspect --format '{{.State.Health.Status}}' pattern-tei-rerank

# Re-enable a warmup pass (e.g. to smooth first-request latency)
WARMUP_TOKENS=512 docker compose -f pattern-tei-infra/docker-compose.yml up -d pattern-tei-rerank

# Pin a version
TEI_RERANK_IMAGE=olkowa/pattern-tei-rerank:0.1.0 \
  docker compose -f pattern-tei-infra/docker-compose.yml up -d

⁠Links

• Stack repo: Olk/pattern-tei-infra⁠
• Consumers: Olk/architecture-pattern-mcp⁠ · Olk/agent-pattern-mcp⁠
• Sibling image: Olkowa/pattern-tei-embed⁠
• Upstream warmup PR: huggingface/text-embeddings-inference#884⁠
• Image license: MIT⁠
• Model license: Apache 2.0 (gte-reranker-modernbert-base⁠)