n8500x/jev repository overview

⁠jev

Multiple-choice classification from next-token logits, served as a local HTTP API. The method is the one from NobodyWho's Jev in 25 lines of Python⁠: present the choices as lettered options, run one forward pass, read the logits of the option letters at the last position, softmax over just those. No generation, no sampling, deterministic, milliseconds per request on CPU.

• Image: n8500x/jev — llama-cpp-python⁠ 0.3.36 (vendors llama.cpp from 2026-10-01) compiled from source, CPU-only, no model baked in: mount any instruct GGUF already on-prem.
• Air-gap: no pip, no Hugging Face, no telemetry at runtime (HF_HUB_OFFLINE=1, LLAMA_CURL=OFF). Dependencies are pinned and the exact lock file ships inside the image (docker run --rm n8500x/jev requirements).
• Contract: same mount / env / UID conventions as n8500x/llamacpp⁠ — /models, MODEL_PATH, PORT, CTX_SIZE, THREADS, API_KEY; non-root UID 1001 / group 0 (OpenShift restricted-v2 clean).

⁠Run

docker run -d --name jev -p 8080:8080 \
  -v /srv/models/Qwen3-4B-Q4_K_M.gguf:/models/model.gguf:ro \
  n8500x/jev

Or docker compose up -d — the bundled docker-compose.yml⁠ carries the annotated configuration. The image is self-documenting:

docker run --rm n8500x/jev compose > docker-compose.yml
docker run --rm n8500x/jev readme
docker run --rm n8500x/jev requirements      # pip freeze of the image
docker run --rm n8500x/jev docs              # index of everything below
docker run --rm -v "$PWD":/out n8500x/jev export   # copy it all out

⁠Docs and examples travel with the image

Nothing in a gated network can follow a link, so the whole of this directory's documentation is baked in under /usr/share/jev and reachable three ways: docker run --rm n8500x/jev docs <path>, … export into a mounted directory, or GET /docs / GET /docs/<path> on a running instance.

docker run --rm n8500x/jev docs OPERATIONS.md
docker run --rm n8500x/jev python3 /usr/share/jev/examples/evaluate.py \
  --url http://jev:8080 --data /usr/share/jev/examples/sample.jsonl --input-label Email
curl -s localhost:8080/docs/docs/PROMPTS.md

Any instruct GGUF works. Size is a real trade-off: a 0.6B answers a 50-token prompt in ~0.3 s on a laptop CPU but is only reliable for coarse, obvious classes; an 8B takes ~2.4 s and handles nuance. On the bundled examples/sample.jsonl (15 emails, legitimate / spam / phishing) Qwen3-0.6B scores 6/15 and Qwen3-8B 13/15 — run examples/evaluate.py on your own labelled sample before choosing. Use the coder models you already have for code-shaped inputs (log lines, stack traces, SQL) and a general instruct model for prose. Qwen3 / Qwen3.6 templates (including the MoE coder builds) are handled: thinking is switched off through the template's own enable_thinking flag.

Reference numbers (i5-12450H laptop, Docker Desktop/WSL2, Haswell-baseline build, the post's 50-token prompt via demo, warm):

The whole cost is one llama_decode prefill (tokenize + logits copy are < 1 ms); a server-class CPU with AVX-512 and a matching build is several times faster.

⁠API

curl -s localhost:8080/v1/classify -H 'content-type: application/json' -d '{
  "input": "Payroll asks for your password on a non-company sign-in page.",
  "choices": ["Legitimate", "Spam", "Phishing"],
  "input_label": "Email"
}'

{
  "choice": "Phishing", "label": "C", "index": 2, "confidence": 0.677,
  "probabilities": {"Legitimate": 0.168, "Spam": 0.155, "Phishing": 0.677},
  "logprobs":      {"Legitimate": -1.783, "Spam": -1.864, "Phishing": -0.390},
  "logits":        {"Legitimate": 27.800, "Spam": 27.719, "Phishing": 29.193},
  "choice_mass": 0.982,
  "labels": {"A": "Legitimate", "B": "Spam", "C": "Phishing"},
  "label_tokens": {"A": 32, "B": 33, "C": 34},
  "prompt_tokens": 56, "latency_ms": 477.1,
  "model": "Qwen3 0.6B Instruct", "template": "gguf"
}

Measured with this image on Qwen3-0.6B-Q8_0.gguf, the model the post used, with the image's default instruction. With the post's own instruction ("instruction": "Choose one option.", which is what demo sends) the same build gives 26.271 / 27.041 / 29.589 → Phishing 0.897, against the post's 26.254 / 27.262 / 29.614 → 0.885; the small drift is the CPU kernel. Qwen3-8B-Q4_K_M on the same email: Phishing 1.000, choice_mass 1.000 either way.

Request fields:

choice_mass is the share of the model's whole next-token distribution that lands on the option letters. Close to 1 means the model is answering the question; a small value means it wanted to write something else (thinking not switched off, wrong template, instruction too vague) and the probabilities are not to be trusted. test.ps1 fails below 0.5.

⁠Prompt construction

The GGUF's own chat template is rendered with system + user turns and add_generation_prompt, with thinking disabled (enable_thinking=False, which Qwen3-family templates honour). If the model is a thinking family and the template did not already close an empty <think></think> block, one is appended so the first generated token is the answer letter — exactly the trick in the post. CHAT_TEMPLATE=chatml forces the post's literal Qwen ChatML prompt instead; THINK_SKIP=always|never overrides the detection.

The user turn is "{input_label}: {input}\n\n{A. choice}\n{B. choice}…", as in the post.

Why the default instruction adds "Answer with the letter only." Larger models read the post's bare Choose one option. as an invitation to explain, and then the letter logits are a forced ranking rather than an answer. choice_mass makes that visible; the extra sentence fixes it:

The ranking (choice) was the same in every case; what changes is whether the probabilities mean anything. Pass "instruction": "Choose one option." to get the post's exact prompt.

⁠CLI

docker run --rm -v ./m.gguf:/models/m.gguf:ro n8500x/jev demo
docker run --rm -v ./m.gguf:/models/m.gguf:ro n8500x/jev classify \
  -i "ORA-01555: snapshot too old" -c Network Storage Database Application --input-label "Log line"

demo reproduces the post's example with the post's exact instruction and prints its three lines (logits, log probabilities, probabilities). classify uses the image default instruction (override with --instruction). --json gives the full API result, --show-prompt prints what the model saw.

⁠Configuration

Memory ≈ model file + KV cache (CTX_SIZE × layers; ~150 MB per 1k tokens on an 8B, far less on small models). The model is mmaped, so leave page-cache headroom in mem_limit. One model instance per container, requests serialised; scale with replicas.

⁠How it differs from the 25 lines

The arithmetic is unchanged: logprobs = logits − logaddexp.reduce(logits), probabilities = exp(logprobs).

⁠Build

.\build.ps1            # compiles llama-cpp-python (~2 min on 12 cores), tags date+latest
.\test.ps1             # fetches Qwen3-0.6B-Q8_0 (610 MB, test only), runs API + CLI checks
.\test.ps1 -Port 18081 -ModelsDir ..\llamacpp\models   # same checks against another GGUF dir
.\build.ps1 -Push      # push both tags, record .push_tag

test.ps1 asserts the post's verdict (Phishing), a choice_mass ≥ 0.5, the batch endpoint, input validation, and the demo / classify CLI paths. It stops the server container before the CLI runs because each CLI call loads its own copy of the model.

The CPU ISA is fixed at build time (pip cannot build ggml's runtime-dispatched variants): the default is a Haswell baseline (AVX2/FMA/F16C/BMI2) that runs on any x86-64 server since 2013. For AVX-512 hosts build a second tag:

.\build.ps1 -CmakeArgs "-DGGML_NATIVE=OFF -DGGML_AVX=ON -DGGML_AVX2=ON -DGGML_FMA=ON -DGGML_F16C=ON -DGGML_BMI2=ON -DGGML_AVX512=ON -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON"

The flags used are recorded in the image label ai.jev.ggml_cmake_args. Build-time pip trusts the index hosts (PIP_TRUSTED_HOST, build stage only) because of host TLS interception; the runtime stage has no pip and no CA material.

To bump llama-cpp-python: change the pin in requirements.txt and the LLAMA_CPP_PYTHON_VERSION arg in the Dockerfile (they are cross-checked).

⁠Caveats

• Labels are scored by their first token, tokenized standalone. Single capital letters are one token in Qwen/Llama tokenizers; SentencePiece models that prefix a space marker may score slightly differently from the post.
• This gives the model's preference among the options, not calibrated accuracy. Judge a prompt + model on a labelled sample before relying on it.
• One request at a time per container (one llama.cpp context). For throughput, run replicas or use /v1/classify/batch.