chiefj/perplexity

By chiefj

•Updated 4 months ago

Perplexity scoring microservice

Image

Developer tools

722

Overview Tags

chiefj/perplexity repository overview

⁠Perplexity Scorer

Layer 3 perplexity scoring microservice — Score 4 in the LLM Observability composite quality index. Runs on port 8007. Zero GPU required.

⁠What It Does

Scores the cross-entropy perplexity of an LLM response.

Scorer path	When used	Cost
Provider logprobs (primary)	`token_logprobs` array provided	Zero inference
GPT-2 124M ONNX (fallback)	No logprobs supplied	~25ms on CPU

Returns null (skipped) when skip conditions are met. The composite weight drops to 0.00 automatically.

⁠Run Locally (Python)

cd packages/python/perplexity

pip install -e ".[dev]"

SKIP_CONSOLE_EXPORTER=true SKIP_OTLP_EXPORTER=true \
  pytest tests/ --cov=src -v

Start the server:

PYTHONPATH=src uvicorn api.rest.v1.app:app --host 0.0.0.0 --port 8007 --reload

⁠Run with Docker

⁠Pull and run the published image

docker pull chiefj/perplexity:latest

docker run -d \
  --name perplexity \
  -p 8007:8007 \
  -e SKIP_OTLP_EXPORTER=true \
  chiefj/perplexity:latest

⁠Health check

curl http://localhost:8007/health
# {"status":"ok","scorer":"provider_logprobs"}

⁠Score a response (with provider logprobs)

curl -X POST http://localhost:8007/v1/score/perplexity \
  -H "Content-Type: application/json" \
  -d '{
    "trace_id": "abc123",
    "span_id": "def456",
    "response_text": "The mitochondria is the powerhouse of the cell.",
    "completion_tokens": 12,
    "prompt_type": "chat",
    "token_logprobs": [-1.2, -0.9, -1.5, -2.1, -0.8, -1.3, -1.0, -0.7, -1.4, -2.0, -1.1, -0.6]
  }'

Expected response:

{
  "trace_id": "abc123",
  "span_id": "def456",
  "perplexity": 3.31,
  "score": 0.72,
  "weight": 0.10,
  "skipped": false,
  "skip_reason": null,
  "high_perplexity_flag": false,
  "prompt_type": "chat",
  "scorer_used": "provider_logprobs"
}

⁠Score a response (GPT-2 fallback — no logprobs)

curl -X POST http://localhost:8007/v1/score/perplexity \
  -H "Content-Type: application/json" \
  -d '{
    "trace_id": "xyz789",
    "span_id": "qrs012",
    "response_text": "Paris is the capital of France and a major European city.",
    "completion_tokens": 15,
    "prompt_type": "rag"
  }'

⁠Skipped response (too few tokens)

curl -X POST http://localhost:8007/v1/score/perplexity \
  -H "Content-Type: application/json" \
  -d '{
    "trace_id": "skip1",
    "span_id": "skip2",
    "response_text": "Yes.",
    "completion_tokens": 2,
    "prompt_type": "chat"
  }'
# {"skipped": true, "skip_reason": "completion_tokens_too_few", "weight": 0.0, ...}

⁠Build the image locally

cd packages/python/perplexity

docker build \
  -f build/Dockerfile \
  -t chiefj/perplexity:local \
  .

⁠Run with Docker Compose

cd packages/python/perplexity

docker compose -f deploy/docker/docker-compose.yaml up -d

Stop:

docker compose -f deploy/docker/docker-compose.yaml down

⁠API Reference

Full contract: contracts/openapi/v1.yaml⁠

⁠`POST /v1/score/perplexity`

Field	Type	Required	Description
`trace_id`	string	✅	Distributed trace ID
`span_id`	string	✅	Parent span ID
`response_text`	string	✅	LLM completion text
`completion_tokens`	int	✅	Token count — `< 10` triggers skip
`prompt_type`	enum	✅	`chat` \| `code` \| `rag` \| `classification`
`token_logprobs`	float[]	❌	Provider log-probs (primary path)
`finish_reason`	string	❌	`content_filter` triggers skip

⁠`GET /health`

Returns {"status": "ok", "scorer": "<provider_logprobs|gpt2_onnx|unavailable>"}.

⁠Skip Conditions

Condition	`skip_reason`	`weight`
`completion_tokens < 10`	`completion_tokens_too_few`	0.00
`finish_reason == "content_filter"`	`finish_reason_blocked`	0.00
No logprobs AND GPT-2 unavailable	`scorer_unavailable`	0.00

⁠Baselines & Alerting

`prompt_type`	Expected perplexity	HIGH flag threshold (`3 ×`)
`chat`	15–35	> 105
`code`	8–20	> 60
`rag`	12–28	> 84
`classification`	6–15	> 45

high_perplexity_flag: true indicates the model was likely confused, hallucinating, or output low-quality text.

⁠Composite Weight

State	`w_perplexity`	`score` contribution
perplexity available	0.10	`1/log(perplexity)` → normalized to [0, 1]
perplexity skipped	0.00	other scorer weights renormalized externally

⁠Environment Variables

Variable	Default	Description
`GPT2_MODEL_PATH`	`gpt2`	HuggingFace model ID or local path for GPT-2 ONNX
`SKIP_CONSOLE_EXPORTER`	`false`	Suppress OTel console span output
`SKIP_OTLP_EXPORTER`	`true`	Disable OTLP gRPC/HTTP exporter
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://localhost:4317`	OTLP collector endpoint
`DEPLOYMENT_ENV`	`dev`	OTel resource `deployment.env` attribute

Copy .env.example to .env to configure locally.

⁠CI / CD

The pipeline (perplexity-ci.yml) triggers only when files under packages/python/perplexity/ change and the target branch is main:

Event	Jobs triggered
Pull request → `main` (perplexity path)	`test` (Python 3.11 + 3.12)
Push/merge → `main` (perplexity path)	`test` → `build-and-push`

⁠GitHub Secrets required

Go to Settings → Secrets → Actions and add:

Secret name	Value
`DOCKERHUB_USERNAME`	`chiefj`
`DOCKERHUB_TOKEN`	your Docker Hub access token

⁠Docker Hub tags published on merge to main

chiefj/perplexity:latest
chiefj/perplexity:stable
chiefj/perplexity:v<version>

⁠Algorithm

Primary path (provider logprobs):
  perplexity = exp( -1/N × Σ token_logprobs )

Fallback path (GPT-2 124M ONNX, CPU):
  perplexity = exp( cross_entropy_loss )   # ~25ms / 200 tokens

Normalized score:
  contribution = 1 / log(perplexity)
  score        = clamp((contribution - low) / (high - low), 0, 1)

Tag summary

Recent tags

Content type

Image

Digest

sha256:1790fab76…

Size

109.4 MB

Last updated

4 months ago

docker pull chiefj/perplexity