chiefj/faithfulness repository overview

# 1. Pull and run
docker pull chiefj/faithfulness:latest
docker run -p 8006:8006 chiefj/faithfulness:latest

# 2. Check it's alive
curl http://localhost:8006/health

# 3. Score a response
curl -s -X POST http://localhost:8006/v1/score/faithfulness \
  -H "Content-Type: application/json" \
  -d '{
    "trace_id": "trace-001",
    "span_id":  "span-001",
    "rag_context": "The Eiffel Tower is located in Paris, France. It was built in 1889 by Gustave Eiffel.",
    "response_text": "The Eiffel Tower is in Paris. It was built in 1889.",
    "completion_tokens": 14
  }' | python3 -m json.tool

{
  "trace_id": "trace-001",
  "span_id":  "span-001",
  "score": 1.0,
  "skipped": false,
  "skip_reason": null,
  "total_qualifying": 2,
  "entailed_count": 2,
  "sentence_results": [
    {"sentence": "The Eiffel Tower is in Paris.", "label": "entailment", "entailment_prob": 0.912},
    {"sentence": "It was built in 1889.", "label": "entailment", "entailment_prob": 0.887}
  ]
}

[Request arrives]
        |
        v
[FAITHFULNESS-ST-01 | Incoming call: response_text + rag_context + completion_tokens]
        |
        +-- [FAITHFULNESS-IN-01 | Input validation: null? too short? blocked?]
            |
            +-- [FAITHFULNESS-DC-01 | Should we score?]
                |
                |-- finish_reason = "content_filter"
                |   └── [SKIP] score=null, skip_reason=finish_reason_blocked
                |       └── Downstream: w_faith re-normalised to 0 (no penalty, no reward)
                |
                |-- rag_context is null
                |   └── [SKIP] score=null, skip_reason=rag_context_null
                |       └── Downstream: cannot grade what was never retrieved
                |
                |-- len(rag_context) < 50 chars
                |   └── [SKIP] score=null, skip_reason=rag_context_too_short
                |       └── Business rule: stub contexts carry no grounding signal
                |
                |-- completion_tokens < 10
                |   └── [SKIP] score=null, skip_reason=completion_tokens_too_few
                |       └── Business rule: refusals, one-word answers not scoreable
                |
                `-- All conditions pass
                    |
                    v
            [FAITHFULNESS-PR-01 | Split response into sentences (spaCy en_core_web_sm)]
                    |
                    +-- All sentences < 5 words
                    |   └── [SKIP] score=null, skip_reason=no_qualifying_sentences
                    |
                    `-- Qualifying sentences found
                        |
                        v
                [FAITHFULNESS-PR-02 | Is context > 512 words?]
                        |
                        |-- YES → split into ≤400-word chunks
                        |         for each sentence: NLI vs every chunk
                        |         take MAX entailment across chunks
                        |         (benefit-of-the-doubt: one supporting chunk is enough)
                        |
                        `-- NO  → NLI: batch all (context, sentence) pairs at once
                                |
                                v
                        [FAITHFULNESS-PR-03 | DeBERTa NLI · T=1.5 temperature scaling]
                                |
                                |  logits → scale by 1/T → softmax
                                |  → (P_entailment, P_neutral, P_contradiction)
                                |  → label = argmax
                                |
                                v
                        [FAITHFULNESS-PR-04 | Compute entailment fraction]
                                |
                                |  score = entailed_sentences / total_qualifying
                                |
                                v
                        [FAITHFULNESS-OUT-01 | Return score ∈ [0.0, 1.0]]
                                |
                                v
                        [FAITHFULNESS-CS-01 | Composite scorer uses w_faith=0.40]
                                |
                                |  composite = w_coh * coherence_score
                                |            + w_faith * faithfulness_score
                                |            + w_... * ...
                                |
                                |  if faithfulness_score IS NULL:
                                |    redistribute w_faith to remaining metrics

[FastAPI port 8006]
      │
      ▼
score_faithfulness()  ← pure service, zero infra imports
      │
      ├── SentencizerPort ← SpacySentencizerAdapter (en_core_web_sm)
      └── NliScorerPort   ← DeBertaNliAdapter (cross-encoder/nli-deberta-v3-base)

Dependency arrows point INWARD. Domain never imports infra.

{"status": "ok", "model_id": "cross-encoder/nli-deberta-v3-base"}

# Install
pip install -e ".[dev]"
python -m spacy download en_core_web_sm

# Test
./scripts/test.sh

# Docker
docker-compose -f deploy/docker/docker-compose.yaml up

w_faith = 0.40  when score IS NOT NULL
w_faith = 0.00  when score IS NULL  →  redistribute to other metrics