Learning path · Evaluation & Quality · 76
Faithfulness and Relevance
Faithfulness measures whether answers are supported by context; relevance measures whether they address the question.
Why it matters
- Orthogonal failures need different fixes—prompt vs retrieval.
- Core RAG metrics alongside Precision@K and Recall@K.
- Drive abstention policies when either score is low.
Key ideas
- Supported claims
- Question alignment
- Abstention thresholds
An answer can be faithful but irrelevant (quotes the right doc but misses the question) or relevant but unfaithful (addresses the ask using invented facts). Score both separately with judges or NLI models. Set production thresholds to trigger shorter replies, extra retrieval, or human escalation. Report slice metrics in dashboards executives actually read. Tune abstention copy for low scores—users respond better to scoped "here is what we know" answers than empty errors. Ship only after eval gates pass on representative production failures.
Updated 2026-08-09 · Full learning path