- 01.The three independent metrics that decompose RAG quality -- context relevance, faithfulness, and answer relevance -- and why keeping them separate is the entire point.
- 02.The failure pattern most teams misread: high faithfulness plus low answer relevance means the system is accurately summarizing the wrong thing.
- 03.Why enterprise RAG quality is inseparable from access control, document freshness, and multi-tenant architecture.
The Story
Consider a large employer that launched an internal HR assistant over benefits, leave, and travel policy documents. Employees started getting answers that were articulate, cited policy language, and were often slightly wrong on the newest rules. The team's first instinct: upgrade the model -- "it's hallucinating."
After trace review, the pattern was obvious. The retriever kept surfacing older policy versions because the index sync lagged behind the source-of-truth document system. The answer stayed faithful to retrieved context -- the model wasn't making things up. The context itself was stale. The team thought they had a model problem. They actually had a retrieval freshness problem.
A SaaS support bot asked: "Can I cancel and get a prorated refund on the annual enterprise plan?" It retrieved the refund policy for self-serve annual plans, summarized it accurately, and cited it correctly. The answer was grounded. It was also wrong -- the user was on an enterprise contract with different terms. High faithfulness, low answer relevance. The team initially celebrated because citation quality improved. In reality, the bot had become a better summarizer of the wrong document.
Three systems. Three different "RAG failures." None of them were model problems. All of them were invisible to a single blended "RAG quality" score.
The Core Idea
RAG failures are usually not "the model was wrong." They are "the retriever fetched the wrong thing," "the model stayed grounded to the wrong thing," or "the answer stayed grounded but didn't answer the user's actual question."
Where This Hits in Production
Access control is part of retrieval quality. In enterprise RAG, "did the retriever fetch the right documents?" has a security dimension. AWS's guidance is the cleanest statement: unlike search engines, RAG returns results directly from the LLM, which can bypass permission checks at the original data source. From a PM standpoint, "retrieval relevance" without "authorization correctness" is not enterprise retrieval quality.
Multi-tenant document freshness is a silent killer. If Tenant A's index is stale and Tenant B's is current, the same user question can look like a model failure in one account and a retrieval freshness issue in another. Triad dashboards need version metadata -- which knowledge base version, which sync timestamp, which tenant environment.
Multi-hop queries break simplistic triad thinking. When a question requires combining evidence from multiple documents, single-hop context precision can look fine while the system misses evidence scattered across three different policy documents. Context recall becomes the metric that catches these failures -- and most teams don't track it.
Common Mistake
Collapsing the triad into one blended "RAG quality" score.
Why it feels right: a single number looks executive-friendly, easy to trend, and easy to compare across experiments.
What actually happens: a system with context precision 0.88, faithfulness 0.96, and answer relevance 0.63 averages to 0.82 -- which looks fine. But the answer relevance score means users aren't getting their questions answered.
The fix: never collapse the triad into a single release number without also preserving the per-metric view. Gate release decisions on the worst individual metric, not the average.Remember This
1. The RAG triad asks three independent questions: did the retriever fetch the right documents (context relevance), did the model stay grounded (faithfulness), and did the answer address the user's question (answer relevance)? Keep them separate. That separation IS the diagnostic value.
2. The failure pattern most teams misread: high faithfulness plus low answer relevance means the system is accurately summarizing the wrong evidence. The citations look clean. The user is still unserved.
3. In enterprise RAG, access control and document freshness are part of retrieval quality, not separate concerns. A triad dashboard without version metadata and authorization checks is incomplete.
References
1. RAG Triad -- Core Concepts -- TruLens
2. Benchmarking LLM-as-a-Judge for RAG Triad Metrics -- Snowflake
3. Faithfulness Metric -- Ragas
4. Answer Relevance Metric -- Ragas
5. Authorizing Access to Data with RAG -- AWS
6. Evaluate RAG Tutorial -- LangSmith
7. MultiHop-RAG -- arXiv