RAG Has Two Quality Gates: Finding the Right Evidence and Producing a Defensible Answer

A technical leader is reviewing a proposed internal policy assistant. During the demonstration, an employee asks, “Which remote-work policy applies to someone who transferred departments this year?”

The assistant produces a concise answer, quotes policy language and displays a source link. It looks ready. But the review team cannot yet tell whether the passage came from the current approved policy, whether the employee was permitted to retrieve it or whether the answer added a conclusion that the passage did not establish.

Those are not variations of one problem. They are two separate failure points.

A retrieval-augmented generation system first finds candidate evidence and then uses that material to formulate an answer. Microsoft’s overview of RAG in Azure AI Search describes this grounding pattern and the retrieval challenges around relevance, multiple sources, content preparation and access control. The retrieved material can be wrong for the task even if the language model uses it faithfully. Suitable evidence can also be retrieved and then overstated, combined incorrectly or cited unclearly in the generated response.

Treat the system as an evidence supply chain with two independent quality gates:

1. Retrieval gate: Did the system find evidence that is authoritative, applicable, current, permitted and relevant?

2. Answer gate: Does the response stay within that evidence, preserve necessary qualifications and identify its support clearly?

Both gates must pass. A polished response cannot repair unsuitable evidence, and good retrieval cannot excuse an unsupported answer.

Start with one bounded knowledge task, not a general ambition to “chat with company data.” Name the intended users, source collections and decisions or actions the answers may influence. For the policy assistant, the task might be limited to helping employees identify the applicable published policy and understand its stated conditions. It would not automatically include making exceptions, interpreting employment law or approving a request.

Before approving a RAG pilot, confirm that the team has:

• representative questions, including ambiguous and no-answer cases;

• an owner who can identify authoritative sources and applicable versions;

• an owner for access rules;

• prospective owners for retrieval evaluation, answer review and approval;

• a way to determine what a correct refusal or escalation looks like.

RAG earns its additional moving parts when the task genuinely benefits from synthesizing suitable material across documents or sources. If a user only needs to open the current policy, retrieve a fixed entitlement from a structured system or select an approved answer, generation may add risk without adding enough utility.

This is an architecture and acceptance decision, not an installation tutorial, standalone model benchmark, compliance assessment, cost estimate or performance promise. The question is whether the proposed route can satisfy this particular knowledge task with an operating burden the organization is prepared to own.

A useful review follows the evidence through the system rather than judging only the final prose:

1. Select sources. Decide which repositories and records are eligible to answer the bounded task.

2. Attach operating context. Preserve information needed to distinguish authority, version, effective date and access conditions.

3. Prepare and index content. Convert documents into searchable units while retaining the metadata needed for evaluation and filtering.

4. Retrieve candidate passages. Use the user’s question and access context to find material that may support an answer.

5. Supply context to the language model. Pass a limited set of retrieved passages into the generation step.

6. Generate and present the answer. Return the response with enough source information for a reviewer or user to inspect its basis.

Azure AI Search documents content preparation, indexing, incremental refresh, retrieval and permission-aware patterns as parts of a RAG setup. These are concrete examples of available mechanisms, not proof that a particular configuration will meet a company’s requirements.

In the policy scenario, the first failure could occur when the index contains both a superseded policy and its replacement but retrieval returns the older passage. Another retrieval failure could expose a restricted document because the request was not filtered using the employee’s access context.

The second failure occurs later. Even if retrieval returns the correct passage, the generated answer might convert “may be eligible after manager review” into “is eligible,” omit an exception or merge rules from incompatible versions.

This separation is also reflected in RAG evaluation research. ARES evaluates systems along distinct dimensions that include context relevance, answer faithfulness and answer relevance. Its research results do not establish business readiness for a company workflow, but the dimensional split supports an important design choice: evaluate the evidence supplied to the model separately from the answer produced from it.

A passage should clear the retrieval gate only when the team can answer all of the following questions.

Is the source authoritative for this question?

A policy draft, archived copy or departmental summary may contain relevant words without being authorized to decide the issue. Define authority for the bounded task before evaluation. Where multiple sources have different roles, specify which one controls and which ones only provide explanation.

Is the version applicable and sufficiently fresh?

Check effective dates, status and relevant organizational scope. “Newest document” and “applicable document” are not always identical. A historical policy may still apply to an earlier event, while a recently uploaded draft may not yet be in force.

Freshness also has a system dimension. If source changes are not reflected in the searchable collection within the task’s required interval, retrieval may faithfully return material that operations already considers obsolete.

Was the passage retrieved within the user’s permissions?

Access must be enforced during retrieval, not inferred from whether the final answer displays a citation. Microsoft’s documented security filter pattern for Azure AI Search stores principal identifiers in filterable fields and applies a filter to the query. The documentation is explicit that the principal is a string used by the filter, not authentication or authorization by itself, and that the filter must be applied to every relevant query in this pattern.

For an evaluation, include users with different permissions and verify the returned passages, not just the wording of the answer. Citation formatting cannot prevent an unauthorized passage from entering the generation context.

Does the collection cover the task?

Retrieval cannot find an authoritative answer that is absent from the eligible source set. Coverage checks should identify missing policies, attachments, regional variants and referenced definitions that are necessary to interpret the returned passage.

Is the passage relevant enough to support this question?

Inspect the text and its context rather than accepting a similarity or ranking score as a verdict. A high-ranked passage may share terminology while addressing another employee group, period or process.

If acceptable evidence is missing, conflicting, stale or inaccessible, the retrieval gate fails. The system should return a bounded no-answer or route the question for review instead of presenting the material as an adequate basis for a policy answer.

Once retrieval passes, review the generated response against the actual passages supplied to the model.

1. Map material statements to evidence. Each statement that could change the user’s understanding or action should be supported by a retrieved passage.

2. Check the strength of the wording. The answer should not turn permission into entitlement, possibility into certainty or guidance into a binding rule.

3. Preserve qualifications. Conditions, exceptions, scope limits and unresolved conflicts must remain visible when they affect the conclusion.

4. Keep versions compatible. The response should not silently combine material from policies that apply to different periods, regions or employee groups.

5. Make citations inspectable. A citation should identify the supporting source and passage clearly enough for verification. Its presence alone does not prove that the cited text supports the claim.

6. Check uncertainty and refusal behavior. If the evidence supports only a partial answer, the response should say so rather than filling the gap with plausible prose.

When an answer fails this gate, preserve the question, retrieved passages and failed response. Replace or withdraw the answer where appropriate, send source defects back to the source owner and send generation or citation defects to the answer-behavior owner. Retest the same case after correction so the team can see whether the relevant failure was actually resolved.

A demonstration usually shows questions the system is expected to answer well. An evaluation set must also contain cases capable of exposing weak evidence and unsupported conclusions.

Include questions that cover:

• routine requests with one clear source;

• ambiguous wording or missing user context;

• users with different access rights;

• policies with current and superseded versions;

• conflicts between summaries and controlling documents;

• questions that require several compatible passages;

• questions for which the eligible sources contain no adequate answer.

For each case, establish the expected evidence or expected refusal before reviewing the generated response. Then record retrieval and answer support as separate outcomes. A plausible answer with unsuitable passages fails retrieval even if its conclusion happens to be correct. A response that overstates suitable passages passes retrieval but fails answer support.

Automated evaluation can assist with repeated checks such as context relevance and faithfulness. ARES, for example, describes automated judges supported by a smaller set of human annotations. That does not remove the need for sampled human review of company-specific authority, applicability, permissions and consequential cases. Those judgments depend on the task and the organization’s own source rules.

Separate gates only help if failures have named destinations. Assign accountability across the evidence supply chain:

• Source authority and versioning owner: decides which records control, which versions apply and when old material should leave the eligible collection.

• Access owner: defines who may retrieve each source and confirms how identity and permission changes reach the retrieval layer.

• Ingestion and refresh owner: maintains the transfer from source systems into the searchable collection and handles failed or delayed updates.

• Retrieval evaluation owner: investigates missing, irrelevant, stale or improperly accessible passages.

• Answer review owner: assesses support, qualifications, uncertainty and citation clarity.

• Correction owner: coordinates withdrawal, replacement, escalation and retesting when an answer or source fails.

• Pilot decision owner: accepts, revises or stops the pilot against predefined conditions rather than relying on the implementation team’s overall impression.

The NIST AI RMF Playbook organizes voluntary suggested actions around Govern, Map, Measure and Manage. That framing is useful here as proportionate governance support: define accountability, map the task and its risks, measure the separate gates and manage identified failures. It is not a substitute for organization-specific legal, security or compliance review.

Document the handoff as part of the workflow. A failed freshness check belongs with the source or refresh owner. Unauthorized retrieval belongs with the access and retrieval owners. An unsupported conclusion belongs with answer review. The final decision owner needs visibility across all of them.

For a broader treatment of operational responsibility around AI-enabled workflows, see Designing AI operations without noise.

Set acceptance conditions before running the pilot. Thresholds should reflect the bounded task, the consequences of a wrong answer and any failures the organization considers unacceptable. Do not choose one universal score and discover after the evaluation what it concealed.

Use three possible decisions:

Approve

Approve a bounded pilot only when retrieval and answer support both meet their predefined conditions, required owners accept their responsibilities and no unacceptable failure remains. Keep the approval limited to the tested users, sources and actions.

Revise and retest

Revise when the weakness has a specific remedy and a credible retest path. Examples include correcting source status, repairing refresh behavior, applying access context consistently, improving retrieval for a defined question class or constraining answer wording to the evidence.

A revision is not complete when the configuration changes. It is complete when the failed cases are rerun and the affected gate clears its condition without creating another critical failure.

Stop or choose a simpler route

Stop the RAG route when authoritative evidence cannot be maintained, access restrictions cannot be applied reliably, unsupported answers cannot be contained or the task does not justify the operating burden. Return to search, curated material, deterministic lookup or maintained answers where those approaches satisfy the task more directly.

One retrieval score, citation rate, aggregate average or successful demonstration should not override a critical failure. The HELM evaluation methodology provides broader support for using multiple metrics to expose tradeoffs rather than allowing accuracy alone to dominate evaluation. For this RAG decision, the practical implication is narrower: keep the two gates visible and preserve unacceptable conditions as hard decision constraints.

Passing these technical acceptance checks shows that the bounded system met its defined evaluation conditions. It does not by itself demonstrate business benefit, guarantee security or establish legal compliance.

Choose one representative policy question from your existing workflow. Identify the current authorized source, inspect the evidence returned by the proposed route and record retrieval and answer support separately.

The observable result is simple: your team should be able to explain which gate passed or failed. That is a stronger basis for the policy-assistant decision than accepting a convincing response at face value.