Test Retrieval Before You Rewrite the Prompt

A support assistant gives an outdated cancellation policy. Adding “be accurate” to the prompt may change the wording without fixing the source of the error. A retrieval-augmented system has at least two opportunities to fail: it can deliver the wrong evidence, or it can misuse the right evidence.

Make a small evidence set

Choose questions with known, authoritative answers. For each, record the document, revision and passage containing the answer. Include questions that should have no answer in the collection. Track access scope too: a successful search for a restricted document is a security defect if the requester cannot read it.

Capture the actual retrieved passages in rank order for a failing question. Is the current policy absent from the index? Did chunking separate an exception from its rule? Did a nearly identical old document outrank the new one? These are different defects with different fixes.

Run the two-stage experiment

First, measure whether the expected passage appears within the number of passages sent to the model. Then supply the correct passage directly in a controlled run. If the answer improves, focus on indexing, retrieval or ranking. If it still fails, inspect the answer instructions, context order and output validation.

In the cancellation example, the index might contain both policy versions. Adding a current-version filter can outperform a more elaborate prompt. However, a historical question may legitimately need the old policy, so make the date context explicit.

Keep citations testable

A citation should identify a passage that supports the specific claim. Merely naming a document is insufficient. Track retrieval misses, unsupported generated claims and correct abstentions separately. That breakdown tells the next engineer which component to change and which regression test should fail if the bug returns.

Further reading

Lewis et al.: Retrieval-Augmented Generation

Related guide: Compare Language Models on the Work You Actually Do.