On this page
A reference is the beginning of the check
Hypothetical situation: an employee asks a policy assistant how to submit an expense. The answer includes a citation and reads convincingly. The cited document was replaced last month, however, and the new process requires a different approval route. The reference helps a reviewer find the source. It does not, by itself, establish that the source remains authoritative.
This is a content-lifecycle test, rather than another discussion about choosing a vector index or chunk size. The question is whether the application handles changes to the information that people are expected to follow. Start with one policy, its owner and a clear expected answer.
NIST’s July 2024 Generative AI Profile recommends reviewing and verifying sources and citations during pre-deployment measurement and ongoing monitoring. The exercise below is an editorial way to practise that principle. It does not claim that a particular retrieval product automatically checks document authority or recognises every superseded source.
Include a question whose correct response depends on the policy’s effective date. A reviewer should be able to identify the applicable instruction from the source evidence without seeing the generated answer first. This provides an independent expectation for the test. Where the business itself has not decided which version applies, resolve that ownership question before treating a fluent answer as correct.
Establish which version governs the answer
Ask the information owner to identify the approved source, its effective date and the question it should answer. Keep publication date, effective date and ingestion time separate. A document may be uploaded recently while describing an old policy; another may be approved today but take effect next month. The answer needs the relevant business date, not simply the newest file timestamp.
Build a small controlled collection with invented policy material. Give the documents clear version identifiers so the team can inspect which passage was retrieved. Avoid using a live policy change as a casual experiment. The purpose is to observe behaviour without causing employees to follow incorrect instructions.
Write the expected response before running each case. Include when the system should stop and refer the reader to the policy owner. If two equally authoritative sources disagree, a confident blended answer is not a reasonable acceptance condition. The owner must decide how the conflict is resolved.
Exercise three changes deliberately
First replace the policy with a revised version that changes one meaningful instruction. Run the same question again after the team’s documented refresh process. Record the retrieved version and the answer. Do not invent an expected refresh time: establish the application’s actual process with its owner and test against that commitment.
Next introduce an approved-looking document with a conflicting instruction. Check whether the result exposes the disagreement or quietly selects a convenient passage. Finally remove a document from the permitted collection. Test both retrieval and the citations shown to the user. An old answer remaining in a conversation may need separate treatment from a fresh request.
The worksheet records intended checks, not guarantees about cache invalidation, indexing or deletion. Those behaviours belong to the actual implementation. Use its logs and source records to explain what happened, and ask an authorised engineer to inspect them without exposing sensitive content to the review group.
| Change | Expected behaviour to agree | Evidence to inspect |
|---|---|---|
| Superseded source | Use the applicable approved instruction | Retrieved version and effective date |
| Conflicting sources | Expose unresolved authority rather than inventing agreement | Both passages and escalation outcome |
| Removed source | Follow the agreed removal and refresh policy | Fresh retrieval result and citation target |
Give content changes an operational owner
A retrieval team can operate the index without owning the policy inside it. Assign separate responsibilities for approving the source, refreshing the collection and investigating a wrong answer. Make the handover explicit enough that a policy owner knows whom to contact after an urgent amendment.
Retain the test question, expected answer, document versions and observed result. Repeat the relevant cases after changes to ingestion, retrieval or source selection. Keep a short record of accepted limitations so support staff do not have to rediscover them during an incident. A successful result for one policy does not establish that every document collection behaves identically.
For a practical lab, ask participants to challenge a perfectly formatted answer whose citation points to the wrong effective version. The useful skill is following the evidence far enough to decide whether the instruction should be used. The team should finish with a source-maintenance action and a named owner, rather than a general request to make the model more accurate.
Sources & further reading (1)
Historical evidence checked . Review practices and hypothetical exercises are editorial recommendations, not product guarantees.
