Many Retrieval-Augmented Generation (RAG) projects look impressive in demonstrations but face user skepticism once deployed: "Where did this sentence come from?" "Why wasn't the clearly existing policy found?" "Did it just make up an answer?" The issue may not lie with the model itself; it could stem from documents, chunking strategies, retrieval mechanisms, reranking processes, prompt engineering, or citation display. For RAG to move into production, robust evaluation and hallucination prevention mechanisms are essential.
Target Audience
This guide is designed for developers who have already built a knowledge base Q&A prototype, enterprise IT professionals, heads of customer service knowledge bases, and compliance teams. If you haven't yet set up RAG, start with our basic setup tutorial; however, if your team plans to use it in real-world scenarios, the evaluation step cannot be skipped.
Evaluation Objectives
RAG quality can be broken down into four questions: Was the correct information found? Is it ranked at the top? Did the model answer based on that information? Are citations accurate? The corresponding metrics are recall, reranking quality, faithfulness, and citation accuracy. Relying solely on whether users feel "the response is good" is insufficient, as fluent errors pose the greatest risk.
Step 1: Build a Test Set
The test set must derive from real-world queries rather than editorial speculation. Sources can include customer service logs, employee search terms, historical tickets, and product FAQs. Ideally, each query should be paired with a standard answer, correct sources, acceptable variations, and conditions for refusal to answer.
For example, for the question "What is the accommodation standard for out-of-town business travel?", the standard answer must specify which policy document, specific clause, and applicable city tier it references. If a particular city is not covered by the policy, the entry should be marked as "requires manual verification" rather than allowing the model to guess.
The test set does not need to be large initially; 50 to 100 high-frequency questions are sufficient to expose significant issues. After launch, add failure cases from weekly operations to the test set to establish a regression evaluation cycle.
Step Two: Evaluate Retrieval
First, set aside the model’s answer and examine only the retrieval results. For each query, check whether the correct source appears within the top five or top ten chunks. If it is missing entirely, the issue lies in document processing or the retrieval strategy; if present but ranked too low, the problem is with re-ranking; if the correct chunk exists yet remains unused by the model, the fault likely stems from the prompt engineering or generation phase.
Retrieval evaluation can be semi-automated: compare retrieved results against standard source IDs, supplemented by manual spot checks. In enterprise scenarios, retrieval failures typically arise from unreasonable chunking, lost titles, missing synonyms, overly strict permission filtering, or outdated documents.
Test Set JSON Example
[
{
"question": "What is the accommodation allowance for out-of-town business travel?",
"expectedSourceId": "policy-travel-2026",
"expectedAnswerKeywords": ["city tier", "accommodation standard", "reimbursement cap"],
"shouldRefuse": false
},
{
"question": "Will the accommodation allowance change next month?",
"expectedSourceId": null,
"expectedAnswerKeywords": [],
"shouldRefuse": true
}
]Simple Evaluation Script Concepts
for (const item of testset) {
const chunks = await retrieve(item.question);
const hit = chunks.some((chunk) => chunk.sourceId === item.expectedSourceId);
console.log({
question: item.question,
retrievalHit: item.shouldRefuse ? "skip" : hit,
topSources: chunks.slice(0, 5).map((c) => c.sourceId),
});
}The interface validation screenshot must include the evaluation table, details of failed cases, and a side-by-side comparison highlighting citations. Without screenshots of failure scenarios in RAG tutorials, readers will struggle to learn how to pinpoint issues.
Step 3: Verify Answer Faithfulness
Faithfulness refers to whether an answer contains only content supported by the source materials. The prompt must explicitly instruct the model to state "I don't know" when information is insufficient and prohibit filling gaps with common knowledge. After generation, a separate evaluation model can check whether every key sentence in the answer is supported by the cited snippets. For high-risk scenarios, manual spot checks are still recommended.
A common issue involves models stitching together two adjacent clauses to create new rules or mixing old policy versions with new ones. Solutions include version filtering, source prioritization, date-based prompting, and citation constraints.
Step 4: Design a Refusal Mechanism
A robust RAG system must know how to say "I don't know." Declining to answer is not a failure; it protects the user. You can establish rules such as refusing when retrieval scores fall below a threshold, when fewer than one source citation exists, when the query involves content outside permitted access levels, or when the question demands predictions, advice, or commitments unsupported by available materials.
Avoid cold, robotic refusal messages. Instead, respond with: "The current knowledge base lacks sufficient evidence to answer this; please refer to [Specific Document] or contact [Department Name]." This guides users on their next steps.
Step 5: Close the User Feedback Loop
Every response should include feedback options such as "Helpful," "Inaccurate," "Wrong Source," or "Outdated Document." These inputs must feed into a backend system categorized by issue type. Often, the fault lies not with the model but with missing documents in the knowledge base, unclear document titles, or outdated policies lacking maintenance. A RAG system can inadvertently expose gaps in an organization's knowledge management practices.
Monitoring Metrics
After launch, it is recommended to continuously track six key metrics: query volume, hit rate, refusal rate, follow-up question rate, negative feedback rate, and human handoff rate. Query volume indicates whether the system is being used; hit rate reflects knowledge base coverage; a refusal rate that is too low may suggest the system answers too confidently, while one that is too high could indicate poor retrieval performance; a high follow-up question rate suggests answers are unclear; a high negative feedback rate points to insufficient credibility; and the human handoff rate helps identify which issues require additional documentation or tool integration.
Additionally, regularly sample citations for review. Many systems exhibit accurate citations in their initial launch phase, but after several iterations involving changes to chunking strategies, re-ranking algorithms, or prompts, citation accuracy may begin to drift. Conducting weekly spot checks on 20 to 50 high-frequency Q&A pairs incurs minimal cost yet enables early detection of quality degradation.
Defining Responsibility Boundaries
RAG systems should clearly disclose the source and scope of their answers within the user interface. For internal policy inquiries, a disclaimer such as "Generated based on current knowledge base materials" is appropriate; for customer service queries, phrasing like "The following information has been compiled by the system from public policies; complex cases will be transferred to human agents" works well. Avoid creating an impression that AI-generated answers are inherently equivalent to final rulings. Clearly defining these boundaries of responsibility actually fosters greater trust.
For internal teams, it is also essential to clarify who holds the ultimate authority for interpretation. For instance, HR-published documents should serve as the definitive source for personnel policies, product manuals for technical specifications, and operational announcements for customer service guidelines. While RAG can enhance lookup efficiency, it cannot replace formal policy publication workflows. The sooner these boundaries are explicitly defined, the fewer disputes will arise later.
If answers may be used in external communications, it is advisable to retain a complete evidence package in the backend: the user's query, retrieved snippets, model-generated response, citation sources, and the final sent version. In the event of a dispute, this allows teams to trace why the system provided a specific answer at that time and determine whether the issue originated from the knowledge base, retrieval process, or human review stage.
Common Pitfalls
| Symptom | Layer to Diagnose | Fix Strategy |
|---|---|---|
| Correct document not in top 5 | Retrieval layer | Adjust chunking, hybrid search, synonym expansion |
| Correct document in top 5 but answer is wrong | Generation layer | Prompt requires sentence-by-sentence citation; prohibit adding external information |
| Entire document cited but specific paragraph missing | Citation layer | Save citations to include chunk ID and paragraph position |
| Model answers when it should refuse | Refusal layer | Must refuse for low retrieval scores, no citations, or out-of-permission queries |
| Performance degrades over time in production | Operations layer | Add negative feedback weekly to the test set for regression evaluation |
Failure case recording template:
{
"question": "What is the accommodation allowance for out-of-town business travel?",
"expected": "cites travel-policy-2026, clause 4",
"actual": "cited the superseded 2024 policy",
"failedLayer": "retrieval",
"fix": "archive the old policy and filter on updatedAt during retrieval"
}Alternative Approaches
For high-risk scenarios such as legal, medical, or financial compliance applications, position RAG as a "retrieval and drafting assistant," with final answers confirmed by human experts. If the required answer is highly structured, implement a rule engine or database query directly instead of relying on model generation. For Q&A based on public documents, you can also validate demand first using AI search products enhanced with traditional search capabilities.
Summary
The core of preventing hallucinations in RAG is not a single prompt but an entire quality system: real-world test sets, retrieval evaluation, faithfulness checks, citation constraints, refusal mechanisms, and user feedback. Only when answers are supported by their sources will enterprise users feel confident entrusting their knowledge bases to AI.