Retrieval-augmented generation was sold as the fix for made-up answers. Give the model the right documents and it will stop inventing things. Anyone who has shipped one knows how that promise actually lands: the system now produces confident wrong answers with a citation attached, which is worse than before, because the citation makes it look checked.
Table of Contents
The confusion comes from treating RAG as a fact-checking step. It is not. RAG changes the question the model is answering. Instead of “what do you know about this”, it is asked, “what does this pile of text say about this?”. Both questions can be answered wrongly, and the second one has more ways to go wrong.
Here is where the errors actually come from, roughly in order of how often they explain a bad answer.
Failure one: retrieval found something relevant, not everything needed
The core assumption of RAG is that the passages most similar to the question contain what the question needs. That assumption is weaker than it looks.
A 2026 EMNLP findings paper that diagnoses evidence-based generation puts it directly: similarity retrieval finds any relevant chunk more reliably than it finds all the evidence required for multi-fact queries, and realistic top-k budgets impose a ceiling on how much evidence can arrive. The same work points out that chunking discards document-level context, so a passage that depends on a heading, a date, or a table three pages earlier arrives stripped of the thing that made it true.
This is why RAG systems answer simple lookups well and multi-part questions badly. “What is our refund window” retrieves cleanly. “Can a customer on the legacy plan get a refund after 40 days if they were migrated in March?” needs three facts that may live in three documents, and the retriever has no way to know that answering requires all three. Whatever it fails to fetch, the model fills in, and it fills in fluently.
Failure two: the model reads the middle of the context badly
Even when the right passage is retrieved, position matters. The widely cited “Lost in the Middle” work in TACL showed a U-shaped curve: models use information best when it sits at the start or the end of the input, and performance degrades when the needed fact is buried in the middle of a long context. That held even for models explicitly built for long inputs.
Pile twenty retrieved chunks into a prompt, and you have created exactly the conditions where a correct but unremarkable chunk gets ignored in favour of a wrong chunk that happened to land first or last.
The practical implication is unglamorous. Fewer, better-ordered chunks beat more chunks. If you are stuffing the context because “the model can handle 100k tokens”, you are taking on a quality cost you will not see in a demo, because demos ask one question with one obvious answer.
Failure three: misleading evidence is worse than no evidence
The failure that surprises people most is that retrieval can make a system less accurate than not retrieving at all.
A NeurIPS 2025 datasets and benchmarks paper evaluated RAG systems against supporting, misleading and unrelated evidence and found that, when exposed to potentially misleading retrievals, all the tested systems performed worse than their own zero-shot baselines. Human annotators, given the same evidence, still did better. In other words, a model handed a plausible but wrong document tends to adopt it, and the retrieval step has converted a knowledge problem into an authority problem.
Anyone who has watched a support bot confidently quote an outdated policy page knows this pattern. The retriever did its job by similarity. The document was on-topic. It was also from 2023, and nothing in the pipeline checked that against the question.
Failure four: nobody asked the model to check its own claims
Most RAG pipelines end with a model generating an answer. Almost none end with a step that decomposes the answer into individual claims and asks whether each one is supported by the retrieved text. That gap is where the citation-attached hallucination lives.
There is a growing body of work on exactly this. Claim-level verification frameworks break an answer into atomic claims, label each as entailed, contradicted or unsupported by the retrieved context, and map the decisions back to spans, which turns an opaque score into something you can audit. Chain-of-verification approaches do something similar by having the model plan verification questions and answer them independently of its first draft, which reduced hallucination in the published evaluation. Retrieval methods that generate an explicit rationale for why a passage was retrieved have also improved robustness to noisy retrieval.
None of these need a research team to implement. The smallest version is a second pass with a different prompt and the same context: list every factual claim in this draft that the provided documents do not support. It is cheap and it catches the specific class of error users complain about.
What actually reduces it, ranked by effort against payoff
Fix retrieval before touching the model. Most RAG hallucinations are retrieval bugs wearing a generation costume. Add hybrid search so exact terms like product codes and error strings are not lost to embeddings. Add a reranker over a larger candidate set. Decompose multi-hop questions into sub-questions and retrieve for each. If your questions are multi-part, this is the highest-return change available.
Filter and date your corpus. Misleading evidence is the worst failure mode, and it is usually a stale-document problem. Keep a last-updated field on every chunk, prefer the current version when duplicates exist, and remove retired pages instead of leaving them to win on similarity.
Order and trim the context. Put the highest-confidence passages at the start and the end. Cut anything below a relevance threshold even if it means sending fewer passages. It feels wrong to send less context; the position research says it often is not.
Require attribution per claim, not per answer. Ask for the supporting passage next to each claim, and treat a claim without one as a failure rather than a stylistic issue. The moment the model has to name its source for each sentence, the incentive to fill gaps disappears.
Build abstention into the product. A system that says “I could not find this in the documents” is more useful than one that guesses, and far more trusted after the first month. Give the model an explicit permission to refuse, and test that it uses it.
Evaluate at claim level. Answer-level “was this right” scores hide the problem. Label individual claims as supported, contradicted or unsupported on a sample of a few hundred real questions, and track the rate over time. Public benchmark families such as RAGTruth exist precisely because sentence-level faithfulness annotation is what surfaces these failures, and a small internal set built the same way will tell you more than any leaderboard.
What does not work as advertised
Adding more context is the most common non-fix. Beyond the position effect, extra passages add more opportunities for a misleading chunk to outrank a correct one.
Using a bigger model helps with fluency and reasoning and does almost nothing for faithfulness to a document. A larger model reproduces a wrong source more convincingly.
Treating citations as proof is the third. A citation shows which document was retrieved, not that the sentence next to it is supported by that document. Users read the presence of a link as verification, which is exactly why a claim-level check matters more than a prettier source list.
A one-week version
If you want to know whether your system has these problems, do four things. Take fifty real questions from your logs, including at least ten that are multi-part. Run them through the pipeline and record, by hand, whether each answer’s claims are supported. Then re-run with half the retrieved chunks and see whether the supported rate moves, which tells you whether you are over-stuffing context. Add a claim-checking second pass and re-run again. Finally, count how often the system should have abstained and did not.
That is a day of work and it will tell you more than another round of prompt tweaking. The pattern it usually reveals is the one worth internalising: the generator is rarely the main culprit, and the retrieval pipeline is rarely as good as its demo suggested.
If you are still choosing the shape of the system itself, the beginner-level explanation in generative AI and chatbots covers how retrieval fits alongside prompting and fine-tuning, and the support-desk case in AI chatbots in customer service shows why an unsupported answer costs more there than anywhere else. Teams wiring this into an older platform will recognise the data-quality problems described in integrating generative AI into legacy applications, which is usually where the misleading documents come from in the first place. If you are assembling this without a backend team, the build-versus-buy trade-offs in AI customer service and no-code tools are the next decision, and AI agents in the workplace covers what changes when the retrieval loop is run by an agent instead of a single prompt.





