In This Article
Key Takeaways
- A wrong answer that arrives with a citation is more dangerous than a refusal, because it survives review. NIST AI 600-1 names this failure class "confabulation" — confidently stated but erroneous content.
- Generation is rarely the broken layer. Parsing, chunking, embedding, retrieval, ranking, and context assembly all sit upstream, and a single test splits the pipeline in half.
- Pure vector search loses exact strings. The BEIR benchmark found BM25 to be a robust zero-shot baseline; hybrid retrieval plus a cross-encoder reranker is the standard repair.
- Approximate vector indexes trade recall for speed, and some apply metadata filters after the index scan — so a chunk can exist, be permitted, and still never be seen.
- Without a gold set of real questions mapped to the passages that answer them, every fix you make is a guess.
The answer is wrong and it looks right
Someone asks your assistant a question. It returns a clean paragraph, cites a document by name, and the answer is wrong. Not obviously wrong — wrong in the specific way that only a person who knows the subject catches. That person is usually senior, and now you are explaining the system instead of shipping it.
The first instinct is to blame the model, and it is almost always the wrong instinct. In a retrieval system the model does not know your corpus. It sees the handful of passages your pipeline handed it and writes the best answer it can from that material. Give it a stale passage and it writes a confident stale answer. Give it half a rule and it writes a confident half rule. NIST's Generative AI Profile (NIST AI 600-1) defines the failure class precisely: "Confabulation: The production of confidently stated but erroneous or false content," and notes that confabulations "are a natural result of the way generative models are designed."
So a confident wrong answer is a symptom, not a diagnosis. The useful question is which of the six layers above the model produced the bad input. Each layer fails with its own signature, and you can test them in an order that eliminates most of the search space in the first twenty minutes.
Layer 1: parsing and chunking destroyed the answer
Whatever your splitter throws away, no amount of retrieval quality gets back. This is the least glamorous layer and the most frequently broken one.
The classic failure is a fixed-size character splitter run over structured documents. It cuts mid-sentence, mid-table, and mid-clause. As an illustrative case: a policy states a rule in one paragraph and the exception that governs your situation in the next. The splitter puts them in adjacent chunks. Retrieval returns the first one, because it matches the question better. The system then produces an answer that is precisely, confidently wrong — it states the rule and omits the exception that reverses it.
Two more common variants. PDF table extraction that flattens a grid into a stream of numbers with no column headers, so a retrieved "chunk" is arithmetically meaningless. And headings stranded away from their body text, so the paragraph that only makes sense under "Applicability: contracts awarded before this date" gets retrieved without that qualifier attached.
The repairs are structural. Split on section and subsection boundaries first, paragraphs second, sentences only as a fallback. Attach the heading path, page number, and document identifier to every chunk as metadata. For tables, prepend the header row to each row-level chunk. Anthropic's engineering write-up on Contextual Retrieval pushes this further: generate 50 to 100 tokens of chunk-specific context from the parent document and prepend it before both embedding and keyword indexing. Anthropic reports the top-20 retrieval failure rate falling from 5.7% to 3.7% with contextual embeddings, to 2.9% when contextual BM25 is added, and to 1.9% with reranking on top — measured on their own evaluation set, so treat the shape of the result as the transferable part, not the exact figures.
Layer 2: embedding mismatch
Three failures hide here, and all three produce retrieval that is quietly mediocre rather than obviously broken.
Missing instruction prefixes. Many retrieval models are trained asymmetrically, with one prefix for queries and another for passages. The E5 model card is blunt about it: the prefixes are required, "otherwise you will see a performance degradation." In practice documents are embedded by an ingestion job and queries are embedded by the serving path, written months apart by different people, and the prefix silently gets dropped on one side.
The model changed and the index did not. Vectors from two different embedding models are not comparable. A partial re-index leaves you with a corpus where some vectors mean something different from the others. The giveaway symptom is quality that splits by document age rather than by question type.
Leaderboard rank is not your corpus. Public embedding leaderboards such as MTEB rank models on general benchmark tasks. Your corpus of agency policy, clinical protocol, or maintenance procedure is not a general benchmark. The BEIR paper (Thakur et al.) makes the caution concrete: across heterogeneous zero-shot tasks, "BM25 is a robust baseline," while dense models "often underperform" out of distribution. Rank on a leaderboard is a hypothesis about your corpus, not a measurement of it.
Layer 3: retrieval found the wrong thing, or found it and dropped it
Exact strings. Dense embeddings are built to blur near-synonyms together, which is exactly wrong for identifiers. Statute cites, memo numbers, part numbers, error codes, and ticket IDs differ by one character and mean entirely different things, and a vector index will happily return the neighbor. Lexical search does not have this problem. Run both channels and fuse the two ranked lists with reciprocal rank fusion — the method from Cormack, Clarke and Buettcher (SIGIR 2009), which in their experiments produced better results than any of the individual systems being combined.
Approximate index recall. HNSW indexes are approximate by construction. The pgvector documentation states it plainly: after adding an approximate index "you will see different results for queries," and the ef_search parameter — default 40 — buys recall at the cost of speed. If you are pulling a top-5 from millions of vectors at default settings, some correct chunks are never scored at all. That looks identical to a chunking problem from the outside.
Filters applied in the wrong order. pgvector also notes that with approximate indexes, filtering "is applied after the index is scanned." A permission filter, date filter, or document-type filter that runs after the scan can empty out a candidate list even though matching, permitted documents exist in the corpus. To a user this reads as "the system does not know about that document," when in fact the system found it and threw it away. In an access-controlled corpus this is also a correctness issue in the other direction, which is why the filter belongs inside the search rather than after it.
Layer 4: no reranking
Bi-encoder retrieval is a similarity lookup — the query and the document are embedded separately and never actually compared against each other. A cross-encoder reads the pair together and scores it. The Sentence Transformers documentation states that cross-encoders "achieve better performances than Bi-Encoders" but cannot be run across a whole corpus, and recommends the retrieve-and-rerank pattern: pull a large candidate set with the bi-encoder, then re-score those candidates with the cross-encoder.
The symptom this fixes is specific and easy to test for: the correct chunk is in your top 50 but not your top 5, so the generator never sees it. When that is your failure mode, reranking is the highest-value change available, because it alters only the ordering of results you were already retrieving.
Layer 5: context assembly and stale duplicates
Position matters. In "Lost in the Middle" (Liu et al., TACL), the authors found that "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts" — and that this holds "even for explicitly long-context models." Dumping forty chunks into the prompt in raw score order buries your best passage in the worst position.
Stale duplicates are the most common cause of a confident wrong answer in a real corpus. The 2019 procedure, the 2023 revision that superseded it, and a training deck that quotes the 2019 language are all sitting in the same index. Retrieval works perfectly and returns the wrong one. No embedding upgrade fixes this. No model fixes this. It is a records management problem wearing an AI costume, and it is why the very first thing worth auditing is how many chunks in your corpus plausibly answer the same question and whether they agree.
The repairs are version metadata carried from ingest, explicit supersession marking, a recency preference in ranking where it is appropriate, and — the part no software supplies — a named human with the authority to retire a document.
Layer 6: no grounding check
Nothing in a default pipeline verifies that the sentence the model wrote is actually supported by the text it was given. Two controls close that gap.
The first is citation validation: require span-level citations in the output format, then mechanically check that each claim maps to a retrieved chunk, and flag or drop the ones that do not. The second is abstention. The system has to be able to return "this is not in the corpus." A system that can never say that is not a retrieval system; it is a generator with retrieval decoration, and it will confabulate every time coverage is thin.
Then measure it. The open-source Ragas framework provides faithfulness (does the response stay true to the provided context without adding unsupported information), context precision, context recall, response relevancy, and noise sensitivity. The pairing that matters most for this article's problem is faithfulness against context recall: together they separate "we retrieved the wrong passage" from "we retrieved the right passage and wrote past it."
The diagnostic ladder
Run these in order. Each rung eliminates layers, so you stop as soon as one fires.
Build a gold set before you change anything
A few dozen real questions from real users, each paired with the exact passage a knowledgeable colleague says answers it, plus the expected answer. This is a half-day of subject-matter-expert time and it is the difference between engineering and guessing. Every rung below depends on it.
The ceiling test
Paste the correct passage into the prompt by hand and ask the question again. Right answer means generation is fine and retrieval is your problem. Still wrong means the issue is your prompt, your citation format, or your model. One test, half the pipeline eliminated.
Compare recall at 50 against recall at 5
If the gold passage is usually in the top 50 but not the top 5, the retrieval is finding it and the ranking is losing it. Add a cross-encoder reranker over the larger candidate set. If it is not in the top 50, keep climbing.
Check whether the chunk exists at all
Search the raw parsed text for the answer string. Not present means parsing or chunking destroyed it. Present but split across two chunks means your boundaries are wrong. Present as one clean chunk means the problem is embedding or indexing, not ingestion.
Probe with keyword search alone
Query the exact string using BM25 only. Found by BM25 but not by the vector index points at embedding mismatch or a missing lexical channel. Found by neither points at indexing or filtering — check filter order and your approximate-search parameters.
Audit for duplicates and supersession
For each gold question, count how many chunks plausibly answer it and check whether they agree. Two confident answers that disagree is a corpus problem, and no retrieval tuning will resolve it. Fix this before touching models.
Only now, score faithfulness continuously
Run your metrics against the gold set on every deployment and gate releases on them. A retrieval regression that ships silently is how a system that was trusted last quarter becomes a system nobody trusts this quarter.
Symptom to layer map
| Symptom | Most likely layer | First test |
|---|---|---|
| Cites a real document but states a superseded rule | Corpus: stale duplicates | Duplicate audit (rung 6) |
| Fails on statute cites, form numbers, part numbers | Retrieval: no lexical channel | Keyword-only probe (rung 5) |
| Right chunk sits in top 50, never in top 5 | Ranking: no reranker | Recall at 50 vs 5 (rung 3) |
| Answer is half right; the exception is missing | Chunking boundaries | Search the parsed text (rung 4) |
| Good on old documents, bad on new ones | Embedding or index version skew | Re-embed a known chunk and compare |
| Returns nothing for documents the user may see | Post-filtering or index recall | Check filter order and search parameters |
| Context is correct, answer still wrong | Generation, prompt, or citation format | Ceiling test (rung 2) |
What finished looks like
A RAG system you can put in front of a skeptical reviewer has four properties, none of which are about the model. Every answer traces to a passage the reviewer can open and read in its original context. The system can say it does not know, and does say it. Retrieval quality is a number on a dashboard rather than a shared impression. And a change that degrades that number fails a gate before a user ever sees it.
If you want the conceptual groundwork underneath any of this, our companion explainers on how RAG actually works, embeddings, and vector databases cover the mechanics, and LLM evaluation covers the measurement side in more depth.
Who builds this kind of thing
You have a retrieval system in front of real users, or about to be, and it is returning answers that are fluent, cited, and wrong often enough that someone senior has stopped trusting it. Nobody on the team can say which layer is at fault, so the fixes have been model swaps and prompt edits that did not move anything.
What Precision Federal builds for it
Precision Federal is a federal software and AI firm, and RAG systems for federal document corpora is one of its capability areas. The work is the whole pipeline rather than a model swap: structural chunking that follows section and paragraph boundaries and carries headings, page numbers, and markings as per-chunk metadata; hybrid lexical-plus-dense retrieval fused with reciprocal rank fusion; cross-encoder reranking over the top candidates; access-controlled retrieval where the permission filter is combined into the search instead of applied after it; provenance that ties each generated statement back to a source chunk, page, and document a reviewer can open; and a continuous evaluation harness built on a domain gold set that blocks regressions at deploy. Deployment paths are cloud, on-premise with self-hosted embedding and serving models, or split across both for corpora that cannot leave the boundary.
What an engagement looks like
A scoped assessment first: build the gold set with your subject-matter experts, run the ladder above against your live system, and deliver a written diagnosis that names the failing layer and what fixing it involves. If the diagnosis calls for a build, that is a separate, separately scoped piece of work. Nothing is quoted — price, schedule, or result — before the corpus has been looked at.
Where this is not the right fit
Three cases, said plainly. If nobody in your organization owns the corpus — if no one has the standing to mark a 2019 procedure superseded — retrieval engineering will not save you, and we would rather tell you that than bill for it. If your corpus is mostly scanned images, engineering drawings, or handwriting, that is a document-understanding problem with a different risk profile and it should be piloted narrowly before anyone commits to a program. And this is a federal practice: for a general-purpose commercial chatbot there are firms better suited to the work, and we will say so.
Sources: NIST AI 600-1 — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile; Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (TACL); Thakur et al., "BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models"; Cormack, Clarke and Buettcher, "Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods" (SIGIR 2009); Anthropic — Introducing Contextual Retrieval; Sentence Transformers — Cross-Encoders; pgvector documentation; intfloat/e5-large-v2 model card; Ragas — available metrics. Analysis and framing by Precision AI Academy.
Common questions
Why does my RAG system give confident wrong answers? Because the generator answers using whatever chunks it was handed. If retrieval supplies a stale, partial, or unrelated passage, a well-behaved model still writes a fluent paragraph around it. NIST AI 600-1 calls this class of failure confabulation — "the production of confidently stated but erroneous or false content."
How do I tell whether the problem is the model or the retrieval? Run the ceiling test. Paste the correct source passage into the prompt by hand and ask again. Right answer means generation is fine and retrieval is your problem; still wrong means the prompt, the citation format, or the model. One test eliminates half the pipeline.
Do I actually need a reranker? If the correct chunk regularly appears in your top 50 but not your top 5, yes. Sentence Transformers documentation notes cross-encoders achieve better performance than bi-encoders but cannot scale across a whole corpus, so the recommended pattern is a large bi-encoder candidate set re-scored by a cross-encoder.
Why is pure vector search not enough? Dense embeddings handle meaning well and exact strings poorly, so statute cites, form numbers, and error codes get blurred together. The BEIR benchmark found BM25 to be a robust zero-shot baseline while dense models often underperform out of distribution. Running both channels and fusing the rankings covers both cases.