RAG vs Extraction: When to Use Each

Two questions, two machines

Picture a folder with four thousand contract PDFs in it, and two questions about them.

The first question: what is the period-of-performance end date on one specific contract in that pile? That is one fact. It is written down, once, in one sentence, in one of those documents. It needs to end up in a date field, and if it is wrong by a year, something downstream breaks.

The second question: across these four thousand awards, how has the language around data rights changed? No single sentence answers that. The answer is a paragraph. Producing it means reading widely and then saying something about the whole body of text.

These need different machinery. The first is a job for extraction: locate the exact text in the exact document and hand back a pointer to it. The second is a job for retrieval-augmented generation — RAG: find the relevant material and let a generative model write prose conditioned on it.

The two get confused because they share a first step. Both search. The difference is what happens after the search, and that difference is not a detail — it changes what the output is made of, how you test it, and whether anyone can check it.

What RAG actually does

RAG was named in Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (NeurIPS 2020). The paper describes models that combine parametric memory — what the model learned during training and carries in its weights — with non-parametric memory, which in their system was a dense vector index of Wikipedia reached through a neural retriever.

The mechanics in practice are simple to describe. A user question comes in. A retriever scores the corpus and returns the top handful of passages. Those passages get placed into the model's prompt. The model writes an answer.

The retriever is a real design decision, not plumbing. Sparse keyword scoring such as BM25 matches on terms. Dense retrieval embeds questions and passages into vectors and matches on proximity; Karpukhin et al. (EMNLP 2020) reported that dense passage retrieval beat a strong Lucene-BM25 system by 9–19 percent absolute in top-20 passage retrieval accuracy on open-domain question answering benchmarks. Worth reading precisely: that result is on Wikipedia-style benchmarks. A corpus dense with part numbers, contract identifiers, and program acronyms behaves differently from encyclopedia prose, which is why hybrid keyword-plus-vector setups exist and why the only trustworthy answer for your data is to measure retrieval on your data.

The defining property of RAG is what comes out: new text. The model can paraphrase, aggregate across passages, hedge, order things, and summarize. It can also write a sentence the retrieved passages do not support. All of those follow from the same capability.

What extraction actually does

Extraction covers a family of tasks whose output is a location rather than a composition.

Extractive question answering takes a passage and a question and selects a contiguous span of the passage as the answer. The shape of the task was fixed by SQuAD, and its second version, Rajpurkar, Jia and Liang (2018), added over 50,000 unanswerable questions — forcing systems to determine when no answer is supported by the paragraph and abstain. Devlin et al. (NAACL 2019) reported SQuAD v1.1 test F1 of 93.2 and SQuAD v2.0 test F1 of 83.1 for BERT fine-tuned on the task.

Sequence labeling tags every token and groups the tags into entities. The canonical framing is the CoNLL-2003 shared task on language-independent named entity recognition (Tjong Kim Sang and De Meulder, 2003). Modern field extraction — dates, dollar amounts, organizations, identifiers — is usually this task wearing a business costume.

The property that matters: an extraction result is a document ID, a start and end offset, and the literal string at that offset. Nothing was written. Something was pointed at. Which means correctness can be checked mechanically — open the file, go to the offset, compare the bytes. No second model, no human judgment, no rubric.

The decision rule

Ask what consumes the output.

• It lands in a typed field — a date, a dollar amount, a CAGE code, a part number, a name: extraction.
• It gets compared to a rule or a threshold downstream: extraction. Rules need values, not prose about values.
• A reviewer will need it highlighted in the original document: extraction, because that highlight is the offset.
• A person reads it as a paragraph: RAG.
• The question spans many documents and no single passage holds the answer: RAG.
• The answer requires counting, comparing or ranking across sources: extract the values first, then compute over them. This is a query, not a generation problem, and treating it as generation is how systems end up confidently wrong about arithmetic.

A useful and slightly deflating observation: a fair share of projects that begin as "we need RAG over our documents" turn out, on inspection, to be a database query over fields that nobody has extracted yet. The chat interface was masking a missing schema.

How each one fails

RAG's failures cluster in the generation step, not the search step.

Faithfulness. Maynez et al. (ACL 2020) found that neural abstractive summarization models are highly prone to hallucinating content unfaithful to the input document, with substantial hallucinated content in every system they evaluated by human annotation. Handing a model the right passage does not oblige it to stay inside that passage.

Citations are claims, not proof. The ALCE benchmark (Gao et al., EMNLP 2023) measures fluency, correctness and citation quality together; on the ELI5 dataset, even the best models evaluated lacked complete citation support 50 percent of the time. That is one benchmark from one moment in a fast-moving field, and current models differ. The durable lesson is structural: a printed citation is the model asserting that a source supports a sentence, and assertions need checking.

Position sensitivity. Liu et al., "Lost in the Middle" (TACL 2024), found performance is often highest when relevant information sits at the beginning or end of the input context and degrades significantly when the model has to reach for information in the middle — including in models built for long contexts. Stuffing fifty passages into the prompt is not free. We wrote more about that budgeting problem in context engineering.

The retrieval ceiling. If the retriever never returns the passage, no model quality recovers it. Measure recall at k separately before blaming the generator; teams routinely tune prompts for weeks to fix what was a retrieval miss.

Extraction's failures are the mirror image, and they are more boring — which is part of the appeal.

It cannot answer what is not written. If the answer requires joining a clause on page 3 to a definition on page 40, a span extractor returns nothing, and returning nothing is the correct behavior. It needs labeled examples, or at minimum an annotated evaluation set, where a RAG pipeline can be stood up over an unlabeled corpus in an afternoon. Boundaries are genuinely ambiguous — deciding where "period of performance" starts and stops is annotation-guideline work, and it is where the real hours go. And abstention has to be designed in from the start; that was the whole point of SQuAD 2.0. An extractor that always returns its best guess is more dangerous than one that returns nothing, because nothing is visible and a plausible wrong span is not.

The pattern that uses both

Most durable systems are not a choice between the two. They are a sequence: retrieve, then extract, then — only if a human needs prose — generate over the extracted spans alone.

In that arrangement the generative model stops being the source of facts and becomes the formatter of facts that already carry addresses. Each sentence is bound to the span that produced it, so "where did that come from?" resolves to a file and an offset rather than to a document a reviewer must now read in full.

The trade-off is real and worth stating plainly. This is more engineering than pointing a model at a vector store, and it constrains the writing — output built from bound spans reads tighter and less freely than output a model composes at will. If the requirement is a discursive narrative that ranges across a subject, this pattern will feel stiff. If the requirement is a conclusion someone has to defend line by line, the stiffness is the product.

When the output has to survive review

Traceability stops being an engineering preference the moment someone else has to sign for the output. The NIST AI Risk Management Framework (NIST AI 100-1, January 2023) frames its MEASURE function around evaluating systems against trustworthiness characteristics and providing a traceable basis to inform decisions. The practical translation is a single test: pick any sentence in the output and ask where it came from. Extraction answers with a document and an offset. Generation with citations answers with a document that a human now has to open and read to see whether it actually supports the sentence.

Deployment boundaries push in the same direction. In the Department of Defense, a DoD Provisional Authorization for a cloud service offering is issued by the DISA Authorizing Official based on FedRAMP plus additional DoD security requirements at Impact Levels 4, 5 and 6, per DISA's Cloud Authorization Process briefing (June 2024). The architectural consequence is concrete: inside such a boundary, every call to an outside API is a question someone has to answer. Task-specific extraction models are often small enough to run where the data already lives; the largest generative models may simply not be available there. That constraint sometimes settles the design before accuracy ever enters the conversation.

What we will not tell you. We are Precision Federal, and our own work sits mostly on the extraction side of this line — small models that read a body of data and produce a written conclusion with every statement traced back to the exact record it came from. That is a bias, so here are its edges. We will not call a system "traceable" because a model printed citations; without a mechanical check that the cited span exists and contains the claim, that is a promise, not a control. We do not issue authorizations, we are not a third-party assessment organization, and we do not sell an accredited platform — we build the model layer so it can live inside a boundary that someone else accredits. And we will not tell you extraction is the answer when what your users actually need is synthesis across a body of documents. Sometimes the honest answer is RAG with a human reviewer and a modest claim about what it produces.

Sources: Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020); Karpukhin et al., Dense Passage Retrieval (EMNLP 2020); Rajpurkar, Jia & Liang, Know What You Don't Know (SQuAD 2.0, 2018); Devlin et al., BERT (NAACL 2019); Tjong Kim Sang & De Meulder, CoNLL-2003 Shared Task; Maynez et al., On Faithfulness and Factuality in Abstractive Summarization (ACL 2020); Gao et al., Enabling LLMs to Generate Text with Citations (EMNLP 2023); Liu et al., Lost in the Middle (TACL 2024); NIST AI 100-1, AI Risk Management Framework; DISA, DoD Cloud Authorization Process (June 2024). Analysis and framing by Precision AI Academy.

Common questions

What is the difference between RAG and extraction? Retrieval-augmented generation retrieves relevant passages and lets a generative model write new text conditioned on them; the output is prose. Span extraction returns a pointer into a specific document — a document ID, a character offset, and the literal text at that location. RAG produces sentences. Extraction produces addresses.

Is RAG just search plus a chatbot? Structurally, yes: a retriever selects passages and a generative model conditions on them. The original formulation by Lewis et al. (NeurIPS 2020) describes it as combining parametric memory — the model weights — with non-parametric memory, a dense vector index accessed by a neural retriever. The design work lives in the retriever and in how retrieved text is presented to the model, not in the chat window.

Does adding citations make a RAG system grounded? Not by itself. A citation is a claim by the model that a source supports a sentence, and that claim can be wrong. The ALCE benchmark (Gao et al., EMNLP 2023) found that on the ELI5 dataset even the best models evaluated lacked complete citation support 50 percent of the time. A citation becomes evidence only when something mechanically verifies that the cited span exists and contains the claim.

When should I use extraction instead of RAG? When the answer is a value that lands in a typed field, gets compared to a rule, or has to be highlighted in the source document for a reviewer. Use RAG when a person needs a paragraph that synthesizes across many documents and no single passage contains the answer.