Most RAG Hallucinations Are Extraction Errors: Seven Patterns for a Typed Generation Contract

The naive pipeline asks the LLM for an answer and trusts what comes back. There is no separation between found and invented, no place to record confidence, and no way to flag not found as a first-class outcome. We fill the schema instead: every field is a question the pipeline asks, and every answer is checkable.

Below are the seven patterns that keep generation on the typed-contract side. The first six are the contract itself. The seventh names a constraint the contract must respect when the model is small.

The answer schema is a contract: structured retrieval in, a typed Pydantic object out, with citations, fidelity flags, and pipeline-feedback fields, never “the answer as a string”. Everything the generation brick does is in service of filling that contract, field by field.

Take “what is the premium amount?”. The naive baseline returns a sentence: "The premium is $124 per month, per the contract.". To use that downstream you have to parse the number back out of the prose, and nothing tells you where it came from or how sure the model was. The series returns a row instead: Amount(value=124.0, currency="USD", unit="month", evidence=[Span(line_start=42, line_end=42, quote="premium of $124 / month")], answer_found=True, confidence=0.95). The value is already typed, the evidence points at the exact line, and the self-assessment fields travel with it. The caller reads structured values, not text it has to re-read.

Ask for Amount(value, currency, unit) and let Python compare with a visible exchange rate; computation inside the model erases the audit trail and silently does conversions no human can replay. The LLM extracts, Python computes.

Take “is the premium higher than 100 euros per month?”. Ask the LLM directly and it answers "yes, the premium is around 130 EUR / month", having converted $124 to euros with a rate it never shows you. You cannot check the arithmetic, and next month the same question might convert at a different invisible rate. The series extracts the raw value instead, Amount(value=124, currency="USD", unit="month"), and lets Python do the comparison with a rate that sits in the audit log, 1 USD = 0.92 EUR. That gives 114 euros, so the answer is True, and every step, the extraction, the rate, and the comparison, can be replayed and checked by hand.

The LLM inside the retrieval scope cannot see whether the next page continues a list; retrieval pulls one extra page and the pipeline checks for a section boundary, so completeness is deterministic and grounded in structure. A model that can only see what was sent to it cannot judge what was missing.

Take “list all exclusions”. The exclusions section runs from page 7, and the next section, “Limitations”, starts on page 10, so the answer lives on pages 7 to 9. Naive RAG hands those pages to the LLM and asks “is the list complete?”. The model can only see what it was given, so it reads pages 7 to 9 and says yes, with no way to know whether a tenth exclusion was ever retrieved. The series checks the structure at retrieval time instead: does the next page still belong to the exclusions section? If it does, fetch it; once it crosses into “Limitations”, stop. Completeness becomes a fact about where the section ends, not a guess the model makes about text it cannot see.

Off-corpus, partial, and complete each route to a different next action, and two flags force clearer decisions than a single scale. The float-confidence frame collapses three different outcomes into one number.

Take “list all exclusions” again, but this time retrieval found page 7 and missed page 8. A single confidence = 0.6 tells the orchestrator almost nothing: it cannot tell whether two of three exclusions came back or whether the whole answer was invented. The series splits that into two booleans. answer_found = True says the question is answerable from this document, and complete_answer_found = False says the list that came back is partial. Those two flags map to three clear next moves: keep retrieving when the answer is found but incomplete, ship when it is found and complete, and refuse when it is not found at all. A single float collapses those three outcomes into one number and loses the decision.

A dispatcher composes BASE + one shape fragment + optional constraints and records which fragments applied, so a wrong format six months later is traceable, unlike a mega-prompt that grows uncommented clauses. A single prompt that everyone keeps appending to is a common RAG codebase smell; composing it from named fragments is what keeps each answer’s prompt reconstructable.

Take “effective date?”, whose answer shape is Date. A naive baseline runs one generic prompt that always asks for “an answer”, whatever the question. The series composes the prompt from parts instead: the BASE prompt, plus a Date shape fragment that says "return YYYY-MM-DD", plus the DateAnswer schema. The audit log records exactly which parts were used, prompt_id=BASE@v3 + shape=date_v2. So when the model returns “July 1st 2026” six months later and someone has to explain the wrong format, the team can rebuild that exact prompt from the log and reproduce the bug, instead of guessing which clause of a thousand-line mega-prompt fired.

The schema already constrains the work, so the extra “thinking” adds latency and wanders off the schema without adding precision. Reasoning models earn their place on open-ended tasks; on schema-constrained extraction, they are over-engineered.

Take “what is the deductible amount?”. A team hoping for cleaner extraction switches to a reasoning model. It thinks for eight seconds and returns the same number a plain model returned in one, because the schema had already pinned the output to a single typed field. The reasoning tokens cost latency and money and bought nothing, since there was no open-ended judgment to make. The stance that follows is simple: use the smallest model that fills the schema reliably, and save reasoning for the steps that genuinely need it, such as the LLM arbiter at the end of retrieval.

A frontier model can fill a compound schema (extract + convert + format) in one call; a small model cannot, and asking for it invents the derived fields silently. Model size sets the granularity of the contract, not the shape of it.

Take “is the premium higher than 100 euros per month?” with a compound schema, PremiumComparison(raw_amount: Amount, converted_eur: float, exchange_rate: float, over_100_eur: bool). GPT-4.1 fills every field correctly in one call: it extracts $124/month into raw_amount, converts to 114.08 EUR with a visible rate, and returns over_100_eur=True. Hand the same schema to llama-3.2-3B and the model returns raw_amount populated, converted_eur invented at 117.6 (from an imaginary 0.95 rate the log never carries), and over_100_eur=True derived from that invented rate. The small model does not fail JSON validation. It fills every field. Three of the four are fabricated.

The fix is to decompose the contract into stages the small model can handle in isolation. First call: extract Amount(value=124, currency="USD", unit="month"). Python takes over: rate lookup (logged, 1 USD = 0.92 EUR), conversion (124 * 0.92 = 114.08 EUR), threshold (114.08 > 100 = True). Optional second call if a natural-language summary is needed: hand the completed compound row back with a plain format prompt. Same final row, same accuracy, no fabricated intermediate.

The rule: the size of the model determines the number of extraction calls, not the shape of the schema. Big model = one dense call. Small model = a chain of typed extracts with Python computing in the middle and one optional format call at the end. The typed contract survives either way; only the granularity changes.

Article B05 (picking a model) benchmarks the granularity trade-off across a fleet of thirteen models (link to come).

The seven patterns share one move: refuse the LLM-as-oracle frame, and treat generation as filling a typed contract. The schema is the contract; citations and self-assessment fields are first-class outputs; the validator runs before the user sees anything. Pattern 7 adds the operational nuance: the same contract holds across model sizes, but a small model needs it broken into stages the model can fill without inventing derived values. The deep-dives (8A, 8B, 8C) ship runnable code on real documents; this piece is the catalogue that names each pattern and points to the code that implements it.

The typed-contract discipline (Pydantic schema + citations + self-assessment fields) holds in every domain. What changes per question is the schema shape. The contract pattern stays the same. Five sectors below, five answer shapes, one validator running the same checks across all of them.

The validator runs the same checks across all five rows: line spans within document bounds, verbatim quotes match cited lines, format constraints respected (a Duration actually parses as a duration, an Amount carries currency and unit). The legal row demonstrates the two-boolean split (Pattern 4): the answer was found in this document but is not complete without a cross-reference, so the orchestrator continues retrieval instead of shipping a partial answer. The medical row shows the audit trail (Pattern 1): the caveat captures a known-unknown the LLM could not resolve.

The numbered articles develop each pattern in code, with runnable notebooks:

Most writing on LLM generation comes out of chatbots and open-ended text. The series assumes something different: the answer must be auditable, and the LLM must refuse to invent.