Loop Engineering for RAG Generation: An LLM Cascade from a Cheap Local Model Up to a Hosted Flagship

There are two reasons to care about model selection, and they reinforce each other.

The cost angle is direct. Flagship models cost an order of magnitude more per call than a small local one. On an extraction job with thousands of field-level calls, or a corpus assistant answering thousands of questions a day, the model bill dominates the running cost. Most of those calls do not need a flagship. Routing the simple ones to a cheaper model is money left on the table until you do it.

The loop angle is about correctness under that constraint. You cannot cut cost by guessing that a small model will cope. You start with a model, check whether its answer holds, and escalate to a stronger one only when the check fails. The loop is what makes the cost saving safe: the cheap model is the default attempt, not a gamble, because a failed attempt is caught and retried on a bigger model rather than shipped.

Cost pushes you down the model ladder. The loop stops you from falling off it. This cheap-first, escalate-on-failure shape has a name in the literature: an LLM cascade.

The design in Section 3 rests on two measured facts. Take them first, because they are the reason the cascade is shaped the way it is.

Look at what twenty local models did on the raw one-shot task: read the retrieved snippet, return the typed value. Same prompt, same fields, temperature 0.

Three conclusions come straight off that chart, and each one shapes the design.

Bigger is not better. A 4B model (qwen3:4b) and a 7B (mistral:7b) top the local field. A 12B and a 14B (gemma3:12b, phi4:14b) sit below them. Two models from the same family, one twice the size of the other, tie. Parameter count is a poor predictor. The model family and how it was trained matter more. This is exactly why the loop must not climb a blind size ladder: the next rung up is often not the next size up.

There is a floor. The sub-2B models fall off a cliff (qwen2.5:1.5b at 12%, llama3.2:1b at 6%), and a 0.5B model we tried returned empty JSON on most fields: not a wrong answer, no answer at all. Starting there burns a loop iteration on a rung that was never going to hold. Criteria exist to start above the floor, not at it.

Cost is not speed. The hosted flagship answered in about 1.4 seconds per field, faster than most local 7B-to-14B models (roughly 2.6 to 7.6 seconds each on a single consumer GPU). The small-local play buys a lower bill and documents that never leave the machine. It does not buy latency. If speed is the constraint, the cheap local model is often the wrong default, and that belongs in the dispatcher’s criteria.

One more sobering number. The best small local model, run on its own on the raw task, got roughly a third of the fields right. Fast (a few seconds a field) and not good enough to ship. A small local model is viable only when the loop backs it: a tight retrieval snippet, the right prompt content, validation on every field, and code doing the hard formatting. The next three sections are that backing.

Before spending a bigger model, spend the prompt. The single largest jump in the whole benchmark did not come from more parameters. It came from putting the task’s business vocabulary, the glossary the fields assume, into the prompt.

Both models jump. The small local model goes from 38% to 62% correct. The flagship goes from 62% to 100%. The flagship is not magic without the glossary either. It needed the definitions to finish the job.

The concrete failure this fixes is vocabulary the model does not share with the document. A field labelled deductible per claim in one document is retention in another and a bare number in a third; a premium is called one thing by one insurer and another thing by the next. A small model shown the raw snippet guesses. Hand it the definition of the field it is filling, the synonyms, the units, and the guess turns into a read. This is the same brick-2 vocabulary that lifts retrieval, reused at generation time, and it is cheaper than any model upgrade because it helps every rung at once.

Here is the mechanism the two facts point to. Pick a starting rung by criteria, validate its output and escalate only on failure, and split the task instead of escalating when the failure is a hard transformation. Three moves, in that order.

The starting model is not the smallest one you have. Three criteria set a sensible floor, so the loop begins at the right rung rather than the bottom.

Confidentiality. A confidential document cannot be sent to a hosted API at all. That forces a local model regardless of what would be cheapest or strongest in the cloud, and the starting choice becomes the best local model that fits the machine. (Detecting and labelling a document as confidential is a later-volume concern; this article assumes the label arrives as input.)

Typed-output reliability. The series answer is always a typed object. The extraction contract from the benchmark is small and strict: a pure number, its currency, the precision qualifier kept separate, and a boolean for whether the field was there at all.

class ExtractedValue(BaseModel):
    value: float | None
    currency: str | None
    precision: str | None
    found: bool

Strong models emit that cleanly. Weaker and smaller ones emit it less reliably, dropping the currency, folding "per event" into the number, or inventing a value when they should have set found=False. So a small starting model is viable only when paired with validation: you do not trust its typed output, you check it (Article 8C) and let the failure drive the escalation.

Transformation complexity. Some jobs hide a hard one-shot transformation that trips small models even when retrieval was perfect. That is the third criterion, and it has its own fix in §3.3.

The dispatcher that reads these three is small.

class GenerationPlan(BaseModel):
    model: str
    needs_validation: bool
    needs_step_split: bool
    max_escalations: int = 2

def plan_generation(doc, shape):
    if doc.confidential:
        return local_plan(shape)
    return tier_plan(shape)

local_plan pins the best local model and sets needs_validation=True, because a small model’s typed output always gets checked. tier_plan starts from Article 6C’s suggested tier and only asks for validation when that tier is small. Both read needs_step_split off the answer shape, the third criterion, developed in §3.3.

With a starting model chosen, generation becomes a short bounded loop, not a single call.

def generate_with_escalation(question, context, plan, ladder):
    for model in ladder.from_(plan.model, up_to=plan.max_escalations):
        answer = generate(question, context, model=model)
        if validate(answer, context):        # Article 8C
            return answer
    return NotAnswered()

Run the chosen model. Validate the result with the checks from Article 8C: is the typed shape correct, do the cited spans exist, does the quoted text actually appear in the source. If validation passes, ship, and the cheap model just did a flagship’s job for a fraction of the cost. If it fails, escalate: rerun the same field on the next model up, and validate again. The loop is bounded, so a pathological field cannot spin forever; it escalates to the strongest model and, if even that fails, returns a typed not answered rather than a wrong value.

This is the generation-side echo of Article 10. There, a page starts on the cheap parser and only escalates to a heavier one when a cheap check says the parse is not good enough. Here, a field starts on the cheap model and only escalates when validation says the answer is not good enough. Same loop-engineering shape: pay for the expensive tool only on the cases that need it, and let a deterministic check, not a guess, decide which cases those are.

Escalating to a bigger model is not the only fix for a failing one, and it is not always the right one. Some failures are a complex transformation, not a weak model, and the cure is to decompose the step rather than pay for more capacity.

The worked example is a monetary amount in a clean numeric format. Shown 3M, a small model may not reliably return 3000000, a three followed by six zeros; it stumbles on doing the recognition and the scaling in one shot. A bigger model papers over it, but so does splitting the transformation: let the model only point at the source line, and let deterministic code turn 3M into 3000000. Each half is easy. The model reads, the code computes, and the hard formatting never depends on model capacity at all.

That split has a second payoff: the raw document can stay entirely local. The model picks a line, code formats it, and only anonymous fragments are ever eligible to leave for a hosted formatter downstream. But the split is not a free lunch, and the benchmark is blunt about it. Running the fully-local “model points, code formats” architecture lowered accuracy on the same fields compared with letting a capable model extract and format in one shot. The reason is simple: pointing at the right line is itself hard, and a wrong line dooms the field before the code ever runs.

So the split is a real tool with a real cost. Reach for it when the failure is a hard transformation the model keeps missing, or when confidentiality forces the document to stay local. Measure the accuracy you trade for it. Do not assume decomposition is always a win; on a task where a capable model can extract and format cleanly in one pass, one pass wins.

The loop therefore has two escape hatches when validation fails, not one: escalate the model, or split the task into steps the current model can handle. The dispatcher’s needs_step_split flag records which one this field gets.

This selection is not a new idea bolted on. It closes a loop the series already opened.

Article 6C (dispatch) reads a suggested model tier off the parsed question: a simple factual lookup and a multi-hop synthesis carry different tiers before generation ever runs. That is the question-side signal. This article adds the document-and-answer-side signal (confidentiality, typed-output reliability, transformation complexity) and, crucially, the feedback: 6C proposes a starting tier, generation runs it, and Article 8C’s validation confirms the tier was enough or triggers the escalation. The three fit together: 6C suggests, generation attempts, 8C judges, the loop escalates.

Model choice is a brick decision, not a constant. The cheap model is the right default when a criteria-driven dispatcher picks a sensible starting rung, validation guards the output, and a bounded loop escalates only the cases that fail, splitting the task instead when the failure is a hard transformation rather than a weak model. The sweep is the reason to trust the shape and to distrust the shortcuts: size does not order accuracy, a small local model on its own gets a third of the fields, and the biggest single lever is the vocabulary you put in the prompt, not the parameter count you pay for. Cost comes down because most fields never leave the cheap rung. Correctness holds because the ones that need more get it, on evidence, not on a hunch.

6. Further reading and sources

The series articles this cascade sits between, already published:

On the model-cascade idea, the external anchors: