every turn of a web agent does not deserve a frontier model. "what's your return policy" and "yes, the blue one" are not the same class of problem as "plan a three-step refund across two orders", yet the naive build sends all of them to the same expensive endpoint. cost and latency bleed out on turns a much smaller model would have nailed.
the fix is routing. a cheap first pass handles the turn; an escalation signal decides whether the hard ones get handed up. done well, the user never sees the seam.
the two passes
a small, fast model (local, or a cheap hosted tier) takes every turn first. same system prompt, same tools, same context. for the bulk of turns, acknowledgements, single unambiguous tool calls, short clarifications, it answers directly and the turn is over at a fraction of the cost and latency.
when the turn is hard, the small pass does not answer. it escalates, and the frontier model takes over with full state.
deciding to escalate
this is the part that actually matters, and the part most write-ups wave away. escalation is layered, cheapest signal first:
- heuristics on the input. very long context, more than one tool plausibly in scope, an explicit multi-step request ("first... then..."). cheap to compute, catches the obvious hard cases before any model runs.
- a structured self-assessment from the small pass. the small model returns its answer plus a field,
confidenceorneeds_escalation, in its structured output. low confidence routes up. models are not perfectly calibrated, so this is a signal, not a verdict. - tool-call ambiguity. if the small pass wants to call a tool but the arguments are underspecified, or it proposes two conflicting calls, escalate rather than guess.
the rule of thumb: escalate on uncertainty, not on certainty of difficulty. a false "this is easy" is the expensive failure, a wrong action shipped to the user. a false "this is hard" only costs you one unnecessary big-model call. bias the threshold accordingly.
the seam
the handoff has to be invisible. the frontier model receives the full conversation, the small pass's draft if any, and the reason it escalated. it does not restart the turn from scratch. if the user notices a stutter, a tone shift, or a repeated question at the boundary, the routing has failed even when the answer is right.
what it costs (your numbers)
| | small pass only | escalated | blended |
|---|---|---|---|
| share of turns | [%] | [%] | 100% |
| p50 latency | [ms] | [ms] | [ms] |
| p95 latency | [ms] | [ms] | [ms] |
| cost / session | [£] | [£] | [£] |
the table above is a template, not a claim. fill it from your own traffic. the only thing that generalises is the shape: in a typical support or commerce agent, most turns are easy, so the blended cost and latency sit much closer to the small pass than the big one. measure your own escalation rate before you quote a saving.
when not to do this
- low-traffic agents. the routing layer is complexity; if you serve a handful of turns a day, the saving will not pay for the extra moving part.
- domains where every turn is genuinely hard. if there is no easy class to skim off, you are escalating everything and paying for two models to do one job.
- anywhere a wrong action is catastrophic and you cannot tolerate a miscalibrated "easy". raise the threshold until escalation is almost free, or skip routing entirely.
routing is an optimisation. reach for it once the agent works and the bill or the latency is the problem, not before.
building an agent and watching the bill climb? that's our day job.