Cost-Aware LLM Pipelines: Match Compute to Criticality

Most items in a large workload are easy, and a few are genuinely hard. The cheapest reliable pipeline is the one that spends accordingly: a deterministic fast path for the easy majority, the full agent loop reserved for the long tail, and routing that decides which is which before you pay for the expensive option.

flowchart LR
  R["request"] --> RT["router"]
  RT -->|"most"| CH["cheap path
deterministic, majority"]
  RT -->|"few"| AL["full agent loop
long tail"]
  classDef key fill:#e8f1fb,stroke:#1e1e1e,color:#1e1e1e
  classDef good fill:#eaf6ec,stroke:#1e1e1e,color:#1e1e1e
  class RT key
  class CH good
  linkStyle 1 stroke:#2f9e44,color:#2f9e44

The mistake: one expensive path for everything

The first version of an LLM pipeline usually sends every input through the same heavy treatment: the largest model, a long multi-step agent loop, every tool the agent might need, and a generous token budget so nothing gets truncated. It works, the quality looks good on a demo set, and then it meets real volume. Cost scales with traffic at the per-item price of the most expensive treatment you have, latency is dominated by the worst case applied to every case, and you discover the bill grows faster than the value because most of what you are paying for is unnecessary. The problem is not that cost scales linearly - a stateless per-request pipeline should - it is the coefficient: a high fixed per-item cost charged uniformly, even on inputs that never needed it.

The reason it is wasteful is that workloads are rarely uniform. In most real streams, a large share of inputs are easy - they match a known shape, the answer is unambiguous, and a far cheaper method would get them right. A smaller share are genuinely hard and need the full reasoning loop. Spending top-tier compute on the easy majority is the single largest avoidable cost, and it is invisible until you measure the distribution of difficulty instead of the average.

So the framing I start from is not 'which model is best' but 'what is the cheapest correct treatment for each class of input, and how do I route to it.' Match compute to criticality. The hard cases deserve the expensive path; the easy ones do not, and pretending they all deserve it is how you build a pipeline you cannot afford to scale.

Tier the work: deterministic fast path, agent loop for the long tail

The core pattern is tiering. Put a cheap, deterministic stage in front of the expensive one and let it resolve everything it confidently can. The fast path can be ordinary code - schema checks, lookups, rules, a cached or precomputed answer - or a small, cheap model with tight output constraints. It handles the easy majority at a fraction of the cost - near-zero latency when it is plain code, still far cheaper when it is a small model - and it never invokes the agent loop at all.

The expensive tier - the large model, the full tool-using agent loop, the higher token budget - is reserved for what the fast path could not resolve. That is the long tail: ambiguous inputs, ones that need several rounds of structured retrieval via explicit tool calls and reasoning, ones where the rules disagree or the confidence is low. Because the tail is a small fraction of volume, you can afford to be lavish there: more steps, more tool calls, more careful prompting, because you are spending it on the cases that actually need it.

The load-bearing decision is the escalation boundary - what the fast path treats as 'I am sure' versus 'escalate.' Set it too eager and you push easy work to the expensive tier and lose the savings; set it too permissive and the fast path emits confident wrong answers. I treat that boundary as a tunable threshold, not a fixed rule, and I want a measured false-pass rate on the fast path before I trust it. The fast path is only allowed to be cheap on inputs where being cheap does not cost correctness; everything else escalates, and escalation is a feature, not a failure.

Cut the cost of the expensive tier itself

Tiering decides how often you pay full price. The next set of levers reduce what full price is. The first is prompt caching. Agent prompts tend to carry a large, stable prefix - system instructions, tool definitions, schemas, shared context - and a small variable suffix that is the actual input. Providers can cache the static prefix so you are not reprocessing it on every call, subject to a cache window, so the win depends on sustained volume hitting the same prefix before the entry expires. Structure the prompt so the stable part comes first and is byte-identical across calls, and the savings are real on any workload with a fixed harness and high call volume. The discipline is to stop concatenating dynamic content into the prefix; keep the cacheable part cacheable.

The second is inference routing by difficulty. Tiering already routes between code and the agent; inside the agent tier you can route again between model sizes - but only when a cheap signal already separates the bands, such as the fast path's confidence score, the input length, or which rule fired. Given that signal, a capable small model handles the middle band and the largest model is reserved for the genuinely hard residue. This is the same match-compute-to-criticality idea one level down, and it pays off only when the classification is near-free; if telling the bands apart costs as much as the routing saves, skip it.

The third is batch versus on-demand. If a workload is not latency-sensitive - overnight reprocessing, backfills, periodic re-scoring of a large set - run it through a batch path rather than the interactive one. Batch inference is materially cheaper per token because it relaxes the latency guarantee, and a lot of fleet-scale LLM work is genuinely offline. Reserve on-demand for the calls a human or a live system is waiting on, and push everything else to batch.

Spend the savings on quality where it counts

Cost discipline is not about making the system cheaper at the expense of being worse. It is about freeing budget so you can be more expensive exactly where it improves the result. Once the easy majority is handled by a cheap path, the long tail gets a larger slice of compute than a flat pipeline would ever have allowed - more reasoning steps, more verification, a second pass when confidence is low. Tiering does not lower your ceiling on hard problems; it raises it, because you stopped wasting the budget on easy ones.

This only holds if you measure quality per tier, not in aggregate. A blended accuracy number hides the failure mode that matters: the fast path quietly getting a class of inputs wrong because nobody escalated them. So I track the fast path's error rate separately from the agent tier's, and I watch the escalation rate over time. A drifting escalation rate is an early signal that the input distribution moved and the boundary needs retuning - the same way any threshold drifts when the world underneath it changes.

The honest tradeoff to name is that tiering adds moving parts - a router, a boundary to calibrate, per-tier metrics to maintain. That complexity is justified at fleet scale and overkill for low volume. The decision rule is simple: if a single expensive path is affordable at your volume, keep it; the moment cost scales past the value, tier it, and tier it where the difficulty actually splits.