Most of your AI platform's traffic doesn't need a frontier model

There’s a specific line item I keep finding in AI platform bills, and it’s always the same shape.

Some enormous fraction of total spend — half, two-thirds, once considerably more — goes to a frontier model doing a task like: read this alert and tell me if it’s a database problem, a network problem, or a deploy problem.

Three-way classification. Against a fixed taxonomy. Being served by the most expensive model available, at roughly the cost of a small country’s coffee budget, because that’s the model that was wired up when someone built the feature in an afternoon eighteen months ago and it worked, so nobody touched it again.

I don’t think this is a failure of discipline. I think it’s a missing architectural construct.

---

The year small models got good

The context has genuinely shifted, and it’s worth being specific about how, because “small models are getting better” has been said every year and mostly meant nothing.

What changed in 2026:

• The economics gap widened into a canyon. Serving a 7B model runs roughly 10–30x cheaper than a 70–175B model. That’s not a margin you optimize around; it’s a different business.
• Domain-specific small models started winning outright. There are now credible reports of ~2.6B-parameter models outperforming models 250x their size on narrow domain tasks. Specialization beats scale when the domain is narrow enough — which was always theoretically true and is now practically demonstrable.
• The hardware showed up. Snapdragon X2 on Windows laptops, Snapdragon 8 Gen 4 on Android flagships, around 60 TOPS, no discrete GPU required. Local inference stopped being a demo.
• Forecasts followed. Dell called 2026 “the year of the SLM”; widely cited projections have organizations using small task-specific models at roughly 3x the rate of general-purpose LLMs by 2027.

Fine. But “use smaller models” is advice, not architecture, and advice doesn’t survive contact with fifteen teams shipping features on their own timelines. The interesting question is structural: why does model choice keep getting made once, badly, and never revisited?

Because in most organizations it isn’t a platform decision. It’s a string in a config file.

---

Model tier as a platform primitive

Here’s the construct I’d argue for: expose tiers, not model names.

Tier	What it is	Fits
T0	No model. Code, regex, lookup table.	Deterministic parsing, known enumerations, anything with a spec
T1	Small task-specific model (~1–8B), often fine-tuned	Classification, extraction, routing, reformatting, fixed rubrics
T2	Mid-size general model	Summarization, drafting, straightforward multi-step tasks
T3	Frontier model	Open-ended reasoning, novel situations, ambiguous input
T4	Frontier + extended reasoning	Genuinely hard planning, deep analysis, high-stakes one-offs

T0 is not a joke and it’s not filler. A meaningful share of what teams are paying a language model to do is if/else with extra steps and a nondeterminism budget attached. “Parse this well-formed JSON and pull three fields” is a T0 task that I have, more than once, found running on T3. The cheapest, fastest, most reliable model is the one you didn’t call.

Making the tier the interface changes three things:

Model choice becomes reviewable. “This service is on T3” is a claim someone can challenge in a design review. “This service calls claude-opus-5” is a config value nobody reads.

Upgrades stop being migrations. New model lands, you evaluate it, you place it in a tier. Applications inherit it. Nobody opens fifteen PRs.

Cost attribution gets a shape. Spend by tier tells you something actionable. Spend by model name tells you what you already knew.

---

Routing, honestly

The naive version of this — a classifier that predicts difficulty per request and routes accordingly — mostly disappoints, and I’d rather say why than have you find out.

Predicting whether a specific request needs a bigger model is roughly as hard as answering it. The classifier is wrong often enough to matter, and now you have two problems: the model output is bad, and you can’t tell whether it’s the model or the routing. Debuggability gets materially worse.

What works better is routing by task type, with confidence-based escalation as a safety valve:

"""Tier routing for an AI platform. Route by task type; escalate on
low confidence. Every decision is recorded so the routing policy
can be audited and revised from data rather than from opinion."""

from dataclasses import dataclass, field
from enum import IntEnum


class Tier(IntEnum):
    T0_DETERMINISTIC = 0
    T1_SMALL = 1
    T2_MID = 2
    T3_FRONTIER = 3
    T4_REASONING = 4


@dataclass
class TaskPolicy:
    """Set from evaluation results, not from intuition."""
    name: str
    default_tier: Tier
    max_tier: Tier                      # escalation ceiling — cost containment
    escalate_below_confidence: float | None = None
    eval_accuracy: dict[Tier, float] = field(default_factory=dict)
    last_evaluated: str = ""            # ISO date; stale policies are a bug


POLICIES = {
    # Bounded taxonomy, high volume — small model, escalate rarely.
    "alert_classification": TaskPolicy(
        name="alert_classification",
        default_tier=Tier.T1_SMALL,
        max_tier=Tier.T2_MID,
        escalate_below_confidence=0.75,
        eval_accuracy={Tier.T1_SMALL: 0.94, Tier.T2_MID: 0.96, Tier.T3_FRONTIER: 0.96},
        last_evaluated="2026-07-15",
    ),
    # Structured input, structured output — no model required.
    "extract_deploy_metadata": TaskPolicy(
        name="extract_deploy_metadata",
        default_tier=Tier.T0_DETERMINISTIC,
        max_tier=Tier.T1_SMALL,
        last_evaluated="2026-07-15",
    ),
    # Was T3 when it shipped. The July eval says T2 now clears the bar —
    # this one shows up in demotion_candidates() until someone acts on it.
    "summarize_incident_timeline": TaskPolicy(
        name="summarize_incident_timeline",
        default_tier=Tier.T3_FRONTIER,
        max_tier=Tier.T3_FRONTIER,
        eval_accuracy={Tier.T1_SMALL: 0.78, Tier.T2_MID: 0.93, Tier.T3_FRONTIER: 0.94},
        last_evaluated="2026-07-15",
    ),
    # Open-ended, high stakes, low volume — pay for the good one.
    "incident_root_cause": TaskPolicy(
        name="incident_root_cause",
        default_tier=Tier.T3_FRONTIER,
        max_tier=Tier.T4_REASONING,
        escalate_below_confidence=0.60,
        eval_accuracy={Tier.T1_SMALL: 0.41, Tier.T2_MID: 0.67, Tier.T3_FRONTIER: 0.88},
        last_evaluated="2026-07-15",
    ),
}


@dataclass
class Decision:
    task: str
    tier: Tier
    escalated_from: Tier | None = None
    reason: str = "policy default"


def route(task: str, prior: Decision | None = None,
          confidence: float | None = None) -> Decision:
    """Return the tier to serve this request at.

    Call once for the initial decision. If the response carries a
    confidence signal below the policy threshold, call again with
    `prior` and `confidence` to get the escalated tier.
    """
    policy = POLICIES.get(task)
    if policy is None:
        # Unknown tasks default high and get flagged. An unrouted task
        # is a gap in the policy table, not a reason to guess cheap.
        return Decision(task=task, tier=Tier.T3_FRONTIER,
                        reason="no policy defined — review required")

    if prior is None:
        return Decision(task=task, tier=policy.default_tier)

    threshold = policy.escalate_below_confidence
    if threshold is None or confidence is None or confidence >= threshold:
        return prior

    if prior.tier >= policy.max_tier:
        return Decision(task=task, tier=prior.tier, escalated_from=prior.tier,
                        reason=f"low confidence ({confidence:.2f}) but at max tier")

    return Decision(task=task, tier=Tier(prior.tier + 1), escalated_from=prior.tier,
                    reason=f"confidence {confidence:.2f} < {threshold}")


def demotion_candidates(min_accuracy: float = 0.90) -> list[tuple[str, Tier, Tier]]:
    """Tasks whose evaluations say a cheaper tier would still clear the bar.

    Run this after every model release. The boundary moves; your
    config does not move by itself.
    """
    out = []
    for policy in POLICIES.values():
        cheaper = [
            tier for tier, acc in policy.eval_accuracy.items()
            if tier < policy.default_tier and acc >= min_accuracy
        ]
        if cheaper:
            out.append((policy.name, policy.default_tier, min(cheaper)))
    return out

Three deliberate choices in there.

max_tier exists because unbounded escalation is how a cost optimization becomes a cost incident. A task that escalates on 80% of requests is paying the small-model bill plus the large-model bill plus a round trip, and is strictly worse than having gone straight to T3. Escalation rate is a metric you must graph, not a mechanism you install and forget.

Unknown tasks default high, not low. A task with no policy is a gap in your table. Failing expensive is recoverable; failing wrong quietly is not.

And demotion_candidates() is the function that makes the whole thing worth building. Which brings me to the part everyone skips.

---

The eval gate is the actual product

Tiering without evaluation is just a more elaborate way to guess.

You need, per task: a set of real production examples with known-good outputs, a scoring function, and a scheduled run against every tier. That’s it. It’s unglamorous and it is the entire difference between a platform capability and a nice diagram.

The output is a table you can defend:

Task	T1	T2	T3	Assigned	Rationale
alert_classification	0.94	0.96	0.96	T1	+2pt not worth 20x
incident_root_cause	0.41	0.67	0.88	T3	small models fail structurally
extract_deploy_metadata	0.99	0.99	0.99	T0	it’s a parser

Note the middle row. Sometimes the frontier model is correct and the answer is “keep paying.” A tiering exercise that always concludes “use the small one” isn’t measuring anything.

And re-run it. The boundary moves. Every model release shifts which tasks a small model can handle, always in the same direction. A task that genuinely needed T3 in early 2025 is a live demotion candidate now, and the only way anyone finds out is a scheduled job that checks. Without it you get one-time savings and then eighteen months of drift — which is the exact failure this whole post is about, just relocated one level up.

This is the same discipline as evaluating an AI SRE agent before it touches production, the same lesson ITBench keeps teaching, and the same reason I keep arguing that harness engineering is where the durable work is. The model is the commodity. The apparatus that tells you which model to use, and notices when the answer changes, is not.

---

What edge does to the architecture

If T1 can run on the device, some of your traffic can stop being traffic.

The consequences are less exotic than the pitch suggests, and better:

• Compliance gets simpler by subtraction. Data that never leaves the device isn’t a data-residency question. This is the strongest argument for edge inference in regulated environments and it has nothing to do with cost.
• Marginal cost approaches zero. The user’s hardware is already paid for.
• Offline works. Which matters enormously in some domains and not at all in others — worth knowing which one you’re in before designing for it.

The trade you’re accepting: you now have a fleet management problem shaped like mobile app distribution. Model versions in the wild you can’t force-update. Capability that varies by device generation. Debugging inference you can’t observe. That’s a real cost, and it’s the reason edge belongs in the architecture when the compliance or offline argument carries it — not when the cost argument does.

---

Where I’d start

Not with a router. With an inventory.

List every place your organization calls a model. For each: what task, what model, what volume, who chose it, and when. That’s a half-day of grep and interviews, and I have never seen it fail to produce at least one genuinely startling line.

Then take the top three by spend and just build eval sets for them. Not a platform. Not a routing layer. Three eval sets. Run them against a small model and see what happens.

You’ll learn within a week whether tiering is worth building for your workload — and if it is, you’ll have built the hard part first, which is the part that makes the rest trustworthy. If it isn’t, you spent a week and found out, which is a fine outcome and considerably cheaper than a routing layer nobody trusts.

The mistake I’d most want to steer you away from is building the elegant router before the boring measurement. That’s the version that gets deployed, becomes load-bearing, and is quietly bypassed by every team within a quarter — because nobody could tell them, with evidence, that the tier they were assigned was the right one.

---

Related: GPU utilization is a lying metric: the unit economics of self-hosted inference · What ITBench says about AI SRE · When not to use AI in production SRE · Context engineering: the window is a budget, not a bucket

Sources: The Best Open-Source Small Language Models (BentoML) · Small Language Models on Edge Devices, 2026 · Small Language Models and Edge AI: The 2026 Shift to Local Intelligence · Small Language Models: Comprehensive Guide 2026