Somewhere in your org, an AI agent has production authority because someone flipped a setting. A vendor console, a Helm value, an IAM policy — autonomy granted as configuration. Now ask the follow-up question: what takes it away? If the answer involves a human noticing, a meeting, and a change ticket, you don’t have a governance mechanism. You have a toggle and a hope.
In the reliability gap I argued that agents should earn authority with measured reliability — and that promotion up the autonomy ladder should be tied to data, not roadmap pressure. This post is the mechanism that makes that enforceable. It’s the subject of a paper I’ve been writing, and SRE handed us the design twenty years ago: the error budget.
The problem is longitudinal, and our controls are point-in-time
The emerging agent-governance stack is good at one kind of question: is this action allowed right now? Policy engines gate execution paths. Risk pricing charges side effects against reserves. Approval flows route scary actions to humans. All of it is per-action, point-in-time.
None of it answers the question that actually decides whether your agent fleet is safe in month six: has this operator’s measured track record earned the standing authority it holds today? An agent that was reliable at deployment silently degrades — a model update, context rot, an environment change — and every per-action gate keeps waving through actions that are individually permissible from an operator that is collectively drifting. That’s how you get the slow-motion version of the agent-sprawl incident: forty agents, all “governed,” none of them being watched over time.
SRE hit the structurally identical problem with software releases: teams shipping features have no natural brake until reliability data pushes back. The error budget was the fix — a quantified allowance whose exhaustion automatically throttles risk-taking, agreed in advance, enforced without a meeting.
Transplant it.
The Autonomy Error Budget
The mechanics, adapted for operators instead of releases:
Partition authority into action classes. Read/diagnose, reversible mitigations, service-impacting mutations, irreversible actions — split by blast radius and reversibility. Each class gets its own SLO, budget, and rung on the ladder. An agent can be autonomous for reversible mitigations and advisory for irreversible ones at the same time. Authority is not monolithic.
Define the SLI as operational correctness. For each class: the fraction of executed actions that addressed the actual condition, violated no policy, and needed no reversal or human correction. Adjudicating that is real work — some wrongness surfaces hours later — so treat adjudication as an explicit, audited pipeline stage, not an operator self-report.
Burn the budget by blast radius, not by count. A wrong read costs nothing; a wrong failover costs an outage. Weighting burn by severity turns the budget into a harm allowance rather than an error tally — the key domain adaptation from classical error budgets, where failed requests are near-fungible.
Two quieter rules turned out to be load-bearing when I measured the design in simulation:
- Denominate the allowance in action volume. A fixed “0.5 harm units per month” allowance calibrated at low traffic gets spuriously exhausted by a perfectly healthy operator the moment incident volume quadruples. Classical budgets get this for free (they’re a fraction of requests); transplants have to earn it:
allowance = (1 − SLO) × expected weighted actions per window. - Only autonomous executions burn the budget. A wrong proposal that a human approved executed under the human’s authority — it feeds the promotion-evidence stream, not the budget. Burn it anyway and you create a demotion spiral: an operator dropped to supervised mode can never climb back, because its budget keeps burning on actions it no longer controls. Under naive burn semantics, the simulated cost of governing a chronically unreliable operator was 5× the optimum — entirely from this one attribution mistake.
The Authority Ladder: demote fast, promote slow
Budget state drives a four-rung state machine per action class: Autonomous → Supervised → Advisory → Disabled. The rungs aren’t the novelty — the transition function is.
Demotion is immediate and mechanical. Budget exhaustion, or a fast-burn alert (consumption running at 10× nominal), drops the operator a rung. No deliberation, no meeting. This is burn-rate alerting with the response inverted: classical burn-rate alerts page a human about a service; here the alert acts on the operator first — demote — and pages second.
Promotion is slow and evidence-gated. A rung up requires a full window of budget health plus affirmative evidence at the target level — supervised-mode concordance, shadow-mode performance. Absence of recent failure is not competence.
The asymmetry looks like bureaucratic caution. It’s actually stability engineering — hysteresis against noisy reliability signals. In the simulation study, a symmetric variant that promoted as eagerly as it demoted oscillated hard: 16 authority transitions versus 1.4 for a borderline operator under just 2% adjudication noise, with 2.5× the realized harm. And when the operator had genuinely degraded, the symmetric ladder re-promoted it into the damage every cycle and lost on both harm and total cost. Fast-down/slow-up is a small steady-state toil premium purchased against exactly the scenario budgets exist for.
Budgets compose with the gates you already have
Budget-derived authority doesn’t replace per-action governance — it bounds it:
effective authority(action) = min( per-action gate(action),
budget-derived authority(class) )Each mechanism computes an admissible authority from its own evidence axis — the action’s risk, the path’s policy, the input’s trustworthiness, the operator’s history — and the executed authority is the minimum. I think this min-of-justifications rule is the natural composition law for autonomy governance in general: a system should never act with more authority than its weakest current justification supports.
The measurements back the composition claim. Per-action gating alone — majors always reviewed, no longitudinal layer — cost 2.3× the composed regime for an unreliable operator, because wrong minor and moderate actions kept executing forever with nothing watching the trend. And for a reliable operator the composed regime slightly beat even the rung-level oracle (5.9 vs 6.8 cost units), because a per-action gate works within a rung at finer granularity than any rung assignment can reach.
What the simulation says
The full study is in the paper (discrete-time simulation, 20 seeds per cell, operator archetypes from reliable to silently degrading; the companion library ships with it). Three results carry the argument:
- Static tiers are U-shaped; budgets track the oracle. Freeze an operator as autonomous and it’s optimal while reliable, catastrophic when it degrades — 3.9× the budget regime’s cost under a silent step degradation. Freeze it as supervised and you pay ~3.9× in permanent review toil on a reliable operator. The budget-governed regime tracked a full-knowledge oracle within ~8% at both extremes — without knowing what the oracle knows.
- Burn-rate demotion beats periodic review, structurally. Against a silent degradation, budget-derived demotion detected 2.7–5.1× faster than a monthly track-record review across a 20× action-volume range. The structure is the point: budget detection latency is set by action volume and geometry; review latency is set by cadence. Matching the budget with humans would mean reviewing every operator’s track record roughly weekly, forever.
- The asymmetry is load-bearing — the oscillation result above. Symmetric ladders look cheaper on a spreadsheet and buy the discount with concentrated harm and authority churn.
Honest caveats: it’s a stylized cost model with synthetic operator trajectories, not an LLM in a loop — the claims are about the governance dynamics, which depend on budget-and-ladder mechanics rather than on how wrongness is generated. And the whole mechanism is only as honest as the adjudication pipeline feeding it.
What to do Monday
- Split your agent’s action space into classes by blast radius and reversibility, and admit that authority should differ per class. One flag for the whole agent is rung zero.
- Write the autonomy budget policy document — SLO per class, what burns the budget, who gets paged on exhaustion, what evidence promotes, which events (model updates!) reset it. Negotiate it once, calmly, not during an incident.
- Wire burn-rate alerts to demotion, not just paging. If a fast burn only notifies a human, your mean time to de-authorize is your mean time to read Slack.
- Check your burn semantics. If human-approved actions burn the agent’s budget, you’ve built a demotion spiral. If your allowance is an absolute constant, a busy week will demote a healthy agent.
- Treat every model or prompt update as an evidence-invalidating event. Partial budget reset, re-earn the rung. Track record only weakly survives a version change.
The vendors will keep shipping autonomy as a feature you enable. Treat it as a budget your agent spends. The toggle version of autonomy fails the way unlimited-release-velocity failed before error budgets: gradually, invisibly, then all at once on a Saturday. SRE already wrote the fix. We just have to point it at the operators.