In January 2026, an autonomous airline booking agent rebooked 1,247 passengers onto incorrect flights during a weather disruption in Toronto. It didn’t crash. It didn’t throw an error. It confidently did the wrong thing, 1,247 times, faster than any human could have caught it.
That agent worked in testing. They all work in testing. The recurring lesson of this year’s agent failures is that they failed “in production conditions that were foreseeable but not tested for.” Reported AI-related incidents are up 21% year over year, and the 2026 reliability data keeps repeating the same diagnosis: most AI agents look perfect in demos and fall apart in production.
Vendors are selling the opposite story. NeuBird shipped agents that “automatically prevent, detect, and fix” incidents; every cloud provider has an autonomous ops agent now. The capability is real. So is the gap between that capability and anything you should trust unsupervised. I’ve spent over a decade building cloud platforms for a safety-critical aviation-SaaS domain, where a wrong automated action isn’t a bad demo — it’s a safety event. That background is why I think the industry is measuring the wrong thing.
Reliability is not accuracy
The instinct is to ask “how good is the model?” and reach for a benchmark score. But an agent task isn’t one decision — it’s a chain of decisions and actions, and errors compound along the chain.
A model that’s 95% accurate per step sounds great until you run it across a 15-step task: the naive ceiling on getting every step right is 0.95^15 ≈ 46%. And that’s the optimistic framing, because the steps aren’t independent. A wrong decision early doesn’t just fail — it becomes the input to every step after it. The agent reasons confidently on top of its own mistake.
So agent reliability is a property of the whole loop: the decisions, the tools it calls, the blast radius of each action, and how cheaply each one can be undone. A single-shot accuracy number is blind to all of it. We need metrics that see the loop.
Two metrics that see the loop
This is the heart of the reliability framework I’ve been developing. Two named measures, both designed to capture what accuracy can’t.
Failure Amplification Factor (FAF)
FAF measures how far one wrong decision spreads into downstream wrong actions. Formally: inject F faults, count the D distinct failure events causally attributable to them — the primary failure plus every secondary failure it triggers — and FAF = D / F. A system where a mistake stays contained to a single action has an FAF near 1. A system where one mistake multiplies has a high FAF — the signature of a cascade.
The airline agent is the textbook case: one wrong read of a weather disruption amplified into 1,247 incorrect rebookings. FAF in the thousands. The model’s per-decision accuracy is almost irrelevant next to that multiplier — what killed the outcome was the absence of anything between the bad decision and its mass execution.
You lower FAF structurally, not by improving the model: checkpoints between decision and action, batch-size caps, “are you sure” gates on fan-out, and reversibility so a wrong action can be walked back before it spawns the next one.
Containment Score
Containment Score measures how well the system bounds the blast radius of a wrong action once it happens. Formally: define the in-scope component set U a fault could reach — tools, memory partitions, peer agents, downstream tasks — and the set A it actually contaminated; CS = 1 − |A| / |U|, where 1 is perfect containment. Not “did the agent avoid the mistake” — it will make mistakes — but “when it did, how much could the mistake touch?”
A high Containment Score comes from the boring, load-bearing engineering: scoped credentials so the agent can’t reach systems outside its remit (the no-anonymous-inference-endpoints principle applied to actions), rate limits on actions, irreversible operations held behind human approval, and a small explicit action set instead of a general-purpose write capability.
Together these two metrics reframe the deployment question. You stop asking “is the model good enough to act on its own?” and start asking “how far does a wrong action spread (FAF), and how well do we bound it (Containment Score)?” Those are answerable with data. “Is the model good enough” is a vibe.
Earn authority, don’t switch it on
The framework only works if authority is earned with measurement, which is the operational thread connecting this to bounded autonomy. That post lays out the autonomy ladder — six rungs from observe-and-summarize up to full unsupervised action. FAF and Containment Score are how you decide whether an agent has earned the next rung.
Concretely: before an action type graduates to less supervision, look at the production record. How often was the agent wrong on this action? When it was wrong, what was the measured FAF — did the mistake stay put or spread? What was the Containment Score — how much could it have touched? If you can’t produce those numbers, the agent hasn’t earned the rung. You’re not deploying autonomy; you’re deploying hope, and the State of Production Reliability data — 78% of orgs had incidents where no alert even fired — shows how that ends.
This is also where multi-agent systems get dangerous. Coordination failures between agents are a new class of reliability failure, and they’re FAF multipliers by nature: one agent’s wrong output becomes another agent’s trusted input. A fleet of ungoverned agents has a high FAF and a low Containment Score almost by construction.
What to do Monday
- Compute FAF for your riskiest agent action. When this action is wrong, how many downstream actions does it trigger? If the answer is “a lot, with nothing in between,” you have an airline-agent waiting to happen.
- Audit your Containment Score. For each agent, list what a fully-wrong action could touch. If the blast radius is “anything its credentials can reach,” scope the credentials before you do anything else.
- Measure both under injected faults, not just in postmortems. FAF and Containment Score are exactly what a fault-injection campaign produces — the same discipline I applied to the MCP layer in chaos engineering for MCP. Break the loop on purpose and read the numbers off the wreckage.
- Put a checkpoint between decision and mass action. The single highest-leverage fix for high FAF is a gate before fan-out. The airline agent needed one human “confirm 1,247 rebookings” step. It didn’t have one.
- Tie promotion to data, not roadmap pressure. No agent climbs the autonomy ladder without a measured FAF and Containment Score at the rung below.
The agents are genuinely capable now — that part of the hype is true. But capability without a reliability framework is exactly how you rebook 1,247 passengers onto the wrong flights and call it automation. The gap is real, it’s measurable, and closing it is engineering work, not a model upgrade.
Comments