Your Agent's Safety Argument Has an Expiry Date. Write Down the Defeaters.

Every team that lets an agent act unattended made an argument to get there, whether or not anyone wrote it down. It usually sounds like this: we ran it in shadow mode for a month, it only touches stateless services, a restart is reversible, and the on-call lead signed off.

That’s a reasonable argument. It’s also dated. It was true about a specific model, a specific prompt, a specific tool contract and a specific state of the telemetry, on the day someone said yes. Three weeks later the provider ships a model update, an MCP server release widens what the restart tool accepts, and the argument that justified autonomy is about a system that no longer exists. The agent keeps its authority anyway, because nothing connects the argument to the authority.

Safety-critical engineering has a tool for this, and it’s older than LLMs: the assurance case.

Assurance cases, briefly

An assurance case is a structured argument that a system is acceptably safe for a particular use. It has claims, evidence for each claim, and the reasoning that connects them. The most common notation is GSN, the Goal Structuring Notation Community Standard (version 3, 2021), maintained by the Safety-Critical Systems Club’s Assurance Case Working Group and free to use. Industries that must justify a system before it operates, such as rail, defence, nuclear and air traffic management, have used safety cases for decades.

The part that matters most for agents is less famous. In their work on defeaters in Assurance 2.0, Bloomfield, Netkachova and Rushby treat a defeater as a reason a claim might not hold, and argue that a credible case has to look for them actively, not just pile up supporting evidence.

For a certified avionics box, defeaters are mostly found during review, because the system doesn’t change after certification. An AI operator changes all the time, and most of its defeaters are ordinary operational events. That’s the opportunity: an agent’s defeaters can be machine-checked.

Scope the case to one action class

An assurance case for “the agent” is useless. The agent restarting a deployment and the agent rotating a database credential need different arguments. Scope each case to one action class at one authority rung, exactly the way authority should be partitioned in the first place:

Top claim: k8s-remediator may restart deployments in checkout without approval.

Then decompose it into the claims a skeptical reviewer would need to accept:

Claim	Evidence	Defeaters (what makes the evidence stop counting)	Falls to
G1. Its proposals were safe on replayed checkout incidents	shadow run: 412 replays, 0 harmful proposals	model changed since the replay; prompt or context pack changed	supervised
G2. A restart is reversible and bounded	PodDisruptionBudget minAvailable=2; tool scoped to one deployment	tool schema drifted; PDB removed or relaxed	advisory
G3. It decides on current, complete telemetry	metrics completeness ≥ 99% over the last hour	completeness below 99%	advisory
G4. Its live track record still supports the rung	autonomy error budget for this action class	budget exhausted	supervised

Two things to notice. Every defeater is something you can already observe: a model id, a content hash, a schema hash, a completeness ratio, a budget. And each claim names its own fallback: the authority the agent keeps if that claim falls. Losing G1, the evaluation evidence, doesn’t make a restart dangerous. It makes the agent’s judgment unverified, so it drops to supervised. Losing G2 or G3 means the action itself or the inputs it acts on can’t be trusted, so it drops further.

The case, as code

The case is data, the defeaters are predicates over live state, and the agent’s effective authority is the minimum over every undermined claim’s fallback. Standard library only. Save as assurance_case.py and run it.

"""An operational assurance case with live defeaters. Stdlib only.

Each claim carries the evidence it rests on and the conditions under which
that evidence stops counting (defeaters). If a claim is undermined, the
agent's authority falls to that claim's fallback rung.
"""
RUNGS = ["disabled", "advisory", "supervised", "autonomous"]

CASE = {
    "claim": "k8s-remediator may restart deployments in 'checkout' without approval",
    "authority": "autonomous",
    "subclaims": [
        {"id": "G1", "claim": "Its proposals were safe on replayed checkout incidents",
         "evidence": "shadow run sr-0912: 412 replays, 0 harmful proposals",
         "fallback": "supervised",
         "defeaters": {
             "model changed since the replay": lambda s: s["model"] != "model-2026-09-01",
             "prompt or context pack changed": lambda s: s["context_pack"] != "ctx-v7",
         }},
        {"id": "G2", "claim": "A restart is reversible and bounded in blast radius",
         "evidence": "PodDisruptionBudget minAvailable=2; restart tool scoped to one deployment",
         "fallback": "advisory",
         "defeaters": {
             "tool schema drifted (scope may have widened)": lambda s: s["tool_schema"] != "a41f09",
             "PDB removed or relaxed": lambda s: s["pdb_min_available"] < 2,
         }},
        {"id": "G3", "claim": "It decides on current, complete telemetry",
         "evidence": "metrics completeness >= 99% over the last hour",
         "fallback": "advisory",
         "defeaters": {
             "telemetry completeness below 99%": lambda s: s["telemetry_completeness"] < 0.99,
         }},
        {"id": "G4", "claim": "Its live track record still supports the rung",
         "evidence": "autonomy error budget for checkout/reversible",
         "fallback": "supervised",
         "defeaters": {
             "error budget exhausted": lambda s: s["budget_remaining"] <= 0,
         }},
    ],
}

def evaluate(case, state):
    authority, fired = case["authority"], []
    for c in case["subclaims"]:
        hits = [name for name, test in c["defeaters"].items() if test(state)]
        if hits:
            fired.append((c["id"], hits, c["fallback"]))
            authority = min(authority, c["fallback"], key=RUNGS.index)
    return authority, fired

BASELINE = {"model": "model-2026-09-01", "context_pack": "ctx-v7", "tool_schema": "a41f09",
            "pdb_min_available": 2, "telemetry_completeness": 0.997, "budget_remaining": 0.62}

if __name__ == "__main__":
    scenarios = [
        ("baseline", {}),
        ("provider ships a model update", {"model": "model-2026-11-04"}),
        ("collector drops a third of series", {"telemetry_completeness": 0.66}),
        ("MCP server release widens restart tool", {"tool_schema": "c07e3b"}),
        ("model update + budget exhausted", {"model": "model-2026-11-04", "budget_remaining": 0}),
    ]
    results = {}
    for name, change in scenarios:
        auth, fired = evaluate(CASE, {**BASELINE, **change})
        results[name] = auth
        print(f"{name:42s} -> {auth}")
        for cid, hits, fb in fired:
            print(f"{'':45s}{cid} undermined: {'; '.join(hits)} (falls to {fb})")
    assert results == {"baseline": "autonomous", "provider ships a model update": "supervised",
                       "collector drops a third of series": "advisory",
                       "MCP server release widens restart tool": "advisory",
                       "model update + budget exhausted": "supervised"}

Output:

baseline                                   -> autonomous
provider ships a model update              -> supervised
                                             G1 undermined: model changed since the replay (falls to supervised)
collector drops a third of series          -> advisory
                                             G3 undermined: telemetry completeness below 99% (falls to advisory)
MCP server release widens restart tool     -> advisory
                                             G2 undermined: tool schema drifted (scope may have widened) (falls to advisory)
model update + budget exhausted            -> supervised
                                             G1 undermined: model changed since the replay (falls to supervised)
                                             G4 undermined: error budget exhausted (falls to supervised)

The evaluator is twenty lines. The work is in the table. That’s the right ratio, because the table is the part a reviewer, an auditor or the next engineer can read and argue with.

Why this beats re-approval on a calendar

The common alternative is a review cadence: re-certify each agent quarterly. That fails both ways.

• Too slow when it matters. A model update in the first week of the quarter leaves the agent running for nearly three months on evidence about a different model. The G1 defeater catches it on the next evaluation tick.
• Too much work when nothing changed. If the model, prompt, tool schemas and telemetry are all unchanged and the budget is healthy, the quarterly review re-reads an argument whose premises haven’t moved. Defeaters tell you which evidence needs regenerating, so the review effort goes to what actually changed.

It also composes with the controls you already have. The per-action policy engine still decides whether this restart is allowed. The autonomy error budget still tracks the live record, and here it is one of the defeaters (G4). The telemetry integrity checks feed G3. The assurance case is the layer that says why the whole stack adds up to “acceptably safe for this action class”, and what has to stay true for that to remain so.

Restoring authority

When a defeater fires, the claim isn’t false. It’s unsupported. The way back is to regenerate the evidence against the current system: re-run the replay set on the new model, re-verify the tool scope against the new schema. Then record the new evidence, update the defeater’s reference values (the new model id, the new schema hash) and let the evaluator restore the rung.

Make restoration a deliberate act with a named owner, not an automatic reset when the defeater condition clears. Demote automatically, promote deliberately. The asymmetry is the point.

Limits

• An assurance case is an engineering argument, not a certification or legal opinion. It makes your reasoning inspectable. It doesn’t make it correct.
• You’ll miss defeaters. The ones in the table are the operational ones I’d start with. Unknown unknowns still need human review of the argument itself, periodically and after every incident.
• Fallback rungs are judgment calls. Mine are illustrative. Write down why you chose yours, because that rationale is part of the case.
• The evidence has to be real. “412 replays, 0 harmful” in the example is a placeholder. Zero failures in a few hundred replays still allows a meaningful failure rate, and scoring shadow mode properly is its own problem.

What I’d do this week

1. Pick the one action your most trusted agent takes unattended. Write the top claim in one sentence.
2. List the claims a skeptical reviewer would need, and the evidence you actually have for each.
3. For every piece of evidence, ask: what change would make this stop counting? That’s your defeater list.
4. Wire the two cheapest defeaters first, model id and tool schema hash, to drop authority automatically.

If you can’t write the argument down, the agent’s authority rests on someone’s memory of a meeting. Write it down, and give it an expiry date.