Your agent triages an incident well. It correlates the deploy with the latency curve, finds the runbook, and writes a summary better than the one the on-call would have written at 3 a.m. Someone in the review says the obvious thing:
“Why are we making a human copy-paste the fix? Just let it run the command.”
That question is reasonable, and the answer is not “AI is dangerous.” The answer is that you have just proposed changing what the system is, and almost none of the engineering that makes the change survivable is in place yet.
Reading is reversible. Acting is not.
An advisory agent’s worst failure is a confidently wrong paragraph. Somebody reads it, disagrees, moves on. You lost four minutes of attention.
An acting agent’s worst failure is an outage it caused while trying to fix a different one. Those two failure modes do not belong in the same risk conversation, and they should not share an approval process.
This is the whole asymmetry, and it is why the safety machinery below exists at all. None of it is needed to summarise an incident. All of it is needed to restart a pod.
I’ve written before about where AI actively hurts in production SRE. This is narrower: not should AI be here, but may it hold the wheel.
The five things that must exist first
Not “should ideally exist.” Must. If any one is missing for the action in question, the agent proposes and a human executes.
1. A reversibility classification for the specific action
Not for the agent. For the action.
“Restart the pod” and “drop the table” are not the same kind of event, and an agent that can do both has been given authority at the wrong granularity. Every action an agent can execute needs a tier assigned before it ships:
- Reversible — restart, scale out, requeue. The inverse is cheap and obvious.
- Recoverable — scale in, rollback a deploy. The inverse exists but costs something.
- Irreversible — delete, terminate, notify customers. There is no inverse.
The tier lives in the tool registry, not in the prompt. A prompt that says “please do not run destructive commands” is a wish. A registry where the destructive command is registered as proposal-only, returning a pending-approval token instead of executing, is a control. One of those survives a model having a bad day.
2. A probe that can tell success from silence
Most remediation code checks that the command returned zero. That is not verification, it is the absence of an exception.
The question a verification probe answers is did the world actually get better? — and it has to be able to return no. If your probe can only report “the API accepted my request,” you have automated the action and not the outcome. The agent restarted the pod, the pod came back, and the error rate is unchanged because the pod was never the problem.
A probe needs a signal, a threshold, and a timeout. If you cannot name all three for an action, you cannot verify that action, and an unverifiable action is not one to automate.
3. A rollback path that is itself tested
Rollback is a first-class outcome, not an exception branch.
The rollback path is the code that runs on the worst day, which means it is the code least likely to have been exercised. If the inverse handler has never run in anger, it is a hypothesis. Test it the way you’d test the forward path — in the scenario suite, on every change, with the same seriousness.
And the honest constraint: if a rollback needs the same failing system to work, it is not a rollback. A remediator that cannot undo its own action while the cluster is degraded has a rollback path in the design document only.
4. An audit record that names a human
When the review asks who authorised this, “the agent” is not an answer that ends the conversation.
Every action needs to be attributable to exactly one named principal — the human whose delegated authority it ran under, or, for genuinely unattended pipelines, a dedicated per-agent identity registered in your IdP and distinguishable in every event. What does not work is a shared service account used by every agent run for everyone, which is a backdoor with a friendly name. I’ve made that argument at length in agents need identities, not API keys.
The record has to be written before you need it and stored somewhere the incident cannot edit. Append-only, outside the blast radius.
5. A kill switch that works while everything else is on fire
Every autonomy design has a stop button in the diagram. Fewer have one that works at 3 a.m. during the incident it is meant to stop.
The test is uncomfortable and worth running: trip it during a game day, with the person who would actually be on call, without the person who built it in the room. Time it. If stopping the agent requires a deploy, a config push through a pipeline, or someone who is asleep, you have a documented intention rather than a control.
Advisory-forever is also a failure
Here is where the cautious version of this argument goes wrong, and I want to be honest about it rather than land on a comfortable “keep humans in the loop” and stop.
An agent that only ever proposes trains its reviewers to approve. The proposals are right most of the time — that is why you deployed it — and being right most of the time is exactly what erodes scrutiny. Approval decays into rubber-stamping, and a rubber-stamped approval is a control that exists on the org chart and not in reality. You end up with the risk profile of an autonomous system and the latency of a manual one, which is the worst available trade.
So permanent advisory mode is a legitimate destination for genuinely irreversible actions. It is a bad default, because a default is not a decision. If you never revisit it, you have not chosen human oversight — you have chosen a control that looks like oversight and does not function as one.
The way out is not to loosen the gate. It is to move the bounded actions across it, so that the gate you kept is the one people still read. That is a separate argument, and I’ll make it properly in its own post.
What to do Monday
- List every action your agent could take, including the ones behind a “suggest” button someone will eventually wire up.
- Tier each one — reversible, recoverable, irreversible. Do it in the tool registry, in code.
- For every action you want to promote to execute, write down the probe signal, threshold, and timeout. If you cannot, it stays advisory.
- Run the rollback in a scenario test. Not once, on every change.
- Time your kill switch during a game day. Note the real number, not the intended one.
None of these are AI problems. They are the operational hygiene that acting on production has always required, and the reason they feel new is that we usually inherit them from a change-management process an agent bypasses entirely.
The agent is not the risky part. The write path is. Build the write path first, and the question of whether the model is good enough becomes answerable instead of theological.
Comments