· 4 min read AIReliabilitySRE

The runner is the API: why your remediator's contract matters more than your handlers

Everyone reviews the handler that restarts the pod. The safety lives in the runner around it — and handlers that skip it silently void every guarantee.

Every code review of a remediation system spends its time in the wrong file.

The reviewer opens restart.py, reads twelve lines that call the Kubernetes API, and asks sensible questions about error handling. Meanwhile the file that determines whether this system is safe — the one wrapping every handler, deciding what may run, checking whether the world got better, and writing the record you will hand an auditor — gets a glance because it “hasn’t changed.”

The handlers are the least interesting code in a remediator. The runner is the product.

Where the safety actually lives

A remediator is not “code that restarts pods.” It is a sequence that every action passes through, without exception:

gate → act → verify → audit, with rollback as a first-class branch rather than an exception path.

Handlers only own the act step, and only the mechanical part of it. Everything that makes the system defensible is in the wrapper:

  • Is this action’s reversibility tier within the current autonomy ceiling?
  • Has the kill switch been tripped, globally or for this tenant?
  • Is there budget left for this class of action in this window?
  • Did the post-action probe say the world got better — and did it have the ability to say no?
  • If not, did the inverse run, and did that get verified?
  • Did exactly one audit event get written, with the tier, the outcome, and the human whose authority this ran under?

Put those in the handlers and you have written them once. You will not write them the same way ten handlers later, and the one that forgets the kill-switch check will be the one that runs during the incident where you needed it.

The contract, concretely

The runner’s contract is the API of the system. Everything else is an implementation detail behind it.

What the runner guarantees to every action:

  1. Nothing executes without a reversibility tier and a policy decision.
  2. Every execution is followed by verification, or it is not considered complete.
  3. Every failed verification triggers the inverse operation, which is itself verified.
  4. Exactly one audit event is emitted per action, whatever the outcome.
  5. Every terminal state is one of a closed, named set — not a free-text string, and not an exception that escaped.

What the runner demands from every handler:

  1. Do one thing.
  2. Declare your reversibility tier.
  3. Expose an inverse.
  4. Be safe to run twice.
  5. Return a structured outcome. Do not raise to signal a business failure.
  6. Contain no policy, no audit, no retry.

Point 4 has become non-negotiable rather than merely wise. Retry is now the normal recovery path in enough protocols and transports that a handler which is not idempotent will eventually be executed twice by infrastructure you do not control. If running your handler twice produces two of something, that is a bug in the handler, not bad luck.

Point 6 is the one people argue with. A handler author reaches for a policy check because they can see a case the runner does not cover. That instinct is correct and the response is wrong: extend the runner. A safety check that lives in one handler is a safety check nine other handlers do not have.

The dangerous pattern

The most damaging line in a remediation codebase is a handler that constructs its own client:

# in a handler — do not do this
client = KubernetesClient(token=os.environ["SA_TOKEN"])
client.delete_namespaced_pod(name, namespace)

That handler has silently opted out of every guarantee above. The gate never ran. The probe never ran. The audit event does not exist. The kill switch is irrelevant, because nothing consulted it. Your architecture diagram still shows a runner in the middle; production disagrees.

This does not usually arrive as sabotage. It arrives as a Friday afternoon and a handler that needs one extra call the runner does not expose yet.

Make it structurally hard, then test for it. Handlers should receive a scoped client from the runner — one that cannot perform ungated mutations — rather than building their own from ambient credentials. And back it with a test that asserts the property directly:

def test_no_handler_constructs_its_own_client():
    for name, handler in registry.items():
        src = inspect.getsource(handler.__module__)
        assert "KubernetesClient(" not in src, f"{name} bypasses the runner"

That test is crude and it works. An architectural rule that lives only in a design document is a rule the third contributor has not read.

Why this makes reviews cheap

The payoff is not elegance, it is review economics.

When the runner owns the cross-cutting properties, adding a handler stops being a safety conversation. The reviewer asks: does it do one thing, is the tier right, does the inverse work, is it safe to run twice? Four questions, all local, all answerable by reading one short file.

Compare a codebase where each handler carries its own policy checks and audit emission. Now every new handler is a full safety review, every reviewer has to hold the whole system in their head, and the answer to “is this safe” is “read all eleven handlers and hope.”

Centralising the contract is what lets you add the twelfth action without re-auditing the eleven that came before. That is the property that makes a remediation system something you can keep extending rather than something you freeze because nobody wants to touch it.

What to do Monday

  1. Find the file that wraps your actions. If there is no such file, that is the finding — you have scripts with a scheduler, not a remediator.
  2. Write the contract down, both directions: what the runner guarantees, what it demands. One page.
  3. Grep for handlers that construct their own clients or credentials. Every hit is a live bypass.
  4. Add the bypass test. Crude is fine.
  5. Move one safety check out of a handler and into the runner. Usually the audit emission, because it is the one most often forgotten and the one you will most regret missing.

The handler that restarts the pod is fifteen lines and will be right. The runner is the part that decides whether being wrong is survivable, and it deserves the review attention you have been spending on the handlers.

Frequently asked questions

What is the runner in an automated remediation system?

The runner is the component every remediation action passes through: it gates the action against policy and reversibility, executes the handler, verifies the outcome with a probe, rolls back on failure, and emits exactly one audit event. Handlers are the small pieces that know how to restart a pod or scale a deployment. The runner is where every cross-cutting safety property lives, which makes its contract — not the handlers — the thing worth reviewing carefully.

Why should safety logic live in the runner rather than in each handler?

Because a safety property implemented per handler is a safety property you will eventually forget. Ten handlers written by different people over eighteen months will not all remember to check the kill switch, emit an audit event, and run a verification probe. Centralising those in the runner means a new handler inherits them by construction, and the review question shrinks from 'is this handler safe' to 'does this handler do one thing correctly'.

What makes a remediation handler correct?

A handler should do exactly one thing, declare its reversibility tier, expose an inverse operation, be safe to run twice, and return a structured outcome rather than raising on failure. It should contain no policy checks, no audit emission, and no retry logic — those belong to the runner. A handler that reaches around the runner to execute something directly is the single most dangerous pattern in a remediation codebase, because it silently opts out of every guarantee the system claims to provide.

How do you stop remediation handlers from bypassing the runner?

Make bypass structurally hard and then test for it. Handlers should receive a scoped client that cannot perform ungated mutations, rather than constructing their own credentials. Back that with a test that asserts every registered handler goes through the runner and that no handler imports a mutation client directly. Architectural rules that exist only in a design document get violated by the third contributor who has not read it.

Comments