The Reconciliation Loop: Why Kubernetes Self-Heals and Your Script Doesn't

The loop never asks “what happened?” It asks “what is true, and what should be true?” — a question you can answer from scratch at any moment.

Ask someone how Kubernetes works and you will usually get an inventory: Pods, Deployments, ReplicaSets, the scheduler, kubelet, etcd. That is a list of nouns. The thing that makes Kubernetes behave the way it does is a verb, and it is the same verb in Terraform, Argo CD, Flux, Crossplane, every operator ever written, and — as it turns out — in a well-built AI remediation agent.

The verb is reconcile, and the idea it implements is fifty years older than any of those tools.

Two ways to make something true

Say you need three replicas running.

Edge-triggered — act on transitions:

def on_scale_event(event):
    if event.type == "SCALE_UP":
        create_pod()

Level-triggered — act on state:

def reconcile():
    desired = spec.replicas          # 3
    actual  = count_running_pods()   # 2
    if actual < desired:
        create_pod()

On a whiteboard, with a reliable network and no crashes, these are equivalent. In production they are not remotely equivalent, and the difference is entirely about what happens when something goes wrong.

A lost message. The edge-triggered system never learns about the scale-up. It stays at two replicas forever, and nothing in it will ever notice — its model of the world is the sum of events it received, and it received the wrong set. The level-triggered system runs again in thirty seconds, sees 2 against 3, and creates a Pod.

A duplicate message. The edge-triggered system creates a fourth Pod. The level-triggered system sees 3 against 3 and does nothing.

A restart. The edge-triggered system comes back with no idea what it missed while down. The level-triggered system reads the world and carries on; downtime cost it a delay, not correctness.

The term comes from hardware interrupts — level-triggered means the line is held high until serviced, so a handler that was busy still sees it afterwards. Applied to distributed systems it gives you one very large property:

A level-triggered system’s correctness does not depend on message delivery. It depends only on being able to observe current state.

That is the sentence to keep. Distributed systems cannot guarantee delivery, so any design whose correctness depends on it is building on sand. Level-triggering sidesteps the problem rather than solving it: at-least-once delivery plus idempotent processing is the same trick at the message layer.

How Kubernetes implements it

Every Kubernetes object has the same shape, and it is not an accident:

spec:                  # desired — written by you
  replicas: 3
status:                # observed — written only by the controller
  readyReplicas: 2
  conditions:
    - type: Available
      status: "False"

Spec is yours. Status is the controller’s. Neither writes the other’s half. That strict ownership split is what makes reconcile recomputable: a controller can read one object and know both what should be true and what it last observed, with no memory of what it did last time and no event history to replay.

A controller is then a loop over a work queue:

func (r *Reconciler) Reconcile(ctx context.Context, req Request) (Result, error) {
    var app App
    if err := r.Get(ctx, req.NamespacedName, &app); err != nil {
        // Already deleted. Nothing to converge toward — not an error.
        return Result{}, client.IgnoreNotFound(err)
    }

    // Observe the world. Never trust a cached memory of what we did before.
    deploy := &appsv1.Deployment{}
    err := r.Get(ctx, deployKey(&app), deploy)

    switch {
    case apierrors.IsNotFound(err):
        return Result{}, r.Create(ctx, buildDeployment(&app))   // one step
    case err != nil:
        return Result{}, err                                     // requeue w/ backoff
    }

    if want := buildDeployment(&app); !equalSpec(deploy, want) {
        deploy.Spec = want.Spec
        return Result{}, r.Update(ctx, deploy)                    // one step
    }

    app.Status.Ready = deploy.Status.ReadyReplicas
    return Result{RequeueAfter: 30 * time.Second}, r.Status().Update(ctx, &app)
}

Four properties of that function are the whole discipline, and every one of them is a rule you can break by accident:

1. It takes a name, not a payload. The request carries only “something about object X may have changed” — never what changed. So the controller must go and look, which means it cannot be wrong about what it missed. Watch events are an optimisation on when to run, never an input to what to do.
2. It re-reads everything. No state is carried between invocations. The function is a pure-ish computation over current world state.
3. It is idempotent. Create happens only inside IsNotFound; Update only on a real diff. Running it a thousand times on an unchanged object produces no effect.
4. It does one step and returns. It does not sit in a loop waiting for a Pod to become ready. It makes progress, returns, and gets called again — which is why a controller restart mid-sequence is harmless.

Because reconcile is idempotent, the controller-runtime machinery is free to call it far more often than there were actual changes: on watch events, on a periodic resync, on every item in the cache at startup, and again with exponential backoff after any returned error. Duplicate work costs CPU. It never costs correctness.

That is also the honest answer to “how does Kubernetes self-heal?” Nothing detects the failure and triggers a repair. The node dies, its Pods stop being Ready, and the next reconcile sees 1 against 3 and makes one Pod. The “healing” is the absence of any special case — the loop was already running and the numbers changed.

Why declarative beat imperative

This is the deeper reason kubectl apply won over kubectl create, and Terraform won over provisioning scripts.

An imperative script is a sequence of transitions whose correctness depends on the starting state. Run it twice and you get two load balancers. Run it against a half-built environment and it fails somewhere in the middle, leaving a mess with no defined next action. The script’s author had to imagine every starting state in advance, which is not possible.

A declarative specification is a description of an endpoint. The system computes the path. Run it twice and the second run is a no-op; run it against a half-built environment and it finishes the job; run it against a drifted environment and it corrects the drift — because “drift” is just the diff, and the diff is the only thing the loop ever looks at.

terraform apply is the same loop with a human-visible pause: refresh (observe), plan (diff), apply (act). GitOps closes it into a continuous loop by making Git the spec and running the comparison forever — which is why an Argo CD or Flux install reverts a manual kubectl edit within minutes. That is not the tool being protective. It is the tool doing the only thing it does.

Where it breaks

The pattern is not free, and the failure modes are specific enough to be worth naming.

Convergence is not instant. Between the spec change and the world matching it, the system is legitimately wrong. If you need transactional behaviour — this and that, atomically — a control loop is the wrong primitive, because it will visibly pass through the state where one is done and the other is not.

The hot loop. Controller A writes a field that controller B reconciles by writing a field that wakes controller A. The two spin at CPU-burning speed, producing no progress. This is why status writes should be conditional on an actual change, and why Result{RequeueAfter: ...} exists.

Observation lag looks like a diff. Reading from a watch cache that is a few hundred milliseconds stale, a controller can see “0 replicas” for a Deployment it created moments ago and create another. The fix is not to read from the live API on every pass — that does not scale — but to make creates deterministic: derive the name from the owner so a duplicate create collides on the API server’s uniqueness check instead of succeeding. The same claim-the-key-first trick that makes retries safe.

Fighting controllers. Two controllers with different opinions about one field will flip it back and forth forever. Kubernetes’ answer is ownerReferences and server-side apply field ownership: exactly one actor owns each field, declared explicitly, and conflicts are surfaced rather than silently won by whoever wrote last.

A loop cannot fix what it cannot observe. If your status only reports “the Pod is Running” and the real requirement is “the Pod serves traffic correctly”, the loop will converge to a world that is broken and report success. The quality of a control loop is capped by the quality of its observation — which is the same reason telemetry integrity is a prerequisite for any kind of automated action, not an optional extra.

The version that matters for agents

This is where a forty-year-old control theory idea stops being background knowledge and becomes an active design decision.

Most AI remediation systems are built edge-triggered, because that is the shape the tooling suggests: an alert fires, a webhook wakes an agent, the agent reasons about the alert and acts. That design inherits every edge-triggered failure mode at once, and adds one of its own:

• The alert is lost or the webhook 500s → nothing happens, and nothing ever notices.
• The alert fires three times → three concurrent agents remediate the same thing.
• The agent acted but the action silently failed → nothing re-checks, because the event was consumed.
• The agent believes it succeeded. A model that called a tool and got a 200 will report the problem fixed. Nothing independently verifies that the world changed.

The level-triggered version separates the loop from the intelligence:

def reconcile(service: str) -> None:
    observed = measure(service)            # real SLIs, from telemetry
    desired  = slo_for(service)

    if observed.meets(desired):
        return                             # converged. Do nothing. Say nothing.

    action = agent.propose(observed, desired, history=recent_actions(service))
    if policy.permits(action):             # the bounded-autonomy gate
        execute(action, idempotency_key=f"{service}:{observed.window_id}")
    # No verification here on purpose: the NEXT pass measures the world again.

The agent proposes; the loop decides whether anything is still wrong. Note what falls out for free. There is no “did it work?” step, because the next pass answers that by measurement rather than by asking the model. A crashed agent resumes by observing, not by replaying. A duplicate trigger is a no-op because the first pass already converged. And an action that made things worse shows up as a larger diff on the next pass — which is exactly the signal an error budget for autonomy needs in order to withdraw permission.

The important inversion: the model’s claim of success is not evidence of success. In an edge-triggered design it is the only evidence you have. In a level-triggered design you never need it, because you re-measure. That single property is most of the distance between a demo and something you would let touch production.

The rule worth remembering

Compare desired to observed and take one step. Never act on the memory of an event.

Every property people admire in Kubernetes — self-healing, drift correction, safe restarts, survivable partitions, apply being safe to run twice — is a consequence of that one sentence, not of anything specific to containers. Which is also why it transfers cleanly to the next thing you build, including the ones with a model in the middle.