Three strategies, three answers to “who is exposed while we find out?” Only one of them keeps the answer small.
Some version of your test suite passed. Staging looked fine. You deploy, and eleven minutes later the error rate is up and you are in an incident.
This is not a testing failure. There is a category of defect that only production can find — the traffic shape, the data volume, the cache that was warm, the dependency under real load, the one customer whose payload is shaped differently from every fixture you have. You are going to discover some bugs in production. That is not a process problem to be eliminated; it is a property of the situation.
The question worth engineering around is not how do we never ship a bug. It is who is exposed while we find out, and how fast can we stop it.
The strategies, ranked by that question
Recreate. Stop the old, start the new. Downtime for everyone. Legitimate for things that genuinely cannot run two versions at once — a singleton batch processor, a migration job — and for nothing else.
Rolling update. Replace instances a few at a time. Kubernetes’ default, and it is fine as a mechanism. But exposure climbs steadily toward 100% and nothing in the mechanism is watching whether it should. maxSurge and maxUnavailable control the speed of the change; they do not control its correctness. A rolling update with no health gate is a big bang with extra steps.
Blue-green. Two complete environments. Green is live, blue gets the new version, you verify blue out-of-band, then switch all traffic at once. Everyone is exposed simultaneously — but the switch back is a single operation and takes seconds. This is the strategy that optimises for time to revert, and it costs double infrastructure while both are up.
Canary. Shift a small slice of traffic — 1% — to the new version, compare it against the old, and increase only if the comparison holds. This is the strategy that optimises for blast radius. A broken release reaches 1% of users for the length of one bake window instead of everyone for however long triage takes.
They are not competitors. Blue-green answers “how fast can we undo it”, canary answers “how few people see it”, and mature setups use canary to decide and a blue-green-style pointer switch to execute the abort.
The canary gate is the whole thing
Shifting 1% of traffic is easy. Every ingress, mesh, and load balancer can do it. The part that determines whether canary analysis is a safety mechanism or theatre is the comparison, and there is one rule that matters more than all the tooling:
Compare the canary against a concurrently-running baseline of the old version. Never against a fixed threshold.
Here is why that is not pedantry. Suppose you gate on “error rate below 0.5%”. Two things go wrong immediately:
- An upstream dependency degrades during your bake window. Both versions see elevated errors. The threshold trips and you abort a perfectly good deploy — and, worse, you learn to distrust the gate.
- The canary gets a favourable traffic slice — fewer of the heavy requests, a warmer cache — and passes the threshold while being measurably worse than the version it is replacing.
Comparing against a baseline running right now, on the same traffic mix cancels out everything both versions experience. What remains is the difference attributable to the version, which is the only thing you were trying to measure. This is Google’s canary analysis guidance and the model Kayenta implements; Argo Rollouts and Flagger both express it as an AnalysisTemplate comparing two label selectors.
A serviceable gate:
# Argo Rollouts — the shape, not a drop-in config
analysis:
templates: [{ templateName: canary-vs-baseline }]
args:
- { name: canary-hash, valueFrom: { podTemplateHashValue: Latest } }
- { name: baseline-hash, valueFrom: { podTemplateHashValue: Stable } }
metrics:
- name: error-rate-delta
interval: 1m
count: 10 # bake: ten one-minute samples
failureLimit: 2 # two bad samples abort — not one
successCondition: result[0] <= result[1] * 1.10 # canary within 10% of baseline
provider:
prometheus:
query: |
sum(rate(http_requests_total{status=~"5..",hash=~"{{args.canary-hash}}|{{args.baseline-hash}}"}[2m]))
by (hash)
/
sum(rate(http_requests_total{hash=~"{{args.canary-hash}}|{{args.baseline-hash}}"}[2m]))
by (hash)Four design choices in there are worth stating explicitly, because each one is a lesson somebody learned the hard way:
failureLimit: 2, not 1. A single bad sample at low traffic is noise. Aborting on noise trains people to disable the gate, which is a worse outcome than a slightly slower abort.
A ratio, not an absolute. <= baseline * 1.10 survives a dependency having a bad hour. <= 0.005 does not.
Bake time is bounded by sample size, not by patience. At 1% of traffic, distinguishing a 0.4% error rate from a 0.5% one may take longer than anyone expects. If your bake window is too short for the statistic to mean anything, you have built a delay, not a gate. Either raise the canary percentage or lengthen the window — and know which one you chose.
Measure what users experience. Error rate and latency at the edge, plus one or two business signals — checkout completions, messages sent. A canary that is technically healthy and converting at half the rate is a failed canary, and no infrastructure metric will tell you.
Sticky routing, or you are testing nothing
Route traffic per request and a single user session bounces between versions. Their first call hits v2, creates a record with a new field; their next hits v1, which does not know about it. You have not tested v2 — you have tested an interleaving of v1 and v2 that will never exist again, and any bug you find may be an artifact of the interleaving rather than of the release.
Route by a stable key — session ID, user ID, a sticky cookie — so a given user consistently gets one version for the duration. This also makes the comparison meaningful, because the canary population is a population rather than a random sample of individual requests.
Related, and frequently overlooked: the canary needs a distinguishing label on every signal it emits. Metrics, logs, and traces all need the version dimension, or the comparison has nothing to group by. This has to be in place before the first canary, not added during the first confusing rollout.
The failure mode nobody plans for
Not “the canary failed”. Not “the canary passed”. The third outcome: the canary is ambiguous.
Error rate is up 0.3%. Latency p99 is up 12ms. Is that the release, or is it Tuesday? Nobody knows. The rollout is sitting at 5%, and there is an engineer staring at a dashboard making a judgement call under time pressure — which is precisely the condition under which people decide to press on.
The only real defence is to decide in advance, while nobody is under pressure:
- Write the abort criteria into the rollout config, not into someone’s head. If it is in the config, the machine decides and the decision is consistent.
- Set a maximum rollout duration. A rollout stuck at 5% for six hours is a decision nobody is making. Time it out and abort — the cost of re-running a good deploy tomorrow is far below the cost of an ambiguous state persisting.
- Default to abort on ambiguity. Aborting a healthy release costs one repeat deploy. Promoting a bad one costs an incident. The asymmetry is large and should be encoded in the default.
Rollback is a code path, and untested code paths do not work
Everyone has a rollback plan. Very few teams have a tested rollback plan, and the distinction only surfaces at the worst moment.
With build-once-promote-everywhere, rollback is re-pointing at the previous digest. It is fast because nothing is built, and safe because that artifact already passed every gate. Without it, rollback means rebuilding an old commit, which takes a full build and may not reproduce what shipped.
Beyond the mechanism, three things decide whether rollback actually works:
Roll back config with code. They shipped together and were tested together. New code with old config, or old code with new config, is a combination nobody has ever run.
Know what you cannot roll back. A destructive migration, a consumed message, a sent email, a third-party write. For these the only path is forward-fix, which is exactly why schema changes should be expand/contract: add the column, backfill, dual-write, switch reads, drop the old column in a later deploy. Every step is independently revertible, which is the only way a schema change and a rollback can coexist.
Rehearse it. Roll back a real deploy in production, deliberately, on an ordinary afternoon. You will find the credential that expired, the dashboard that only shows the current version, the runbook step that references a tool nobody has installed. Finding those on a quiet Tuesday is free; finding them at 2am is not.
Where the strategy stops working
Progressive delivery assumes you can run two versions simultaneously. Several situations break that assumption, and it is better to know which one you are in than to discover it mid-rollout.
Stateful services. Two versions writing the same rows need a schema both understand — expand/contract again. Two versions writing the same cache need compatible serialisation, or v1 will choke on v2’s entries and the failure will look like a cache bug rather than a deploy bug.
Asynchronous work. The canary gate watches the request path. A bug in a background job, a nightly batch, or a queue consumer may not show up in any request-path metric at all. Canary the consumer separately, on its own signals, or accept that this class of change is not covered.
Low traffic. Canary analysis is statistics. At a few requests per minute, 1% is not a sample. Small services should raise the canary share substantially — 20%, 50% — or rely on blue-green and fast reversal instead. Pretending a 1% canary means something at low volume is worse than not having one, because it manufactures false confidence.
Client-side code. You do not control when browsers or mobile apps update, so old clients persist indefinitely. Server changes must stay compatible with the client versions still in the wild, and the rollout dimension you control is the feature flag, not the deployment.
Agents make the automatic gate mandatory
The reason this matters more now is deploy frequency. When a meaningful share of changes are machine-generated and the pipeline is fast, the number of deploys per day goes up sharply — and human attention per deploy goes down proportionally. A gate that depends on somebody watching a dashboard degrades exactly when the deploy rate rises.
Two things follow:
The gate has to be automatic and blocking. Not a dashboard, not a Slack notification. A machine comparison with an encoded threshold that halts the rollout on its own. If a human has to look at something for the gate to work, the gate does not work at scale.
An agent that can remediate should be able to abort a rollout and nothing more. Abort is the ideal automated action: it is reversible, its failure mode is a delayed release rather than an outage, and it returns the system to a known-good state that was running minutes ago. Promotion is the decision that should stay gated, because promotion is the irreversible direction. The general principle — automate the direction that is safe to be wrong about — is the same one behind error budgets for autonomy, and canary abort is the clearest example of it anyone has.
The rule worth remembering
Expose the smallest population you can measure, compare it against the old version running right now, decide the abort criteria before you start, and make sure the path back is one you have actually walked.
You will ship bugs to production. Progressive delivery does not prevent that. It makes the difference between 1% of users for four minutes and all of them for forty — and that difference is most of what reliability engineering is.
Comments