Reliability
55 posts — newest first.
-
The Evidence an AI Operator Owes You Before It Acts
A tool-call log answers none of the eight questions an agent incident raises. A decision-time evidence record answers all eight, before the action runs.
-
Your Agent's Safety Argument Has an Expiry Date. Write Down the Defeaters.
An assurance case says why an agent may act unattended. Defeaters say when that stops holding. Wire them to live state and authority drops on its own.
-
Shadow Mode Isn't Evidence Until You Score It
In a seeded simulation, the agent with 95% concordance made 17 harmful high-severity proposals; the one with 88% made none. Score shadow mode properly.
-
A Three-Agent Vote Can Add 9.6 Points at Most: The Break-Even Math of Multi-Agent Systems
Closed-form models for voting, verifiers, pipelines and fan-out, checked against Monte Carlo and the research. When a second agent pays for itself.
-
I Killed an Agent 120 Times to See What Durable Execution Actually Buys You
SIGKILL at every step boundary under four persistence policies. Only one had zero duplicate side effects, and the fix wasn't on the agent's side.
-
How to Write a Postmortem Someone Will Actually Read
A practical template for blameless postmortems: when to write one, what each section is for, how to word it, and a small linter that catches the usual mistakes.
-
The Generators Were Fine. The Sensor Was on the Wrong Side of the Breaker.
Google's final report on the europe-west2 zone outage: a check that measured the wrong thing, redundancy lost to maintenance, and why multi-zone still degraded.
-
The Agent Didn't Turn Off the Monitor. It Turned Off the Pager.
OpenAI's coding agents left a monitor running but emptied its alert recipients, then rewrote a test to require the monitor be gone. What to change.
-
Liveness, Readiness, Startup: The Probe That Restarts Your Healthy Pods
Each Kubernetes probe answers a different question. Put the database in the wrong one and a 90-second blip becomes twelve restarts. A simulation and the fix.
-
Webhooks: Verify, Store, Say 200, Then Think
A webhook is a message queue whose producer belongs to someone else. The receiver pattern that survives retries, duplicates, replays and out-of-order events.
-
The Mitigation Was the Outage: Reading Google's us-central1 Storage Report Like an SRE
Google's preliminary report on the 8 Oct Cloud Storage incident: an optimisation exhausted the nodes it meant to relieve, and emergency capacity wasn't there.
-
A Backup You Have Never Restored Is a Hypothesis
RPO and RTO are promises you make before an incident and discover during one. Run the drill, time the restore, and find out which of your assumptions is wrong.
-
Raft: How a Cluster of Machines Agrees on a Single Truth
Consensus sounds academic until etcd loses quorum and your cluster freezes. Raft explained: leader election, log replication, and why majorities are the trick.
-
Load Balancing and Health Checks: The Three States Your Checks Don't Have
Balancing decides where load goes; health checks decide where it stops. Most outages here come from a check that can't tell busy apart from broken.
-
Idempotency Keys: The Thing That Makes a Retry Safe
A timeout never tells you whether the work happened. Idempotency keys resolve that — if you get the concurrency and scope right.
-
Rate Limiting Algorithms: Token Bucket, Leaky Bucket, and Sliding Window, Compared
Five algorithms, one trade-off: burst tolerance vs memory vs precision. Plus why LLM APIs limit tokens instead of requests.
-
Retries, Backoff, Jitter, and Circuit Breakers: The Four Controls That Decide Whether You Recover or Melt Down
A retry is a load multiplier. Backoff spreads it, jitter decorrelates it, a breaker stops it — and agent loops break all three.
-
The Model Hardware Standard is the first agent spec where rollback isn't an option
Anthropic's MHS gives agents read/write control of lab and factory hardware. What it guards, what it doesn't, and the controls you have to build yourself.
-
The runner is the API: why your remediator's contract matters more than your handlers
Everyone reviews the handler that restarts the pod. The safety lives in the runner around it — and handlers that skip it silently void every guarantee.
-
The 47th approval click is the bug
Human-in-the-loop is a real control until volume turns it into a reflex. How approval fatigue converts oversight into a rubber stamp — and the fix.
-
Why your AI agent should not be allowed to act yet
Reading is reversible. Acting is not. Five things that must exist before an agent gets a write path — and the honest reason advisory-forever is also a failure.
-
Your AI agent can't tell a quiet system from a broken collector
Autonomous remediation gates on model confidence and never on whether the telemetry is trustworthy. Telemetry integrity belongs in the gate too.
-
Your top AI failure mode is saturation, and the SLI is in a header you throw away
Datadog found ~5% of production AI requests fail, nearly 60% on capacity limits. Provider saturation is an SRE problem you already have the primitives for.
-
Cascading Failure: Backpressure, Load Shedding, and Graceful Degradation
A cascading failure is a feedback loop, not a row of dominoes. Removing the trigger does not stop it — which is why the controls have to be in place beforehand.
-
Timeouts and Deadline Propagation: The Budget Nobody Sets
A timeout is a local guess; a deadline is a shared fact. The difference decides whether your system sheds doomed work or grinds on abandoned requests.
-
Compaction is a reliability event, and you are not measuring it
New research on long-horizon agents finds context compression degrades execution, not knowledge. Treat every compaction as a state transition and instrument it.
-
Model end-of-life is a certificate expiry problem, and you already know how to solve it
Models retire on a schedule, the same model retires on different dates per platform, and 60 days' notice is a floor. Treat model EOL as inventory, not as news.
-
AI made your team faster. Your change failure rate noticed.
DORA's 2026 ROI research names the verification tax and the J-curve. For platform teams the lesson is blunt: codegen moved the bottleneck, it didn't remove it.
-
Agent workflows belong in code, not just visual builders
OpenAI is winding down Agent Builder and platform Evals. Production agent workflows need code, tests, traces, and reviewable contracts.
-
Agents are deleting production and nobody is writing the postmortem
Your incident template assumes a deterministic system and a human decision-maker. Agent incidents have neither. Five sections that fix the template.
-
The biggest outages still start with a config change — and now agents write them
Fresh postmortems keep confirming it: config changes, not code, cause the largest incidents. Agentic ops multiplies the volume. Here's the defense.
-
Google just published the reference architecture for agentic SRE
Google SRE splits autonomous incident response into reasoning, actuation, and evaluation. The split is the lesson — here's what to copy at normal scale.
-
The EU AI Act blinked — your logging requirements didn't
The EU AI Act omnibus moved high-risk deadlines to 2027–28. The logging, oversight, and inventory work is still yours — and reliability needs it anyway.
-
Every observability vendor now sells an AI SRE agent. Here's how to evaluate one.
Datadog, Dynatrace, New Relic, AWS — every incumbent now ships an AI SRE agent. A field guide for evaluating one before it touches production.
-
Frontier models still fail half your incidents: reading ITBench-AA like an SRE
ITBench-AA put frontier models against 59 real Kubernetes incident diagnoses — all scored below 50%. What the benchmark measures and how to use it.
-
Autonomy is a budget, not a toggle: error budgets for AI operators
SRE solved runaway release risk with error budgets. The same mechanism governs AI agents: authority per action class, demoted fast, promoted slow.
-
The reliability gap: a framework for trusting autonomous SRE agents
An autonomous airline agent rebooked 1,247 passengers wrong in one weather event. Trusting agents is a reliability problem — here's how to measure it.
-
Agentic AI Patterns: Where They Break in Production (Part 2 of 3)
Every agentic AI pattern looks clean in a demo. Where each one breaks in production, the signals you're hitting them, and mitigations that actually work.
-
Chaos engineering for MCP: break your tool-call plane before production does
LLM calls fail 1–5% of the time and agent tasks fan out into 10–20 tool calls. How to fault-inject your MCP layer with mcp-chaos before production does.
-
The trust gap: bounded autonomy for AI SRE agents
SREs face 50+ alerts a day at 60% false positives while vendors promise autonomous resolution. The autonomy ladder: what an AI agent should never do alone.
-
The AI-native SRE stack — a 2026 reference guide
A practitioner's map of the AI-native SRE stack in 2026: six layers from telemetry to bounded remediation, and an honest read on where AI pays off.
-
Context engineering: the window is a budget, not a bucket
The context window is your agent's working memory, not a junk drawer. Four operations — write, select, compress, isolate — and a token budget you allocate.
-
Agent sprawl is your next production incident
Teams shipping AI agents are recreating 2015's microservices sprawl with worse observability. The governance surface that contains it before it pages you.
-
Observability for AI systems — what changes when your service calls an LLM
Golden signals miss the failure that pages you: a confident, well-formed, wrong answer. What AI observability adds — context as a span, quality as a signal.
-
Harness engineering: the third phase of AI maturity
Agent = Model + Harness, and in 2026 the harness is the bottleneck. What a production-grade SRE harness contains, with a ~40-line reference implementation.
-
Toil and the 50% rule — what it is, how to measure it, and how to kill it
A primer on toil — the manual, repetitive work that eats SRE teams. Google's six-part definition, the 50% cap, and how AI agents change the playbook.
-
SLI, SLO, SLA, and error budgets — the reliability contract explained
SLIs, SLOs, SLAs, and error budgets — the four numbers every SRE team must agree on, the math behind 'nines,' and what changes when agents burn the budget.
-
What is Site Reliability Engineering (SRE)?
A primer on Site Reliability Engineering — where it came from, how it differs from DevOps and Platform Engineering, and what changes as AI joins on-call.
-
Mental models for applying AI to infrastructure
Tutorials answer how; mental models answer whether. Seven I use as the front gate before any LLM goes near a production system.
-
Prompt engineering for SRE: patterns that actually work in production
Prompt advice is written for chatbots; SRE workloads are different. Six patterns I've shipped to production for SRE LLM tools, and why each earned its place.
-
The MCP gateway pattern: five jobs your agent runtime can't skip
Letting agents call MCP servers directly repeats the no-API-gateway mistake. The five jobs an MCP gateway must do, with reproducible patterns for each.
-
Skills for AI agents that do SRE work
Most agent skills are chatbot prompts in disguise. Three operator-grade SRE skills — opinionated, output-contracted, portable across agent runtimes.
-
When NOT to Use AI in Production SRE
Most AI-for-SRE writing tells you where AI helps. Here are seven places it actively hurts — and the operational rule of thumb I use to decide.
-
Garbage Collection: The Convenience That Shows Up in Your p99
GC frees you from manual memory management — and occasionally freezes your program. Reachability, generational GC, and why GC tuning is a p99 problem.
-
Deadlocks, Locks, and Race Conditions: The Bugs That Only Happen Sometimes
Races and deadlocks only fail sometimes — that's what makes them hard. Locks, atomicity, the four Coffman conditions, and why hung agents are the same bug.