SRE
48 posts — newest first.
-
The Mitigation Was the Outage: Reading Google's us-central1 Storage Report Like an SRE
Google's preliminary report on the 8 Oct Cloud Storage incident: an optimisation exhausted the nodes it meant to relieve, and emergency capacity wasn't there.
-
Evals for AI SRE Agents: The Test Suite Your On-Call Bot Doesn't Have
Replayed incidents, frozen tool responses, deterministic graders, and pass^k instead of one lucky run. How to evaluate an AI SRE agent without fooling yourself.
-
Anomaly Detection for SREs: Thresholds, Baselines, and Where the LLM Belongs
Static thresholds fire every night or never. Rolling baselines learn the outage. A seasonal robust baseline fixes both; the LLM belongs after the detector.
-
A Backup You Have Never Restored Is a Hypothesis
RPO and RTO are promises you make before an incident and discover during one. Run the drill, time the restore, and find out which of your assumptions is wrong.
-
Load Balancing and Health Checks: The Three States Your Checks Don't Have
Balancing decides where load goes; health checks decide where it stops. Most outages here come from a check that can't tell busy apart from broken.
-
Retries, Backoff, Jitter, and Circuit Breakers: The Four Controls That Decide Whether You Recover or Melt Down
A retry is a load multiplier. Backoff spreads it, jitter decorrelates it, a breaker stops it — and agent loops break all three.
-
The runner is the API: why your remediator's contract matters more than your handlers
Everyone reviews the handler that restarts the pod. The safety lives in the runner around it — and handlers that skip it silently void every guarantee.
-
The 47th approval click is the bug
Human-in-the-loop is a real control until volume turns it into a reflex. How approval fatigue converts oversight into a rubber stamp — and the fix.
-
Why your AI agent should not be allowed to act yet
Reading is reversible. Acting is not. Five things that must exist before an agent gets a write path — and the honest reason advisory-forever is also a failure.
-
Your AI agent can't tell a quiet system from a broken collector
Autonomous remediation gates on model confidence and never on whether the telemetry is trustworthy. Telemetry integrity belongs in the gate too.
-
Your top AI failure mode is saturation, and the SLI is in a header you throw away
Datadog found ~5% of production AI requests fail, nearly 60% on capacity limits. Provider saturation is an SRE problem you already have the primitives for.
-
Queueing Theory for SREs: Little's Law and the Utilisation Knee
Latency doesn't rise with load, it rises with 1/(1−utilisation). Two formulas explain the p99 cliff and size every pool in your system.
-
Cascading Failure: Backpressure, Load Shedding, and Graceful Degradation
A cascading failure is a feedback loop, not a row of dominoes. Removing the trigger does not stop it — which is why the controls have to be in place beforehand.
-
Compaction is a reliability event, and you are not measuring it
New research on long-horizon agents finds context compression degrades execution, not knowledge. Treat every compaction as a state transition and instrument it.
-
Progressive Delivery: Canary, Blue-Green, and the Rollback You Never Tested
Every deployment strategy answers one question: who pays for the bug? Rolling, blue-green, and canary answer differently — only one limits blast radius.
-
AI made your team faster. Your change failure rate noticed.
DORA's 2026 ROI research names the verification tax and the J-curve. For platform teams the lesson is blunt: codegen moved the bottleneck, it didn't remove it.
-
Agent workflows belong in code, not just visual builders
OpenAI is winding down Agent Builder and platform Evals. Production agent workflows need code, tests, traces, and reviewable contracts.
-
Agents are deleting production and nobody is writing the postmortem
Your incident template assumes a deterministic system and a human decision-maker. Agent incidents have neither. Five sections that fix the template.
-
AI SRE is officially a category. Buy it like an SRE, not like a CIO.
Gartner's first Market Guide for AI SRE tooling projects 85% adoption by 2029. Six evaluation questions for cutting through the category hype.
-
The biggest outages still start with a config change — and now agents write them
Fresh postmortems keep confirming it: config changes, not code, cause the largest incidents. Agentic ops multiplies the volume. Here's the defense.
-
Google just published the reference architecture for agentic SRE
Google SRE splits autonomous incident response into reasoning, actuation, and evaluation. The split is the lesson — here's what to copy at normal scale.
-
Your runbooks are lying to your agents: RAG for on-call knowledge
On-call agents are only as good as what they retrieve. Making runbook RAG fresh, exact-match aware, and poison-resistant — with a runnable starter repo.
-
Every observability vendor now sells an AI SRE agent. Here's how to evaluate one.
Datadog, Dynatrace, New Relic, AWS — every incumbent now ships an AI SRE agent. A field guide for evaluating one before it touches production.
-
Your agents need identities, not API keys
Every AI agent is a non-human identity — most run on shared, long-lived API keys no IAM review sees. Per-agent identity and your credential blast radius.
-
Token FinOps: the third budget your agents are spending
Error budgets, context budgets — agents add a third: dollars. Agent tasks burn 5–30× chatbot tokens, and cost-per-token is the wrong metric.
-
Tracing the agent loop: OpenTelemetry's GenAI conventions, read like an SRE
Your agent is a distributed system wearing a chat interface. OpenTelemetry's GenAI conventions make it debuggable — what v1.41 covers and what's moving.
-
Frontier models still fail half your incidents: reading ITBench-AA like an SRE
ITBench-AA put frontier models against 59 real Kubernetes incident diagnoses — all scored below 50%. What the benchmark measures and how to use it.
-
Autonomy is a budget, not a toggle: error budgets for AI operators
SRE solved runaway release risk with error budgets. The same mechanism governs AI agents: authority per action class, demoted fast, promoted slow.
-
The reliability gap: a framework for trusting autonomous SRE agents
An autonomous airline agent rebooked 1,247 passengers wrong in one weather event. Trusting agents is a reliability problem — here's how to measure it.
-
Chaos engineering for MCP: break your tool-call plane before production does
LLM calls fail 1–5% of the time and agent tasks fan out into 10–20 tool calls. How to fault-inject your MCP layer with mcp-chaos before production does.
-
The trust gap: bounded autonomy for AI SRE agents
SREs face 50+ alerts a day at 60% false positives while vendors promise autonomous resolution. The autonomy ladder: what an AI agent should never do alone.
-
The AI-native SRE stack — a 2026 reference guide
A practitioner's map of the AI-native SRE stack in 2026: six layers from telemetry to bounded remediation, and an honest read on where AI pays off.
-
Agent sprawl is your next production incident
Teams shipping AI agents are recreating 2015's microservices sprawl with worse observability. The governance surface that contains it before it pages you.
-
Observability for AI systems — what changes when your service calls an LLM
Golden signals miss the failure that pages you: a confident, well-formed, wrong answer. What AI observability adds — context as a span, quality as a signal.
-
Harness engineering: the third phase of AI maturity
Agent = Model + Harness, and in 2026 the harness is the bottleneck. What a production-grade SRE harness contains, with a ~40-line reference implementation.
-
Observability and incident response — the SRE basics
A primer on observability (logs, metrics, traces) and incident response (roles, severities, blameless postmortems) — the disciplines every SRE team runs.
-
Toil and the 50% rule — what it is, how to measure it, and how to kill it
A primer on toil — the manual, repetitive work that eats SRE teams. Google's six-part definition, the 50% cap, and how AI agents change the playbook.
-
SLI, SLO, SLA, and error budgets — the reliability contract explained
SLIs, SLOs, SLAs, and error budgets — the four numbers every SRE team must agree on, the math behind 'nines,' and what changes when agents burn the budget.
-
What is Site Reliability Engineering (SRE)?
A primer on Site Reliability Engineering — where it came from, how it differs from DevOps and Platform Engineering, and what changes as AI joins on-call.
-
Mental models for applying AI to infrastructure
Tutorials answer how; mental models answer whether. Seven I use as the front gate before any LLM goes near a production system.
-
Prompt engineering for SRE: patterns that actually work in production
Prompt advice is written for chatbots; SRE workloads are different. Six patterns I've shipped to production for SRE LLM tools, and why each earned its place.
-
Skills for AI agents that do SRE work
Most agent skills are chatbot prompts in disguise. Three operator-grade SRE skills — opinionated, output-contracted, portable across agent runtimes.
-
Alert fatigue? Let AI triage.
How I built alert-explainer — an open-source service that turns every Prometheus alert into a plain-English brief in 1–4 seconds for under a cent.
-
When NOT to Use AI in Production SRE
Most AI-for-SRE writing tells you where AI helps. Here are seven places it actively hurts — and the operational rule of thumb I use to decide.
-
Building incident-scribe: Slack Thread to Incident Report with Claude
How I built an open-source tool that turns messy Slack incident threads into blameless, structured incident reports in under 30 seconds.
-
Why AI is the Next SRE Superpower
After 15 years in cloud infrastructure and SRE, why I believe AI is the most significant shift in how we operate systems since Kubernetes.
-
NAS vs SAN – A brief comparison.
NAS serves files over ethernet; SAN serves blocks over a dedicated network. A brief comparison of the two storage architectures and when to pick each.
-
What is RAID?
RAID provides redundancy for stored data, protecting against disk failures. The common RAID levels, how they differ, and when to use each.