Incident Response
12 posts — newest first.
-
Evals for AI SRE Agents: The Test Suite Your On-Call Bot Doesn't Have
Replayed incidents, frozen tool responses, deterministic graders, and pass^k instead of one lucky run. How to evaluate an AI SRE agent without fooling yourself.
-
An agent breached Hugging Face in 4.5 days. The controls that would have stopped it are boring.
Hugging Face's July 2026 postmortem is the clearest agent-intrusion writeup we have. The failures were IMDS, admission policy, and one shared credential.
-
Agents are deleting production and nobody is writing the postmortem
Your incident template assumes a deterministic system and a human decision-maker. Agent incidents have neither. Five sections that fix the template.
-
Google just published the reference architecture for agentic SRE
Google SRE splits autonomous incident response into reasoning, actuation, and evaluation. The split is the lesson — here's what to copy at normal scale.
-
Your runbooks are lying to your agents: RAG for on-call knowledge
On-call agents are only as good as what they retrieve. Making runbook RAG fresh, exact-match aware, and poison-resistant — with a runnable starter repo.
-
Every observability vendor now sells an AI SRE agent. Here's how to evaluate one.
Datadog, Dynatrace, New Relic, AWS — every incumbent now ships an AI SRE agent. A field guide for evaluating one before it touches production.
-
Frontier models still fail half your incidents: reading ITBench-AA like an SRE
ITBench-AA put frontier models against 59 real Kubernetes incident diagnoses — all scored below 50%. What the benchmark measures and how to use it.
-
The reliability gap: a framework for trusting autonomous SRE agents
An autonomous airline agent rebooked 1,247 passengers wrong in one weather event. Trusting agents is a reliability problem — here's how to measure it.
-
Agentic AI Patterns: Where They Break in Production (Part 2 of 3)
Every agentic AI pattern looks clean in a demo. Where each one breaks in production, the signals you're hitting them, and mitigations that actually work.
-
The AI-native SRE stack — a 2026 reference guide
A practitioner's map of the AI-native SRE stack in 2026: six layers from telemetry to bounded remediation, and an honest read on where AI pays off.
-
Observability and incident response — the SRE basics
A primer on observability (logs, metrics, traces) and incident response (roles, severities, blameless postmortems) — the disciplines every SRE team runs.
-
Building incident-scribe: Slack Thread to Incident Report with Claude
How I built an open-source tool that turns messy Slack incident threads into blameless, structured incident reports in under 30 seconds.