In July I told anyone evaluating an AI SRE agent to assemble a 20-incident replay set from their own history. The obvious follow-up questions are: how, exactly? And what does a score on twenty incidents actually tell you?
This is the how. It’s also a warning about the second question, because the honest answer is “less than the dashboard implies.”
Evals are not unit tests
A unit test asks: given this input, does the code return this value? It’s deterministic, so one run is enough.
An AI SRE agent breaks every part of that. Its output isn’t a value, it’s a diagnosis plus a list of proposed actions. The same incident can produce a different investigation path on every run, because sampling makes different tool calls look attractive. And “correct” is fuzzy: a diagnosis can name the right cause, name it alongside two wrong ones, or name a symptom of it.
So an eval measures a rate: across a representative set of incidents, run several times each, how often does the agent reach an acceptable outcome? Everything below is about making that rate mean something.
1. Build the golden set from your own incidents
Public benchmarks are useful for reading the market — I went through what ITBench says about AI SRE in July. They aren’t useful for deciding whether an agent works on your systems, because their incidents aren’t drawn from your distribution.
For each incident in your set, capture:
- The trigger: the alert payload exactly as it fired.
- The evidence: every metric, log, trace and config the responders looked at — or could have.
- The answer, as labels: the confirmed root cause written as a small set of tags (
bad_deploy,connection_pool_exhausted), not a paragraph. Labels can be graded by code; prose can’t. - Forbidden actions: anything that would have made it worse. Restarting the database during a replication lag incident, for example.
Write the labels before you run any agent. If you write them after reading the agent’s output, you’ll unconsciously grade towards what it said.
Then stratify. Include easy incidents and hard ones. Include the boring categories that make up most of your pages, not just the war stories. And include incidents where the right answer was “nothing is wrong; the alert is noisy.” An agent that always finds a root cause is a confident liar, and you won’t find out unless some of your test cases have no cause to find.
2. Freeze the world
Your incidents happened weeks ago. The telemetry has rotated out, the deployments have moved on, and if the agent queries live systems it’s investigating today, not the incident.
So record the tool responses and replay them. This is the cassette pattern from HTTP testing: each tool call is keyed by the tool name and its arguments, and the harness serves the recorded response instead of calling anything live. Now the inputs are fixed, and any variance between runs comes from the agent alone.
Two details matter.
An unrecorded call is a finding, not a crash. When the agent asks for something you didn’t record, return an explicit “no recording” response and log it. Either the agent explored a reasonable path you didn’t anticipate — extend the recording — or it went somewhere it had no business going, which is worth knowing too.
Watch for contamination. If the incident’s postmortem is in the agent’s retrieval index, the agent may retrieve the answer rather than diagnose it. That’s the same risk as RAG over your runbooks in reverse. Exclude eval incidents’ postmortems from the index during eval runs, or your scores measure search, not reasoning.
3. Grade deterministically first
Do everything you can in code before you reach for a model to grade a model.
- Root cause: compare the agent’s labels to the golden labels. Exact match is a pass. Partial overlap — the right cause plus wrong ones — gets partial credit, and you decide whether partial counts.
- Safety: if the agent proposed any forbidden action, the run scores zero regardless of how good the diagnosis was. A correct root cause followed by “scale the database to zero” is a failed run. Safety is a gate, not a component of a weighted average.
Use an LLM judge only for what code genuinely can’t check, such as whether the explanation would help a tired engineer at 3am. When you do, know the documented failure modes. The canonical study, Zheng et al. (2023), found:
- Position bias: when two similar answers were swapped, judges often changed their verdict. Only GPT-4 gave consistent results in more than 60% of cases on their test.
- Verbosity bias: in a “repetitive list” attack — padding an answer with a rephrased copy of its own list, adding no information — judges often preferred the padded version, though GPT-4 resisted far better than the others.
- Possible self-preference: GPT-4 showed a 10% higher win rate for its own answers and Claude-v1 a 25% higher one — though the authors say plainly that with their limited data they can’t determine whether this is a real self-enhancement bias.
Those were 2023 models, so don’t carry the numbers forward. The mechanisms are what to carry. The paper’s own mitigations are cheap: swap the order and count a win only when it holds both ways, and give the judge a reference answer. On their math questions, adding a reference answer cut the judge’s failure rate from 14 of 20 to 3 of 20. For SRE evals that means never asking “is this diagnosis good?” — always “does this diagnosis match this confirmed root cause?“
4. Run every incident k times
One run per incident tells you almost nothing about a sampling system. Run each one several times and look at two different numbers.
pass@k is the chance that at least one of k runs succeeds. It answers “can the agent find this?”
pass^k — “pass hat k,” from the τ-bench paper (Yao et al., 2024) — is the chance that all k runs succeed. It answers “does the agent find this every time?”
For on-call, pass^k is the one that matters. If an agent gets the same incident right three times out of five, two different engineers running it on the same page will be given two different stories, and one of them will spend an hour chasing the wrong one. The τ-bench authors saw exactly this gap in customer-service tasks: a GPT-4o agent with over 60% average success fell below 25% on pass^8.
With n runs of an incident and c successes, the unbiased estimators from the paper are:
- pass^k = C(c, k) / C(n, k)
- pass@k = 1 − C(n − c, k) / C(n, k)
averaged across incidents.
5. Count honestly
Here’s the part most eval dashboards skip. The unit of evidence is the incident, not the run. Twenty incidents run five times each is not a sample of 100. Runs on the same incident share the same difficulty, the same evidence and the same trap, so they’re correlated. You have twenty data points with some extra resolution on each.
Twenty data points don’t buy much precision. Using a Wilson interval, which behaves well at small sample sizes:
| Result | 95% interval |
|---|---|
| 12 of 20 incidents (60%) | 39% to 78% |
| 14 of 20 incidents (70%) | 48% to 85% |
| 60 of 100 incidents (60%) | 50% to 69% |
A twenty-incident set can tell you an agent is bad, or roughly workable. It can’t tell you whether a new model at 70% beats the old one at 60% — those intervals overlap almost completely.
For comparing two versions on the same set, compare paired results: which specific incidents flipped from pass to fail, and which from fail to pass? (Formally, that’s McNemar’s test.) Three incidents improving and three regressing is a wash, however the averages moved. And read every regression yourself. The flips are where you learn what changed.
The harness
Standard library only. The agent is a stand-in so the harness runs end to end; replace fake_agent with a call to yours.
import json
import math
import random
from dataclasses import dataclass, field
# --- 1. A golden incident: the alert, the recorded tool responses, the known answer.
@dataclass
class Incident:
id: str
alert: str
cassette: dict # (tool, canonical args) -> recorded response
root_causes: set # the labels a correct diagnosis must name
forbidden_actions: set = field(default_factory=lambda: {"delete", "scale_to_zero", "drop_table"})
def replay_tool(incident, tool, args, misses):
"""Serve tool calls from the recording. An unrecorded call is a finding, not a crash."""
key = (tool, json.dumps(args, sort_keys=True))
if key not in incident.cassette:
misses.append(key)
return {"error": "no recording for this call"}
return incident.cassette[key]
# --- 2. Deterministic grading first. Only reach for an LLM judge for what this can't check.
def grade(incident, result):
if set(result["actions"]) & incident.forbidden_actions:
return 0.0 # an unsafe action fails the run outright
named = set(result["root_causes"])
if not named & incident.root_causes:
return 0.0
precision = len(named & incident.root_causes) / len(named)
recall = len(named & incident.root_causes) / len(incident.root_causes)
return 1.0 if precision == 1 and recall == 1 else 0.5
# --- 3. Metrics that respect how few incidents you have.
def pass_hat_k(n, c, k):
"""tau-bench: probability that all k trials succeed, from c successes in n trials."""
return math.comb(c, k) / math.comb(n, k)
def pass_at_k(n, c, k):
"""Probability that at least one of k trials succeeds."""
return 1 - math.comb(n - c, k) / math.comb(n, k)
def wilson(successes, n, z=1.96):
"""95% interval for a proportion. Honest at small n, unlike +/- 2 standard errors."""
p = successes / n
centre = (p + z * z / (2 * n)) / (1 + z * z / n)
half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
return centre - half, centre + half
# --- 4. Run it. A stand-in agent so the harness runs end to end; swap in yours.
def fake_agent(incident, rng, skill):
misses = []
replay_tool(incident, "get_metrics", {"service": "checkout"}, misses)
right = rng.random() < skill
return {
"root_causes": sorted(incident.root_causes) if right else ["upstream_dns"],
"actions": ["rollback"] if right else ["restart"],
"misses": misses,
}
rng = random.Random(11)
incidents = [Incident(f"INC-{n:03}", "checkout p99 > 2s",
{("get_metrics", json.dumps({"service": "checkout"})): {"p99_ms": 2400}},
{"bad_deploy"}) for n in range(20)]
skills = [rng.choice([0.95, 0.8, 0.6, 0.3]) for _ in incidents] # some incidents are harder
TRIALS, K = 5, 5
per_incident = []
for inc, skill in zip(incidents, skills):
c = sum(grade(inc, fake_agent(inc, rng, skill)) == 1.0 for _ in range(TRIALS))
per_incident.append(c)
p1 = sum(c / TRIALS for c in per_incident) / len(incidents)
pk = sum(pass_hat_k(TRIALS, c, K) for c in per_incident) / len(incidents)
pak = sum(pass_at_k(TRIALS, c, K) for c in per_incident) / len(incidents)
lo, hi = wilson(round(p1 * len(incidents)), len(incidents))
print(f"incidents: {len(incidents)}, trials each: {TRIALS}")
print(f"pass^1 (mean success): {p1:.0%}")
print(f"pass@{K} (solved at least once): {pak:.0%}")
print(f"pass^{K} (solved every time): {pk:.0%}")
print(f"95% interval on pass^1 at n={len(incidents)}: {lo:.0%} to {hi:.0%}")Output:
incidents: 20, trials each: 5
pass^1 (mean success): 72%
pass@5 (solved at least once): 95%
pass^5 (solved every time): 25%
95% interval on pass^1 at n=20: 48% to 85%Same stand-in agent, three very different headlines. “It solves 95% of incidents” and “it reliably solves 25% of incidents” are both true of this run. Only one of them should decide whether it goes on the rota.
6. Make it a gate
An eval you run once is a report. An eval that runs on every change is a control.
Re-run the suite whenever any of these change: the model version, the system prompt, the tool definitions, the retrieval index. Model versions in particular change underneath you on the provider’s schedule, which is why I treat model end-of-life like certificate expiry — and why the eval suite is the thing that tells you whether the replacement is safe to promote. Track pass^k and the flip list over time, and block promotion on new safety failures outright.
What evals can’t tell you
An eval set is a sample of your past. It won’t tell you how the agent handles an incident type you’ve never had, how it degrades as your systems drift away from what the set captured, or whether your engineers will trust it. Those still need shadow mode on live incidents and someone reading the transcripts.
But without the eval set, you can’t tell whether last week’s model upgrade made the agent better or worse — and that’s the question you’ll be asked first when something goes wrong.
Related: Every observability vendor now sells an AI SRE agent · Frontier models still fail half your incidents · Observability for Agents
Comments