Sometime in the last eight months, “AI SRE” stopped being a startup pitch and became a checkbox on every observability renewal. Datadog’s Bits AI went GA in December. Dynatrace launched its Intelligence layer at Perform 2026. New Relic announced an SRE Agent in March. AWS took a DevOps Agent GA in April with “75% lower MTTR” on the tin. If you run infrastructure, someone with a quota has already emailed you about at least two of these.
Category maturation is real — I said as much in the AI-native SRE stack. But market maturity and product maturity are different things, and the gap between them is exactly where bad purchases live. This post is the evaluation I’d run before letting any of them near production.
Know which quadrant you’re being sold
Mezmo’s 2026 market map cuts the space into four segments, and the segmentation holds up:
Closed AI SRE agents — Resolve AI, Traversal, Cleric, Neubird. Purpose-built platforms that own the whole investigation workflow. Typically the strongest raw investigation quality, at the cost of routing your telemetry through their cloud and reasoning you can’t fully inspect.
Incumbents shipping agents — Bits AI, Dynatrace Intelligence, New Relic SRE Agent, AWS’s and Azure’s entries. Lowest friction: they already see your data and inherit your access model. The structural weakness is that each reasons best over its own vendor’s telemetry — and your incidents have never respected vendor boundaries.
Telemetry substrates — Cribl, Apica, Metoro, Mezmo itself. Not agents; the data layer agents reason over. Their pitch is portability: shape and route telemetry so no agent above owns you.
Open-source harnesses — HolmesGPT, K8sGPT, OpenSRE. Full control, full portability, and full responsibility. You’re not buying an agent; you’re staffing one.
None of these is the wrong answer. But each fails differently, and vendors are understandably quiet about which failure you’re signing up for.
The three structural questions
Feature comparisons age in a quarter. These don’t:
1. Where does the telemetry go? If investigation happens in the vendor’s cloud, your incident data — including the embarrassing parts — now lives in someone else’s blast radius, retention policy, and compliance story. If your industry makes anonymous inference endpoints a non-starter, an agent shipping traces to a multi-tenant reasoning service deserves the same scrutiny.
2. Who controls the model and orchestration? Some products let you bring your own model and swap orchestration; others are the vendor’s model, the vendor’s loop, the vendor’s roadmap. This is the classic platform lock-in question wearing a new badge, and it matters more here because the underlying models are improving fast — you want to ride that curve, not wait for your vendor to.
3. What leaves with you? Investigation histories, learned service topology, tuned runbooks. If three years of institutional memory about how your systems fail is stored somewhere you can’t export, the switching cost compounds monthly.
The reliability questions no demo answers
Structural fit says whether you can adopt a product. Whether you should is an empirical question about reliability, and the demo cannot answer it — demos are sampled from the numerator.
I’ve written a full framework for deciding how much to trust an autonomous SRE agent, and the purchasing version compresses to this: measure the agent against your own history before it gets authority over your present. Concretely:
- Run it in shadow. Feed it real incidents — live ones read-only, or replayed ones — and score its conclusions against what your engineers actually found. Twenty incidents will teach you more than every case study the vendor ships.
- Score wrong answers, not just right ones. A tool that’s correct 70% of the time and confidently wrong 30% of the time is worse than one that’s correct 60% and says “I don’t know” for the rest — because the failure mode of a wrong-but-plausible root cause is your responders anchoring on it mid-incident.
- Gate authority on measured evidence. Whatever the agent earns — suggesting, then acting with approval, then acting within limits — should be a budget backed by its own track record, not a rollout date on a project plan.
Calibrate expectations with public benchmarks, too. ITBench’s results — frontier agents failing the large majority of realistic SRE scenarios — are a useful prior. They don’t mean the tools are useless; they mean any vendor number that starts with a 9 is an extraordinary claim.
Which brings up the numbers themselves. The market currently runs on stats like “94% root-cause accuracy,” and the sourcing hygiene is poor: Mezmo’s analysts note that one startup’s widely repeated “82% RCA accuracy” figure at a marquee customer doesn’t appear in any source they could confirm. Sort every claim into independently verified, vendor-reported with a named customer, or unattributed percentage — and notice how much of the category’s marketing lives in bucket three.
Honest caveats
The incumbents’ bundling is genuinely valuable, not just lock-in — an agent that inherits your existing RBAC and billing is weeks faster to adopt, and for a single-vendor shop the cross-platform weakness may never bite. The closed agents’ quality lead is real in several public bake-offs, and “routes telemetry through their cloud” is a solved compliance problem for plenty of teams. Open source is only free if your platform team’s time is. And shadow evaluation has a cost too: someone has to build the replay set and score the outputs. Budget for it — it’s still the cheapest insurance in this market.
What to do Monday
- Write down which quadrant you’re actually shopping in — and which failure mode (data control, lock-in, maintenance) you’re most willing to own. This kills half the vendor list politely.
- Assemble a 20-incident replay set from your last two quarters: the alert payloads, the timeline, and what the root cause turned out to be. This is your benchmark; it beats every public one because it’s sampled from your distribution.
- Ask every vendor the three structural questions in writing. The answers age well precisely because they’re the ones sales decks avoid.
- Demand shadow mode in the POC. Score correct, incorrect-but-flagged, and confidently wrong separately. The third column decides.
- Pre-commit the authority ladder. Before the pilot, write down what evidence promotes the agent from suggesting to acting — so the decision gets made by your criteria, not by renewal-quarter enthusiasm.
The AI SRE category arriving is good news; the on-call load it targets is real, and I’ve spent two years arguing this is where AI genuinely helps. But an agent you adopted on vendor benchmarks is exactly the kind you’ll regret at 3 a.m. Buy the one that survived your incidents on paper before it met them in production.