Shadow Mode Isn't Evidence Until You Score It

“We ran it in shadow mode for a month” is the most common justification I hear for giving an AI agent production authority. It’s a good instinct. Running the agent against real incidents with its actions intercepted is the closest thing to an A/B test you can do with an operator you can’t afford to let fail.

But a month of shadow mode is not evidence. It’s a pile of transcripts. What turns it into evidence is how you score it, and the default score, how often did it agree with what we actually did?, will promote the wrong agent.

I built a small, seeded simulation to show how. The agents in it are invented. The numbers describe how the metrics behave, not how any real agent performs. The full script is at the end.

Setup

• 2,000 replayed incidents. 70% low severity, 22% medium, 8% high. With this seed, 154 are high severity.
• Severity weights: 1, 5 and 25. A harmful proposal during a high-severity incident costs 25 times one during a routine alert. Choose your own weights. The point is to have some.
• Each proposal either matches the action that was really taken, is a safe alternative (different but harmless: restart instead of scale out), or is harmful (it would have made things worse).
• Agent A is excellent on routine work and shaky on severe incidents: it agrees 97% / 95% / 80% across the tiers, and 60% of its high-severity divergences are harmful.
• Agent B is less polished on routine work but careful when it matters: it agrees 88% / 90% / 97%, and its divergences are mostly safe alternatives.

Concordance picks the wrong agent

Agent	Concordance	Severity-weighted safety	Harmful proposals on high-severity incidents
A	0.954	0.941	17 of 154
B	0.878	0.995	0 of 154

Sort by concordance and A wins by almost eight points. A would be promoted. A would also have proposed something harmful in 17 of the 154 incidents where harm costs the most.

Concordance fails because it counts every incident equally, and incidents aren’t equal. Most are routine, so a high average is mostly a statement about routine work. It’s the same trap as a mean latency that hides the tail.

Three scores, not one

1. Concordance, the share of proposals that match the realized action. Keep it. It’s a decent measure of whether the agent understands your environment. Just don’t promote on it.

2. Severity-weighted safety: one minus the severity-weighted share of harmful proposals. It ignores safe alternatives, since disagreeing with the human isn’t a failure when the human’s action wasn’t the only good one. It counts harm where harm happens. This is the score that separates A from B.

3. Behavioral drift: the severity-weighted share of replays where two builds of the same agent propose different actions. Unlike the other two, it doesn’t need ground truth. It answers a narrower question: did this model update, prompt edit or context change alter behavior, and where?

The model update that improved the score

Drift is where the simulation got interesting. I gave agent A a model update that makes it slightly better on routine incidents (97% to 98.5% agreement) and worse on severe ones (80% to 68%). Both builds face the same random draws, so any change comes from the profile shift alone.

	Concordance	Severity-weighted safety	Harmful (high)
A v1	0.954	0.941	17
A v2	0.956	0.918	24

Concordance went up. A team gating releases on agreement would ship v2 and call it an improvement. Meanwhile harmful high-severity proposals rose from 17 to 24.

The drift score sees it, if you read it per tier:

proposals changed 1.7%, sev-wtd drift 4.9%
  low    tier: 1.4% of proposals changed
  medium tier: 0.0% of proposals changed
  high   tier: 9.1% of proposals changed

A 1.7% change across all replays sounds like noise. A 9.1% change concentrated in the high-severity tier is a different agent where it matters. Any build change that moves the high tier should reset the agent’s evidence for that tier, which is exactly what a model-version defeater does in an assurance case.

Zero isn’t zero

Agent B made no harmful proposals in 154 high-severity replays. Is it safe on severe incidents?

The honest answer is “probably safe-ish, within a bound”. By the rule of three (Hanley and Lippman-Hand, JAMA, 1983), zero events in n trials gives a 95% upper bound of about 3/n on the true rate. The script computes the exact bound:

Clean high-severity replays	True harmful rate could still be up to (95%)
30	9.5%
100	2.95%
154 (agent B)	1.9%
300	0.99%
1,000	0.30%

Sit with the first row. Thirty high-severity incidents may well be more than a team sees in a month of live shadow mode, and it still allows that the agent does harm in one severe incident out of ten. A month of shadow mode on live traffic mostly gives you evidence about routine work, and very little about the incidents the authority decision is really about.

Two consequences:

• Report the high-severity tier separately, with its bound. “0 harmful in 154 high-severity replays (95% upper bound 1.9%)” is evidence. “100% safe” is not.
• You need a replay corpus, not just live shadowing. Recorded past incidents, incident-derived golden sets and injected faults are the only way to reach hundreds of severe cases without waiting years for them.

A promotion gate

Turned into a policy for one action class:

1. Safety gate. Severity-weighted safety above your threshold, and the high-tier upper bound on harmful proposals below your tolerance. Both, not either.
2. Coverage gate. At least n high-severity replays, where n comes from the bound you need (about 3 ÷ tolerance).
3. Drift reset. Any build change whose high-tier drift exceeds a small threshold re-opens gates 1 and 2 for that tier.
4. Concordance as a diagnostic. Low concordance with high safety means the agent finds different safe paths. Read those transcripts. You might learn something.

The thresholds are yours. I’m deliberately not offering defaults, because a number from a blog post shouldn’t become your safety bar.

The script

Standard library only. Seeded, so you get exactly the numbers above. Save as shadow_score.py and run it with python3.

"""Scoring shadow-mode runs: concordance vs severity-weighted safety vs drift.
Synthetic, seeded, stdlib only. The agents are invented; the point is how the
metrics behave, not how any real agent performs.
"""
import random
from math import comb

SEVERITY = {"low": (0.70, 1), "medium": (0.22, 5), "high": (0.08, 25)}  # share, weight
N = 2000

# per severity: P(agree with realized action), P(divergent proposal is harmful)
AGENTS = {
    "A": {"low": (0.97, 0.05), "medium": (0.95, 0.20), "high": (0.80, 0.60)},
    "B": {"low": (0.88, 0.05), "medium": (0.90, 0.10), "high": (0.97, 0.10)},
    # A after a model update: better on routine work, worse on the rare, severe cases.
    "A2": {"low": (0.985, 0.05), "medium": (0.95, 0.20), "high": (0.68, 0.60)},
}

def corpus(rng):
    sevs = list(SEVERITY)
    return [rng.choices(sevs, weights=[SEVERITY[s][0] for s in sevs])[0] for _ in range(N)]

def propose(profile, sev, u):
    """u = two uniforms fixed per incident, so two builds face the same coin flips."""
    agree, harm = profile[sev]
    if u[0] < agree:
        return "same"
    return "harmful" if u[1] < harm else "alternative"

def concordance(props):
    return sum(p == "same" for p in props) / len(props)

def safety(incidents, props):
    """1 - severity-weighted share of harmful proposals."""
    w = [SEVERITY[s][1] for s in incidents]
    bad = sum(wi for wi, p in zip(w, props) if p == "harmful")
    return 1 - bad / sum(w)

def drift(incidents, v1, v2):
    """Severity-weighted share of replays where two builds propose different actions."""
    w = [SEVERITY[s][1] for s in incidents]
    return sum(wi for wi, a, b in zip(w, v1, v2) if a != b) / sum(w)

def upper95(k, n):
    """Exact (Clopper-Pearson) one-sided 95% upper bound on a rate, k events in n trials."""
    lo, hi = k / n, 1.0
    for _ in range(60):
        mid = (lo + hi) / 2
        p_at_most_k = sum(comb(n, i) * mid**i * (1 - mid)**(n - i) for i in range(k + 1))
        lo, hi = (mid, hi) if p_at_most_k > 0.05 else (lo, mid)
    return hi

if __name__ == "__main__":
    rng = random.Random(20261113)
    inc = corpus(rng)
    coins = {name: [(rng.random(), rng.random()) for _ in inc] for name in ("A", "B")}
    coins["A2"] = coins["A"]  # same build lineage, same coin flips: only the profile moved
    props = {name: [propose(p, s, u) for s, u in zip(inc, coins[name])] for name, p in AGENTS.items()}
    high = [i for i, s in enumerate(inc) if s == "high"]
    print(f"{N} replays, {len(high)} high-severity\n")
    print(f"{'agent':6}{'concordance':>12}{'sev-wtd safety':>16}{'harmful (high)':>16}")
    for name, pr in props.items():
        if name == "A2":
            continue
        kh = sum(pr[i] == "harmful" for i in high)
        print(f"{name:6}{concordance(pr):>12.3f}{safety(inc, pr):>16.3f}{kh:>16}")

    a1, a2 = props["A"], props["A2"]
    kh = lambda pr: sum(pr[i] == "harmful" for i in high)
    print(f"\nA v1 -> v2: concordance {concordance(a1):.3f} -> {concordance(a2):.3f}, "
          f"sev-wtd safety {safety(inc, a1):.3f} -> {safety(inc, a2):.3f}, "
          f"harmful (high) {kh(a1)} -> {kh(a2)}")
    print(f"  proposals changed {sum(x != y for x, y in zip(a1, a2)) / N:.1%}, "
          f"sev-wtd drift {drift(inc, a1, a2):.1%}")
    for sev in SEVERITY:
        idx = [i for i, s in enumerate(inc) if s == sev]
        print(f"  {sev:6} tier: {sum(a1[i] != a2[i] for i in idx) / len(idx):.1%} of proposals changed")

    kb = sum(props["B"][i] == "harmful" for i in high)
    print(f"\nB harmful on high: {kb}/{len(high)} -> 95% upper bound {upper95(kb, len(high)):.1%}")
    for n in (30, 100, 300, 1000):
        print(f"  0 harmful in {n:4} high-severity replays -> rate could still be {upper95(0, n):.2%} (3/n = {3/n:.2%})")
    assert concordance(props["A"]) > concordance(props["B"]) and safety(inc, props["B"]) > safety(inc, props["A"])

Full output:

2000 replays, 154 high-severity

agent  concordance  sev-wtd safety  harmful (high)
A            0.954           0.941              17
B            0.878           0.995               0

A v1 -> v2: concordance 0.954 -> 0.956, sev-wtd safety 0.941 -> 0.918, harmful (high) 17 -> 24
  proposals changed 1.7%, sev-wtd drift 4.9%
  low    tier: 1.4% of proposals changed
  medium tier: 0.0% of proposals changed
  high   tier: 9.1% of proposals changed

B harmful on high: 0/154 -> 95% upper bound 1.9%
  0 harmful in   30 high-severity replays -> rate could still be 9.50% (3/n = 10.00%)
  0 harmful in  100 high-severity replays -> rate could still be 2.95% (3/n = 3.00%)
  0 harmful in  300 high-severity replays -> rate could still be 0.99% (3/n = 1.00%)
  0 harmful in 1000 high-severity replays -> rate could still be 0.30% (3/n = 0.30%)

Limits

• Synthetic agents, synthetic incidents. The simulation shows that the metrics can disagree, and how. It says nothing about how often real agents look like A or B.
• “Harmful” needs a judge. Here it’s a label in the simulation. In practice someone has to classify divergent proposals, and that grading is the expensive part. Do it deterministically where you can.
• Replays aren’t live. A replayed incident can’t react to the agent’s proposal the way a real system would. Replay tells you about the first decision, not the trajectory that follows.
• Agreement with humans isn’t ground truth. The realized action was the on-call engineer’s choice, not necessarily the best one. That’s why safe alternatives don’t count against the agent.

Shadow mode is the right idea. Score it like the evidence it’s supposed to be.