statistics
4 posts — newest first.
-
Shadow Mode Isn't Evidence Until You Score It
In a seeded simulation, the agent with 95% concordance made 17 harmful high-severity proposals; the one with 88% made none. Score shadow mode properly.
-
Your Prompt Didn't Get Better. Your Eval Set Is Just Small.
Two identical prompts look 3 points apart in 32% of runs on a 50-case eval. How to compare LLM eval runs with paired tests, and how many cases you need.
-
Evals for AI SRE Agents: The Test Suite Your On-Call Bot Doesn't Have
Replayed incidents, frozen tool responses, deterministic graders, and pass^k instead of one lucky run. How to evaluate an AI SRE agent without fooling yourself.
-
Anomaly Detection for SREs: Thresholds, Baselines, and Where the LLM Belongs
Static thresholds fire every night or never. Rolling baselines learn the outage. A seasonal robust baseline fixes both; the LLM belongs after the detector.