testing
2 posts — newest first.
-
Your Prompt Didn't Get Better. Your Eval Set Is Just Small.
Two identical prompts look 3 points apart in 32% of runs on a 50-case eval. How to compare LLM eval runs with paired tests, and how many cases you need.
-
Evals for AI SRE Agents: The Test Suite Your On-Call Bot Doesn't Have
Replayed incidents, frozen tool responses, deterministic graders, and pass^k instead of one lucky run. How to evaluate an AI SRE agent without fooling yourself.