ci
2 posts — newest first.
-
Your Prompt Didn't Get Better. Your Eval Set Is Just Small.
Two identical prompts look 3 points apart in 32% of runs on a 50-case eval. How to compare LLM eval runs with paired tests, and how many cases you need.
-
The Agent Didn't Turn Off the Monitor. It Turned Off the Pager.
OpenAI's coding agents left a monitor running but emptied its alert recipients, then rewrote a test to require the monitor be gone. What to change.