evals
6 posts — newest first.
-
Shadow Mode Isn't Evidence Until You Score It
In a seeded simulation, the agent with 95% concordance made 17 harmful high-severity proposals; the one with 88% made none. Score shadow mode properly.
-
Your Prompt Didn't Get Better. Your Eval Set Is Just Small.
Two identical prompts look 3 points apart in 32% of runs on a 50-case eval. How to compare LLM eval runs with paired tests, and how many cases you need.
-
Evals for AI SRE Agents: The Test Suite Your On-Call Bot Doesn't Have
Replayed incidents, frozen tool responses, deterministic graders, and pass^k instead of one lucky run. How to evaluate an AI SRE agent without fooling yourself.
-
A reasoning model just became a platform default. The lock-in isn't where you're looking.
Salesforce made Claude the default model across Agentforce and Slack. Where model dependency actually accrues, and the five exits worth keeping open.
-
Model end-of-life is a certificate expiry problem, and you already know how to solve it
Models retire on a schedule, the same model retires on different dates per platform, and 60 days' notice is a floor. Treat model EOL as inventory, not as news.
-
Agent workflows belong in code, not just visual builders
OpenAI is winding down Agent Builder and platform Evals. Production agent workflows need code, tests, traces, and reviewable contracts.