ai-engineering-measured
10 posts — newest first.
-
340 of the Top 1,000 GitHub Repos Ship an AGENTS.md. I Linted Them.
A third of the most-starred repos carry agent instructions. What's in them, what research says belongs there, and why my linter was wrong at first.
-
A Three-Agent Vote Can Add 9.6 Points at Most: The Break-Even Math of Multi-Agent Systems
Closed-form models for voting, verifiers, pipelines and fan-out, checked against Monte Carlo and the research. When a second agent pays for itself.
-
I Killed an Agent 120 Times to See What Durable Execution Actually Buys You
SIGKILL at every step boundary under four persistence policies. Only one had zero duplicate side effects, and the fix wasn't on the agent's side.
-
Your Prompt Didn't Get Better. Your Eval Set Is Just Small.
Two identical prompts look 3 points apart in 32% of runs on a 50-case eval. How to compare LLM eval runs with paired tests, and how many cases you need.
-
Your Semantic Cache Is Answering the Wrong Question
Quora and PAWS sweeps: at 0.90 cosine with 5% repeat traffic, 78% of cache hits served the wrong answer, and thresholds do not transfer between models.
-
Hybrid Search Measured: Where Fusion and Reranking Actually Help
BM25, dense, RRF and a cross-encoder reranker on SciFact, with paired confidence intervals. Fusion helped BM25 a lot and dense a little; the reranker hurt.
-
Structured Outputs Guarantee the Shape, Not the Answer
I built JSON-Schema-constrained decoding from scratch and ran it on a real model. Valid output went from 3% to 100%. Correct output went to 40%. Here's the gap.
-
What Temperature Actually Does, Measured, and Why 0 Was Never Deterministic
Temperature, top-k, top-p and min-p from scratch, run on a real model. One setting picked a token the model rated under 1% four times in five.
-
Your Chunking Benchmark Is Measuring Chunk Size
14 chunking strategies on SQuAD with answer-span ground truth. Semantic chunking lost to every alternative, and fixed-k scoring hid the thing that mattered.
-
What a Token Actually Is, and Why a UUID Costs 24 of Them
I built a BPE tokenizer from scratch, then measured what JSON, UUIDs, code and other languages really cost. The len/4 estimate was off by up to 70%.