llm
12 posts — newest first.
-
Your Prompt Didn't Get Better. Your Eval Set Is Just Small.
Two identical prompts look 3 points apart in 32% of runs on a 50-case eval. How to compare LLM eval runs with paired tests, and how many cases you need.
-
Your Semantic Cache Is Answering the Wrong Question
Quora and PAWS sweeps: at 0.90 cosine with 5% repeat traffic, 78% of cache hits served the wrong answer, and thresholds do not transfer between models.
-
Structured Outputs Guarantee the Shape, Not the Answer
I built JSON-Schema-constrained decoding from scratch and ran it on a real model. Valid output went from 3% to 100%. Correct output went to 40%. Here's the gap.
-
What Temperature Actually Does, Measured, and Why 0 Was Never Deterministic
Temperature, top-k, top-p and min-p from scratch, run on a real model. One setting picked a token the model rated under 1% four times in five.
-
What a Token Actually Is, and Why a UUID Costs 24 of Them
I built a BPE tokenizer from scratch, then measured what JSON, UUIDs, code and other languages really cost. The len/4 estimate was off by up to 70%.
-
Most of your AI platform's traffic doesn't need a frontier model
Small models got good enough in 2026. But the win isn't using them — it's making model choice a platform tier with a router, eval gate, and demotion path.
-
Token FinOps: the third budget your agents are spending
Error budgets, context budgets — agents add a third: dollars. Agent tasks burn 5–30× chatbot tokens, and cost-per-token is the wrong metric.
-
Tracing the agent loop: OpenTelemetry's GenAI conventions, read like an SRE
Your agent is a distributed system wearing a chat interface. OpenTelemetry's GenAI conventions make it debuggable — what v1.41 covers and what's moving.
-
Observability for AI systems — what changes when your service calls an LLM
Golden signals miss the failure that pages you: a confident, well-formed, wrong answer. What AI observability adds — context as a span, quality as a signal.
-
What is function calling (tool use)?
A primer on function calling — the JSON-schema contract that lets an LLM invoke your code. The request/response loop, parallel calls, and forced tools.
-
Prompt engineering for SRE: patterns that actually work in production
Prompt advice is written for chatbots; SRE workloads are different. Six patterns I've shipped to production for SRE LLM tools, and why each earned its place.
-
Why AI is the Next SRE Superpower
After 15 years in cloud infrastructure and SRE, why I believe AI is the most significant shift in how we operate systems since Kubernetes.