Claude Opus 5.5’s migration guide has a line I didn’t expect: omit temperature, top_p and top_k, and use prompting to guide behaviour instead. It adds that if you were using temperature = 0 for determinism, it never guaranteed identical outputs on earlier models either.
So the most-tuned knob in LLM engineering is disappearing from at least one frontier API, and the main reason people set it to 0 was never true. That makes this a good moment to understand what these parameters actually did, so you know what you’re losing and what you weren’t getting anyway.
I implemented the four standard knobs from scratch in about 70 lines of numpy and ran them on a real model: SmolLM2-135M-Instruct, 20 prompts, 16 samples each, 64 tokens per sample.
The four knobs in one function
At every step a model produces a score (a logit) for every token in its vocabulary. Sampling turns those scores into one choice:
def filtered(logits, temperature=1.0, top_k=None, top_p=None, min_p=None):
probs = softmax(logits / temperature) # 1. reshape
mask = np.ones(probs.size, dtype=bool)
if top_k is not None:
mask &= top_k_mask(probs, top_k) # 2. keep the k likeliest
if top_p is not None:
mask &= top_p_mask(probs, top_p) # 3. keep enough to cover p
if min_p is not None:
mask &= probs >= min_p * probs.max() # 4. keep anything close to the best
probs = np.where(mask, probs, 0.0)
return probs / probs.sum()- Temperature reshapes the whole distribution. Dividing by 2 is the same as taking the square root of every probability and renormalizing, so unlikely tokens gain ground.
- Top-k keeps a fixed number of candidates however confident the model is.
- Top-p, or nucleus sampling from Holtzman et al., keeps the smallest set of top tokens whose probabilities add up to p.
- Min-p, from Nguyen et al. (an ICLR 2025 oral), keeps every token that’s at least
min_ptimes as likely as the best one.
Order matters. My code applies temperature first and then truncates, as Hugging Face transformers does. Not every server uses that order, so the same numbers can mean different things on a different stack.
What each setting actually samples
The measurement I care about is the tail-token rate: how often a setting picks a token that the model itself, before any temperature, rated under 1%. Those are the tokens where the output stops being what the model believed.
| Setting | Tail-token rate | Distinct openings | Repeated trigrams |
|---|---|---|---|
| T=0.7 | 1.8% | 95.6% | 2.6% |
| T=1.0 | 11.4% | 99.1% | 0.8% |
| T=1.0, top-k 40 | 6.4% | 99.4% | 1.1% |
| T=1.0, top-p 0.9 | 4.4% | 97.8% | 0.9% |
| T=1.0, min-p 0.1 | 0.1% | 97.2% | 2.4% |
| T=1.5 | 79.8% | 100% | 0.0% |
| T=1.5, top-p 0.9 | 67.3% | 100% | 0.1% |
| T=1.5, min-p 0.1 | 2.4% | 99.1% | 0.8% |
“Distinct openings” is the share of the 16 samples per prompt whose first 16 tokens were unique. “Repeated trigrams” measures loops within a sample.
Three things stand out.
Plain T=1.0 isn’t “the model’s natural distribution” in any useful sense. It picked a token the model rated under 1% about once every nine tokens. Over a 500-token answer, that’s dozens of small departures from what the model actually thought.
Top-p falls apart at high temperature. At T=1.5 the distribution is flat enough that the tokens needed to cover 90% of the probability still include a long tail, so top-p barely helps: 67% tail tokens instead of 80%. This is the failure the min-p paper set out to fix.
Min-p does what it claims, at least here. At T=1.5 it held the tail rate to 2.4%, close to T=0.7’s 1.8%, while keeping more distinct openings (99.1% against 95.6%). That’s the paper’s headline in miniature: you can raise the temperature for diversity without letting in the garbage.
This is a 135-million-parameter model, so treat the absolute numbers as illustrative. A frontier model’s distributions are sharper, and its tail tokens aren’t automatically wrong. The mechanics are the same, though, and that’s what changes how you set these knobs.
Temperature 0 and the determinism that wasn’t
Temperature 0 isn’t really a temperature, since you can’t divide by zero. Most implementations treat it as greedy decoding: take the top token, no randomness. So why would it ever vary?
I tested it locally first. Twenty prompts, 128 greedy tokens each, on CPU and on the Apple GPU:
| Comparison | Identical outputs |
|---|---|
| Same prompt, run twice | 20 / 20 |
| Alone vs padded into a batch of 8 | 20 / 20 |
| CPU vs GPU | 20 / 20 |
Perfectly repeatable. So I measured how close it came to not being. At each of 2,316 greedy steps I recorded the log-probability gap between the top two tokens, and the size of the numeric noise between devices:
| Measurement | Value |
|---|---|
| Median gap between the top two tokens | 1.79 |
| Smallest gap seen | 0.00041 |
| Steps with a gap under 0.01 | 5 of 2,316 |
| Largest CPU vs GPU difference in that gap | 0.000036 |
| Difference between running alone and in a batch, on CPU | exactly 0 |
The noise exists, but the closest call I saw was about 11 times bigger than the largest noise. A back-of-envelope extrapolation from those five near-ties, assuming they thin out evenly toward zero, puts a flip at roughly one per 100,000 tokens on this setup.
Now look at a hosted API. Thinking Machines sampled 1,000 completions at temperature 0 from Qwen3-235B and got 80 different outputs. All of them matched for the first 102 tokens. At token 103, 992 said “Queens, New York” and 8 said “New York City.”
Their explanation is the useful part. A hosted model is deterministic for any given batch, but the inference kernels give slightly different numbers at different batch sizes, and the batch size depends on how many other people are using the server at that moment. Your request’s arithmetic depends on strangers’ traffic. With much larger models, longer outputs, lower-precision arithmetic and millions of requests, the near-ties that were rare on my laptop happen all the time. Once one token flips, everything after it follows a different path.
They also showed it can be fixed with batch-invariant kernels, which gave 1,000 identical completions, at a real performance cost. The hosted APIs I checked don’t promise it: Anthropic’s guide says temperature 0 never guaranteed identical outputs, and OpenAI’s own cookbook says determinism with its seed parameter is not guaranteed.
What I’d do with this
If you need the same answer twice, store the first one. Don’t regenerate and hope. That goes for audit trails, caches and anything a human already approved.
If you need to know how variable an answer is, sample it k times on purpose. That’s what pass^k in evals is for.
Use temperature 0, where it’s still accepted, for extraction and classification, because you want the most likely answer. Don’t use it as a reproducibility guarantee.
On self-hosted models, try min-p before top-p when you want diversity. Servers such as vLLM and llama.cpp expose it. Check which order your server applies the filters in.
Where the knob is gone, ask for variety in the prompt. “Give three different options” does on purpose what a high temperature did by accident, and the model can make the options actually differ.
Run it
ajinb/llm-sampling-lab has the samplers with hand-computed tests (numpy only), and the experiment script that produced every number here. The experiment needs torch and transformers, downloads the 270 MB model on first run, and runs on a laptop.
Comments