A semantic cache sounds like free money. Embed each incoming prompt, and if it’s close enough to one you’ve already answered, serve the stored answer instead of paying for a model call. Vendors advertise cost cuts of 40% and more.
The question nobody asks is how often “close enough” is the wrong question. I measured it, and the answer depends on a number most caches never see: how often your users actually repeat themselves.
Setup
Two embedding models, run locally: bge-small-en-v1.5 and all-MiniLM-L6-v2. Two labelled datasets of question pairs:
- Quora Question Pairs, 5,000 pairs: real duplicates and real near-misses, as users wrote them. 37% are true duplicates.
- PAWS, 2,000 pairs: adversarial pairs that share almost all their words. “How do I get from Boston to New York?” against “How do I get from New York to Boston?” This is the case that should worry anyone caching on embeddings.
For every cosine threshold from 0.70 to 0.99 I recorded the share of true duplicates served from cache (the hit rate you want) and the share of non-duplicates served from cache (the wrong answers).
The threshold is half the story
Here’s bge-small on Quora at a few thresholds:
| Threshold | True repeats served from cache | Non-repeats served from cache |
|---|---|---|
| 0.80 | 93% | 35.7% |
| 0.85 | 81% | 23.1% |
| 0.90 | 58% | 10.9% |
| 0.95 | 28% | 2.5% |
| 0.98 | 13% | 0.4% |
A 10.9% false-hit rate at 0.90 might sound tolerable. It isn’t, and the reason is the base rate. The share of cache hits that are right depends on how many of your requests were genuine repeats in the first place:
| Threshold | 50% of traffic repeats | 20% repeats | 5% repeats |
|---|---|---|---|
| 0.85 | 78% of hits right | 47% | 16% |
| 0.90 | 84% | 57% | 22% |
| 0.95 | 92% | 74% | 37% |
| 0.98 | 97% | 89% | 64% |
If half your traffic is people asking the same ten things, a 0.90 threshold serves the right answer 84% of the time. If 5% of your traffic is a true repeat, which is closer to what a mixed-purpose assistant sees, 78% of what the cache serves at 0.90 is the answer to a different question. Same model, same threshold, same hit rate on your dashboard.
This is the arithmetic of any screening test with a rare condition, and a semantic cache is a screening test for “have I seen this before.” You can’t choose a threshold without an estimate of your repeat rate, and the only place to get that estimate is a sample of your own traffic, labelled by a human.
A banking-FAQ experiment reported on InfoQ in November 2025 found the same shape from the other side: at a fixed 0.70 threshold, false-positive rates reached 99% for some embedding models, and getting them under 10% took per-model threshold tuning plus extra checks.
The threshold doesn’t transfer between models
The second model, at the same thresholds:
| Threshold | bge-small: repeats / non-repeats served | MiniLM: repeats / non-repeats served |
|---|---|---|
| 0.85 | 81% / 23.1% | 60% / 13.1% |
| 0.90 | 58% / 10.9% | 42% / 6.8% |
| 0.95 | 29% / 2.5% | 22% / 2.0% |
Both models separate duplicates about equally well overall (AUC 0.88 and 0.87). They just put the boundary in different places. “0.90” is not a level of strictness; it’s a point on one model’s curve. Swap the embedding model in a cache and keep the threshold, and you’ve silently changed both your hit rate and your error rate. That’s a model-migration hazard with no error message.
Word order is invisible
On PAWS, both models were close to a coin flip (AUC 0.60 and 0.62). At a 0.90 threshold, bge-small served 98% of the non-paraphrases from cache. The embeddings for “A to B” and “B to A” are nearly identical, because the bag of meanings is identical. If your users ask about direction, sequence, who-did-what-to-whom, or comparisons, an embedding cache will confidently serve the mirror image.
The same blind spot covers numbers and negation. “Fees on orders over $100” and “over $500” embed almost on top of each other. So do “can I cancel” and “can I not cancel.”
Cheap guards help, with a cost
Since the failures are structural, I added structural checks before accepting a hit: refuse if the two questions differ in their numbers, in negation, in proper nouns, or if they contain the same words in a different order. Forty lines of regex, no model.
| Dataset, bge-small at 0.90 | Without guards | With guards |
|---|---|---|
| Quora: repeats / non-repeats served | 58% / 10.9% | 44% / 6.6% |
| PAWS: non-paraphrases served | 98% | 51% |
The guards cut wrong answers roughly in half on both datasets. They also cost a quarter of the true hits on Quora, because real paraphrases sometimes swap a number for a word or drop a name. That’s the trade: every guard is a precision-for-recall exchange, and you should take it, because a cache miss costs one model call while a wrong hit costs a user’s trust.
Bigger caches find more lookalikes
Finally, I took 1,000 fresh questions and checked them against caches of growing size, drawn from other questions in the same pool:
| Cache entries | Fresh questions clearing 0.85 | 0.90 | 0.95 |
|---|---|---|---|
| 1,000 | 0.7% | 0.4% | 0.2% |
| 5,000 | 2.1% | 1.1% | 0.4% |
| 20,000 | 10.1% | 3.4% | 0.9% |
The chance that something in the cache clears the threshold grows close to linearly with the cache. At 20,000 entries, one fresh question in ten found a 0.85 match. Reading the borderline pairs is instructive: “Can I get pregnant the day before ovulation?” matched “Is it possible to get pregnant 2 days before ovulation?” at 0.924. “What was the best smartphone of 2016?” matched “Which is the best smartphone brand of 2016?” at 0.937. Plausible, fluent, and a different question.
A cache that grows forever gets worse at its job. Scope it per tenant or per topic, expire entries, and don’t let a global cache accumulate a year of everything anyone ever asked.
What I’d do
- Measure your repeat rate first. Sample 200 requests, have someone label which ones a stored answer would have served correctly. That number decides whether a semantic cache can work at all.
- Pick the threshold from your own labelled pairs, per model. Start near 0.95 and loosen only with evidence. Re-run the sweep every time the embedding model changes.
- Add the guards. Numbers, negation, names, word order. They’re free and they halve the damage.
- Prefer exact and normalised matching where you can. Lowercase, strip whitespace, canonicalise. An exact cache has no false positives, and a surprising share of real repeats are exact.
- Keep it small and scoped, and watch the hit rate: a hit rate that climbs as the cache ages is a warning, not a win.
A prompt cache is a different thing, and a safer one: it reuses computation on an identical prefix, so there’s no similarity judgement to get wrong. Reach for that first.
Limits
Question-pair datasets stand in for cache traffic; real traffic has longer prompts and more context. Thresholds here are specific to these two small models. The guards are English-only heuristics, and the base-rate precision figures are computed from the measured rates, not from a live deployment. The cache-size pool is Quora questions, which contain real duplicates, so some of the borderline hits are legitimate.
Code, tests and every sweep: ajinb/semantic-cache-eval. bench.py runs on a laptop in a few minutes and downloads both datasets from Hugging Face. Related: what an embedding is.
Comments