performance
10 posts — newest first.
-
Percentiles, Properly: You Cannot Average a p99
Averaging percentiles across shards or minutes produces a number that means nothing. Here is the arithmetic, plus why p99 gets worse as you add services.
-
How CDNs Actually Work: Edge Caching, TTLs, and the Invalidation Problem
A CDN is a globe-spanning cache in front of your origin. The wins are latency and offload; the hard parts are cache keys, TTLs, and knowing when to invalidate.
-
Queueing Theory for SREs: Little's Law and the Utilisation Knee
Latency doesn't rise with load, it rises with 1/(1−utilisation). Two formulas explain the p99 cliff and size every pool in your system.
-
What is prompt caching?
Prompt caching cuts repeated-prompt cost 50–90% and halves latency. How prefix matching works, TTL economics by provider, and what decides your hit rate.
-
Compilers, Interpreters, and JIT: Why Python Is Slow and PyTorch Is Fast
Compiling ahead of time, interpreting, or JIT-compiling hot paths explains why Python is slow and PyTorch is fast — torch.compile is the same idea.
-
The Memory Hierarchy: Why Data Locality Beats Clock Speed
Each memory level is 10–100× slower than the one above. Cache lines, locality, and why 'keep data near compute' is the biggest lever in LLM inference.
-
Concurrency vs Parallelism: The Distinction That Fixes Your Throughput
Concurrency is dealing with many things at once; parallelism is doing them at once. A refresher on the GIL, async vs threads, and scaling model calls.
-
Caching and Eviction Policies: Why LRU, LFU, and FIFO Aren't the Same Bet
The eviction policy decides whether your cache works. LRU vs LFU vs FIFO, hit rates, invalidation — and how the same bets govern prompt and KV caches.
-
Bloom Filters: Saying 'Definitely Not' in a Few Kilobytes
A Bloom filter answers 'have I seen this before?' in kilobytes, never returning a false negative. How it works, and why databases and caches rely on it.
-
Hash Tables: The Data Structure Behind Almost Everything
The hash table sits under your cache, index, dedup, and vector store metadata. How it turns a key into O(1) access — and what keeps it fast.