performance
7 posts — newest first.
-
What is prompt caching?
Prompt caching cuts repeated-prompt cost 50–90% and halves latency. How prefix matching works, TTL economics by provider, and what decides your hit rate.
-
Compilers, Interpreters, and JIT: Why Python Is Slow and PyTorch Is Fast
Compiling ahead of time, interpreting, or JIT-compiling hot paths explains why Python is slow and PyTorch is fast — torch.compile is the same idea.
-
The Memory Hierarchy: Why Data Locality Beats Clock Speed
Each memory level is 10–100× slower than the one above. Cache lines, locality, and why 'keep data near compute' is the biggest lever in LLM inference.
-
Concurrency vs Parallelism: The Distinction That Fixes Your Throughput
Concurrency is dealing with many things at once; parallelism is doing them at once. A refresher on the GIL, async vs threads, and scaling model calls.
-
Caching and Eviction Policies: Why LRU, LFU, and FIFO Aren't the Same Bet
The eviction policy decides whether your cache works. LRU vs LFU vs FIFO, hit rates, invalidation — and how the same bets govern prompt and KV caches.
-
Bloom Filters: Saying 'Definitely Not' in a Few Kilobytes
A Bloom filter answers 'have I seen this before?' in kilobytes, never returning a false negative. How it works, and why databases and caches rely on it.
-
Hash Tables: The Data Structure Behind Almost Everything
The hash table sits under your cache, index, dedup, and vector store metadata. How it turns a key into O(1) access — and what keeps it fast.