Architecture
23 posts — newest first.
-
One VM, one GPU, 100 customers: what it actually takes to sell access to an open-weight model
An open-weight model serving HTTP takes twenty minutes. Making it something 100 customers can share takes a control plane. The whole path, with numbers.
-
Most of your AI platform's traffic doesn't need a frontier model
Small models got good enough in 2026. But the win isn't using them — it's making model choice a platform tier with a router, eval gate, and demotion path.
-
MCP 2026-07-28 shipped: the stateless core is the headline, header routing is the story
The final MCP spec deletes sessions, mirrors requests into HTTP headers, and turns elicitation into multi-round-trip calls. What breaks, and what to audit.
-
MCP is the hands, A2A is the handshake: the two-layer agent protocol stack
A2A hit v1.0 with 150+ orgs behind it. MCP connects agents to tools; A2A connects agents to each other. What the two-layer stack means for platform teams.
-
The reliability gap: a framework for trusting autonomous SRE agents
An autonomous airline agent rebooked 1,247 passengers wrong in one weather event. Trusting agents is a reliability problem — here's how to measure it.
-
The Five Types of Agentic Memory (and When to Use Each)
Agentic memory is five things — working, episodic, semantic, procedural, entity — each with its own storage, eviction, and failure mode. A decision guide.
-
Agentic AI Patterns: The Decision Guide (Part 1 of 3)
Six agentic AI patterns — ReAct, Plan-and-Execute, Critic, fan-out, HITL gate, Supervisor — with a decision flowchart for picking one before you build.
-
The trust gap: bounded autonomy for AI SRE agents
SREs face 50+ alerts a day at 60% false positives while vendors promise autonomous resolution. The autonomy ladder: what an AI agent should never do alone.
-
Context engineering: the window is a budget, not a bucket
The context window is your agent's working memory, not a junk drawer. Four operations — write, select, compress, isolate — and a token budget you allocate.
-
What are vector embeddings?
A primer on vector embeddings — how meaning becomes something you can search, cluster, and compare, and the failure modes you only see in evaluation.
-
What is function calling (tool use)?
A primer on function calling — the JSON-schema contract that lets an LLM invoke your code. The request/response loop, parallel calls, and forced tools.
-
What is prompt caching?
Prompt caching cuts repeated-prompt cost 50–90% and halves latency. How prefix matching works, TTL economics by provider, and what decides your hit rate.
-
The CAP theorem in AI-native distributed systems
CAP didn't get repealed when LLMs showed up. How the C/A/P trade-offs shift when the datastore is a vector index, context graph, or retrieval layer.
-
What is Retrieval-Augmented Generation (RAG)?
A primer on Retrieval-Augmented Generation — grounding an LLM's answer in documents you trust. Indexing, serving, and the failure modes that bite.
-
Mental models for applying AI to infrastructure
Tutorials answer how; mental models answer whether. Seven I use as the front gate before any LLM goes near a production system.
-
Queues and Message Brokers: The Shock Absorber of Distributed Systems
A queue decouples producers from consumers and absorbs bursts. Backpressure, at-least-once delivery, idempotency, DLQs — now in front of every LLM call.
-
Graph Traversal: BFS, DFS, and Why GraphRAG Is Just a Walk
BFS or DFS — queue or stack — decides everything downstream. A refresher on graph traversal, the visited set, and why GraphRAG is just a walk.
-
Floating Point and Numerical Precision: Why 0.1 + 0.2 ≠ 0.3, and Why ML Cares
Floating-point errors aren't random. Why 0.1 + 0.2 ≠ 0.3, and how the same fundamentals drive the FP32 → BF16 → FP8 march behind affordable LLMs.
-
The Memory Hierarchy: Why Data Locality Beats Clock Speed
Each memory level is 10–100× slower than the one above. Cache lines, locality, and why 'keep data near compute' is the biggest lever in LLM inference.
-
Concurrency vs Parallelism: The Distinction That Fixes Your Throughput
Concurrency is dealing with many things at once; parallelism is doing them at once. A refresher on the GIL, async vs threads, and scaling model calls.
-
Caching and Eviction Policies: Why LRU, LFU, and FIFO Aren't the Same Bet
The eviction policy decides whether your cache works. LRU vs LFU vs FIFO, hit rates, invalidation — and how the same bets govern prompt and KV caches.
-
Hash Tables: The Data Structure Behind Almost Everything
The hash table sits under your cache, index, dedup, and vector store metadata. How it turns a key into O(1) access — and what keeps it fast.
-
Big-O Notation in the Age of Billion-Vector Search
Big-O still decides whether your system survives real data. A refresher on complexity, and why it governs vector search, context windows, and outages.