Blog
All posts — newest first. Or browse by topic.
-
AI SRE is officially a category. Buy it like an SRE, not like a CIO.
Gartner's first Market Guide for AI SRE tooling projects 85% adoption by 2029. Six evaluation questions for cutting through the category hype.
-
The biggest outages still start with a config change — and now agents write them
Fresh postmortems keep confirming it: config changes, not code, cause the largest incidents. Agentic ops multiplies the volume. Here's the defense.
-
Google just published the reference architecture for agentic SRE
Google SRE splits autonomous incident response into reasoning, actuation, and evaluation. The split is the lesson — here's what to copy at normal scale.
-
The EU AI Act blinked — your logging requirements didn't
The EU AI Act omnibus moved high-risk deadlines to 2027–28. The logging, oversight, and inventory work is still yours — and reliability needs it anyway.
-
Your runbooks are lying to your agents: RAG for on-call knowledge
On-call agents are only as good as what they retrieve. Making runbook RAG fresh, exact-match aware, and poison-resistant — with a runnable starter repo.
-
MCP is the hands, A2A is the handshake: the two-layer agent protocol stack
A2A hit v1.0 with 150+ orgs behind it. MCP connects agents to tools; A2A connects agents to each other. What the two-layer stack means for platform teams.
-
Every observability vendor now sells an AI SRE agent. Here's how to evaluate one.
Datadog, Dynatrace, New Relic, AWS — every incumbent now ships an AI SRE agent. A field guide for evaluating one before it touches production.
-
The OWASP Agentic Top 10, translated for platform teams
OWASP's Agentic Top 10 reads like a security doc, but the mitigations are platform controls. Mapping all ten risks to infrastructure you already run.
-
Your agents need identities, not API keys
Every AI agent is a non-human identity — most run on shared, long-lived API keys no IAM review sees. Per-agent identity and your credential blast radius.
-
Token FinOps: the third budget your agents are spending
Error budgets, context budgets — agents add a third: dollars. Agent tasks burn 5–30× chatbot tokens, and cost-per-token is the wrong metric.
-
Tracing the agent loop: OpenTelemetry's GenAI conventions, read like an SRE
Your agent is a distributed system wearing a chat interface. OpenTelemetry's GenAI conventions make it debuggable — what v1.41 covers and what's moving.
-
Frontier models still fail half your incidents: reading ITBench-AA like an SRE
ITBench-AA put frontier models against 59 real Kubernetes incident diagnoses — all scored below 50%. What the benchmark measures and how to use it.
-
Autonomy is a budget, not a toggle: error budgets for AI operators
SRE solved runaway release risk with error budgets. The same mechanism governs AI agents: authority per action class, demoted fast, promoted slow.
-
The reliability gap: a framework for trusting autonomous SRE agents
An autonomous airline agent rebooked 1,247 passengers wrong in one weather event. Trusting agents is a reliability problem — here's how to measure it.
-
Agentic Resource Discovery: I Read the Spec, Then Published a Catalog
Google, Microsoft, and Hugging Face shipped Agentic Resource Discovery. The real ai-catalog.json schema, a working catalog, and the gotchas that break it.
-
The Five Types of Agentic Memory (and When to Use Each)
Agentic memory is five things — working, episodic, semantic, procedural, entity — each with its own storage, eviction, and failure mode. A decision guide.
-
Agentic AI Patterns: The Decision Guide (Part 1 of 3)
Six agentic AI patterns — ReAct, Plan-and-Execute, Critic, fan-out, HITL gate, Supervisor — with a decision flowchart for picking one before you build.
-
Agentic AI Patterns: The Maturity Model (Part 3 of 3)
A five-level agentic AI maturity model, from manual to multi-agent mesh — with a self-assessment and where regulated industries should draw the line.
-
Agentic AI Patterns: Where They Break in Production (Part 2 of 3)
Every agentic AI pattern looks clean in a demo. Where each one breaks in production, the signals you're hitting them, and mitigations that actually work.
-
Making Claude Code Work with Locally Deployed Models
Claude Code can point at Ollama, vLLM, LM Studio, or any private endpoint. When that's the right call, exactly how to configure it, and what you give up.
-
OKF: The Missing Context Layer for AI Agents
The Open Knowledge Format gives agents a structured vocabulary for what data they're touching and where it came from — auditable, not blind, reasoning.
-
Chaos engineering for MCP: break your tool-call plane before production does
LLM calls fail 1–5% of the time and agent tasks fan out into 10–20 tool calls. How to fault-inject your MCP layer with mcp-chaos before production does.
-
The trust gap: bounded autonomy for AI SRE agents
SREs face 50+ alerts a day at 60% false positives while vendors promise autonomous resolution. The autonomy ladder: what an AI agent should never do alone.
-
MCP goes stateless — what the 2026 release candidate means for your SRE tooling
The 2026-07-28 MCP release candidate deletes the session handshake for a stateless HTTP core and hardens OAuth. What changes for your agents, and when.
-
The AI-native SRE stack — a 2026 reference guide
A practitioner's map of the AI-native SRE stack in 2026: six layers from telemetry to bounded remediation, and an honest read on where AI pays off.
-
Context engineering: the window is a budget, not a bucket
The context window is your agent's working memory, not a junk drawer. Four operations — write, select, compress, isolate — and a token budget you allocate.
-
Agent sprawl is your next production incident
Teams shipping AI agents are recreating 2015's microservices sprawl with worse observability. The governance surface that contains it before it pages you.
-
No anonymous inference endpoints — the MCP security principle you're probably violating
The NSA and NIST put MCP on notice: agents are a funnel for prompt injection and privilege abuse. Why 'no anonymous inference endpoints' — and how to comply.
-
Observability for AI systems — what changes when your service calls an LLM
Golden signals miss the failure that pages you: a confident, well-formed, wrong answer. What AI observability adds — context as a span, quality as a signal.
-
Harness engineering: the third phase of AI maturity
Agent = Model + Harness, and in 2026 the harness is the bottleneck. What a production-grade SRE harness contains, with a ~40-line reference implementation.
-
Observability and incident response — the SRE basics
A primer on observability (logs, metrics, traces) and incident response (roles, severities, blameless postmortems) — the disciplines every SRE team runs.
-
Toil and the 50% rule — what it is, how to measure it, and how to kill it
A primer on toil — the manual, repetitive work that eats SRE teams. Google's six-part definition, the 50% cap, and how AI agents change the playbook.
-
SLI, SLO, SLA, and error budgets — the reliability contract explained
SLIs, SLOs, SLAs, and error budgets — the four numbers every SRE team must agree on, the math behind 'nines,' and what changes when agents burn the budget.
-
What is Site Reliability Engineering (SRE)?
A primer on Site Reliability Engineering — where it came from, how it differs from DevOps and Platform Engineering, and what changes as AI joins on-call.
-
What are vector embeddings?
A primer on vector embeddings — how meaning becomes something you can search, cluster, and compare, and the failure modes you only see in evaluation.
-
What is function calling (tool use)?
A primer on function calling — the JSON-schema contract that lets an LLM invoke your code. The request/response loop, parallel calls, and forced tools.
-
What is prompt caching?
Prompt caching cuts repeated-prompt cost 50–90% and halves latency. How prefix matching works, TTL economics by provider, and what decides your hit rate.
-
The CAP theorem in AI-native distributed systems
CAP didn't get repealed when LLMs showed up. How the C/A/P trade-offs shift when the datastore is a vector index, context graph, or retrieval layer.
-
NAS vs SAN for GPU workloads — what changed when AI showed up
File vs block was the old NAS-vs-SAN question. GPU training rewrote it. How the calculus shifts when storage has to keep an H100 cluster fed.
-
What is an AI agent? A primer for cloud engineers
A primer on AI agents — the perceive-reason-act loop, what separates an agent from a one-shot LLM call, and the classical agent types SREs now operate.
-
What is Model Context Protocol (MCP)?
A primer on Model Context Protocol — the open standard that lets AI applications talk to tools through one interface. Hosts, clients, servers, transports.
-
What is Retrieval-Augmented Generation (RAG)?
A primer on Retrieval-Augmented Generation — grounding an LLM's answer in documents you trust. Indexing, serving, and the failure modes that bite.
-
Mental models for applying AI to infrastructure
Tutorials answer how; mental models answer whether. Seven I use as the front gate before any LLM goes near a production system.
-
Prompt engineering for SRE: patterns that actually work in production
Prompt advice is written for chatbots; SRE workloads are different. Six patterns I've shipped to production for SRE LLM tools, and why each earned its place.
-
The MCP gateway pattern: five jobs your agent runtime can't skip
Letting agents call MCP servers directly repeats the no-API-gateway mistake. The five jobs an MCP gateway must do, with reproducible patterns for each.
-
Skills for AI agents that do SRE work
Most agent skills are chatbot prompts in disguise. Three operator-grade SRE skills — opinionated, output-contracted, portable across agent runtimes.
-
Alert fatigue? Let AI triage.
How I built alert-explainer — an open-source service that turns every Prometheus alert into a plain-English brief in 1–4 seconds for under a cent.
-
When NOT to Use AI in Production SRE
Most AI-for-SRE writing tells you where AI helps. Here are seven places it actively hurts — and the operational rule of thumb I use to decide.
-
Building incident-scribe: Slack Thread to Incident Report with Claude
How I built an open-source tool that turns messy Slack incident threads into blameless, structured incident reports in under 30 seconds.
-
Why AI is the Next SRE Superpower
After 15 years in cloud infrastructure and SRE, why I believe AI is the most significant shift in how we operate systems since Kubernetes.
-
Queues and Message Brokers: The Shock Absorber of Distributed Systems
A queue decouples producers from consumers and absorbs bursts. Backpressure, at-least-once delivery, idempotency, DLQs — now in front of every LLM call.
-
ACID, BASE, and Isolation Levels: What 'Consistent' Actually Means
ACID promises correctness; BASE trades it for scale — and your default isolation level is weaker than you think. A refresher on what 'consistent' means.
-
Graph Traversal: BFS, DFS, and Why GraphRAG Is Just a Walk
BFS or DFS — queue or stack — decides everything downstream. A refresher on graph traversal, the visited set, and why GraphRAG is just a walk.
-
Recursion and the Call Stack: Elegant Until It Overflows
Every recursive call costs a stack frame, and runaway depth ends in overflow. Frames, base cases, and why agent loops share the failure mode.
-
Garbage Collection: The Convenience That Shows Up in Your p99
GC frees you from manual memory management — and occasionally freezes your program. Reachability, generational GC, and why GC tuning is a p99 problem.
-
Compilers, Interpreters, and JIT: Why Python Is Slow and PyTorch Is Fast
Compiling ahead of time, interpreting, or JIT-compiling hot paths explains why Python is slow and PyTorch is fast — torch.compile is the same idea.
-
Floating Point and Numerical Precision: Why 0.1 + 0.2 ≠ 0.3, and Why ML Cares
Floating-point errors aren't random. Why 0.1 + 0.2 ≠ 0.3, and how the same fundamentals drive the FP32 → BF16 → FP8 march behind affordable LLMs.
-
Virtual Memory and Paging: The Illusion Every Program Lives In
Virtual memory is the beautiful lie every process lives in. Page faults, thrashing, overcommit — and why paging now manages the KV cache and OOM kills.
-
The Memory Hierarchy: Why Data Locality Beats Clock Speed
Each memory level is 10–100× slower than the one above. Cache lines, locality, and why 'keep data near compute' is the biggest lever in LLM inference.
-
Deadlocks, Locks, and Race Conditions: The Bugs That Only Happen Sometimes
Races and deadlocks only fail sometimes — that's what makes them hard. Locks, atomicity, the four Coffman conditions, and why hung agents are the same bug.
-
Concurrency vs Parallelism: The Distinction That Fixes Your Throughput
Concurrency is dealing with many things at once; parallelism is doing them at once. A refresher on the GIL, async vs threads, and scaling model calls.
-
Caching and Eviction Policies: Why LRU, LFU, and FIFO Aren't the Same Bet
The eviction policy decides whether your cache works. LRU vs LFU vs FIFO, hit rates, invalidation — and how the same bets govern prompt and KV caches.
-
Bloom Filters: Saying 'Definitely Not' in a Few Kilobytes
A Bloom filter answers 'have I seen this before?' in kilobytes, never returning a false negative. How it works, and why databases and caches rely on it.
-
Consistent Hashing: How Distributed Systems Add and Remove Nodes Without Chaos
Hash-mod-N reshuffles almost everything when a node leaves; consistent hashing moves a fraction. The ring, virtual nodes, and why your CDN depends on it.
-
TLS and Public-Key Cryptography, Explained Without the Math
Every https and model API call rides on TLS. How public-key crypto solves key distribution, what a certificate proves, and the cert-expiry realities.
-
How DNS Resolution Works — the Internet's Phone Book, and Its Cache
DNS caching decides how fast failovers propagate and how far a misconfig spreads. The recursive walk from root to authoritative, and why TTLs govern deploys.
-
How TCP/IP Actually Works — and Why the Handshake Still Bites You
Every API call and model request rides on TCP/IP. The three-way handshake, flow control, and why round-trips and TIME_WAIT shape your latency.
-
B-tree vs LSM-tree: The Two Ways Databases Store Your Data
Almost every database is a read-optimized B-tree or a write-optimized LSM-tree underneath. Knowing which explains its performance and failure modes.
-
Hash Tables: The Data Structure Behind Almost Everything
The hash table sits under your cache, index, dedup, and vector store metadata. How it turns a key into O(1) access — and what keeps it fast.
-
Big-O Notation in the Age of Billion-Vector Search
Big-O still decides whether your system survives real data. A refresher on complexity, and why it governs vector search, context windows, and outages.
-
REST vs. GraphQL APIs – how to choose?
REST or GraphQL? How the two API styles differ in data fetching, versioning, caching, and tooling — and how to choose for your service.
-
GraphQL for API Development
GraphQL is a query language for APIs, created at Facebook in 2012. What it is, how it differs from REST, and where it fits in API development.
-
What is Address Resolution Protocol (ARP)?
ARP discovers the MAC address behind an IP address. How the protocol works, the request/reply flow, and where it sits in the networking stack.
-
The CAP theorem
Consistency, availability, partition tolerance — you get two. A walk through the CAP trade-offs, database by database.
-
The 7 layers of ISO OSI model
The OSI model's seven layers give diverse systems a standard way to communicate. What each layer does and why the model still matters.
-
NAS vs SAN – A brief comparison.
NAS serves files over ethernet; SAN serves blocks over a dedicated network. A brief comparison of the two storage architectures and when to pick each.
-
What is RAID?
RAID provides redundancy for stored data, protecting against disk failures. The common RAID levels, how they differ, and when to use each.
-
Amazon Leadership Principles – my thoughts.
What Amazon's Leadership Principles actually mean in hiring and evaluations — and how to prepare for the behavioral rounds of an Amazon interview.
-
Gettings started with Ansible.
Ansible automates configuration and orchestration of infrastructure as code. A getting-started guide: inventory, playbooks, and your first runs.
-
Introduction to YAML!
YAML Ain't Markup Language — a human-readable format for configuration. The syntax, the gotchas, and why it's everywhere in infrastructure.
-
Preparing for AWS Solution Architect (Professional) Certification.
How I prepared for the AWS Solutions Architect Professional exam — study plan, resources, and what I'd do differently after passing.