AI
58 posts — newest first.
-
An agent breached Hugging Face in 4.5 days. The controls that would have stopped it are boring.
Hugging Face's July 2026 postmortem is the clearest agent-intrusion writeup we have. The failures were IMDS, admission policy, and one shared credential.
-
Your cluster has GPUs. That doesn't make it AI-ready.
CNCF's Kubernetes AI Conformance program is a portability standard. Read it as a self-audit instead and it becomes a genuinely useful platform checklist.
-
Compaction is a reliability event, and you are not measuring it
New research on long-horizon agents finds context compression degrades execution, not knowledge. Treat every compaction as a state transition and instrument it.
-
Model end-of-life is a certificate expiry problem, and you already know how to solve it
Models retire on a schedule, the same model retires on different dates per platform, and 60 days' notice is a floor. Treat model EOL as inventory, not as news.
-
AI made your team faster. Your change failure rate noticed.
DORA's 2026 ROI research names the verification tax and the J-curve. For platform teams the lesson is blunt: codegen moved the bottleneck, it didn't remove it.
-
One VM, one GPU, 100 customers: what it actually takes to sell access to an open-weight model
An open-weight model serving HTTP takes twenty minutes. Making it something 100 customers can share takes a control plane. The whole path, with numbers.
-
Agent workflows belong in code, not just visual builders
OpenAI is winding down Agent Builder and platform Evals. Production agent workflows need code, tests, traces, and reviewable contracts.
-
Your agent installs Markdown from the internet and runs it. We spent 20 years learning not to do that.
Agent skills are dependencies with none of the controls. What the 2026 research found, why scanners miss it, and seven fixes for platform teams.
-
Round-robin is malpractice for LLM traffic: what the Inference Gateway actually fixes
The Gateway API Inference Extension is GA. Why a normal Kubernetes Service balances model servers badly, and how to tell if you need an InferencePool.
-
GPU utilization is a lying metric: the unit economics of self-hosted inference
You can run a GPU at 100% utilization and waste most of it. The cost model that predicts your inference bill, and the five levers in payback order.
-
Agents are deleting production and nobody is writing the postmortem
Your incident template assumes a deterministic system and a human decision-maker. Agent incidents have neither. Five sections that fix the template.
-
Most of your AI platform's traffic doesn't need a frontier model
Small models got good enough in 2026. But the win isn't using them — it's making model choice a platform tier with a router, eval gate, and demotion path.
-
MCP 2026-07-28 shipped: the stateless core is the headline, header routing is the story
The final MCP spec deletes sessions, mirrors requests into HTTP headers, and turns elicitation into multi-round-trip calls. What breaks, and what to audit.
-
AI SRE is officially a category. Buy it like an SRE, not like a CIO.
Gartner's first Market Guide for AI SRE tooling projects 85% adoption by 2029. Six evaluation questions for cutting through the category hype.
-
Google just published the reference architecture for agentic SRE
Google SRE splits autonomous incident response into reasoning, actuation, and evaluation. The split is the lesson — here's what to copy at normal scale.
-
The EU AI Act blinked — your logging requirements didn't
The EU AI Act omnibus moved high-risk deadlines to 2027–28. The logging, oversight, and inventory work is still yours — and reliability needs it anyway.
-
Your runbooks are lying to your agents: RAG for on-call knowledge
On-call agents are only as good as what they retrieve. Making runbook RAG fresh, exact-match aware, and poison-resistant — with a runnable starter repo.
-
MCP is the hands, A2A is the handshake: the two-layer agent protocol stack
A2A hit v1.0 with 150+ orgs behind it. MCP connects agents to tools; A2A connects agents to each other. What the two-layer stack means for platform teams.
-
Every observability vendor now sells an AI SRE agent. Here's how to evaluate one.
Datadog, Dynatrace, New Relic, AWS — every incumbent now ships an AI SRE agent. A field guide for evaluating one before it touches production.
-
The OWASP Agentic Top 10, translated for platform teams
OWASP's Agentic Top 10 reads like a security doc, but the mitigations are platform controls. Mapping all ten risks to infrastructure you already run.
-
Your agents need identities, not API keys
Every AI agent is a non-human identity — most run on shared, long-lived API keys no IAM review sees. Per-agent identity and your credential blast radius.
-
Token FinOps: the third budget your agents are spending
Error budgets, context budgets — agents add a third: dollars. Agent tasks burn 5–30× chatbot tokens, and cost-per-token is the wrong metric.
-
Tracing the agent loop: OpenTelemetry's GenAI conventions, read like an SRE
Your agent is a distributed system wearing a chat interface. OpenTelemetry's GenAI conventions make it debuggable — what v1.41 covers and what's moving.
-
Frontier models still fail half your incidents: reading ITBench-AA like an SRE
ITBench-AA put frontier models against 59 real Kubernetes incident diagnoses — all scored below 50%. What the benchmark measures and how to use it.
-
Autonomy is a budget, not a toggle: error budgets for AI operators
SRE solved runaway release risk with error budgets. The same mechanism governs AI agents: authority per action class, demoted fast, promoted slow.
-
The reliability gap: a framework for trusting autonomous SRE agents
An autonomous airline agent rebooked 1,247 passengers wrong in one weather event. Trusting agents is a reliability problem — here's how to measure it.
-
Agentic Resource Discovery: I Read the Spec, Then Published a Catalog
Google, Microsoft, and Hugging Face shipped Agentic Resource Discovery. The real ai-catalog.json schema, a working catalog, and the gotchas that break it.
-
The Five Types of Agentic Memory (and When to Use Each)
Agentic memory is five things — working, episodic, semantic, procedural, entity — each with its own storage, eviction, and failure mode. A decision guide.
-
Agentic AI Patterns: The Decision Guide (Part 1 of 3)
Six agentic AI patterns — ReAct, Plan-and-Execute, Critic, fan-out, HITL gate, Supervisor — with a decision flowchart for picking one before you build.
-
Agentic AI Patterns: The Maturity Model (Part 3 of 3)
A five-level agentic AI maturity model, from manual to multi-agent mesh — with a self-assessment and where regulated industries should draw the line.
-
Agentic AI Patterns: Where They Break in Production (Part 2 of 3)
Every agentic AI pattern looks clean in a demo. Where each one breaks in production, the signals you're hitting them, and mitigations that actually work.
-
Making Claude Code Work with Locally Deployed Models
Claude Code can point at Ollama, vLLM, LM Studio, or any private endpoint. When that's the right call, exactly how to configure it, and what you give up.
-
OKF: The Missing Context Layer for AI Agents
The Open Knowledge Format gives agents a structured vocabulary for what data they're touching and where it came from — auditable, not blind, reasoning.
-
Chaos engineering for MCP: break your tool-call plane before production does
LLM calls fail 1–5% of the time and agent tasks fan out into 10–20 tool calls. How to fault-inject your MCP layer with mcp-chaos before production does.
-
The trust gap: bounded autonomy for AI SRE agents
SREs face 50+ alerts a day at 60% false positives while vendors promise autonomous resolution. The autonomy ladder: what an AI agent should never do alone.
-
MCP goes stateless — what the 2026 release candidate means for your SRE tooling
The 2026-07-28 MCP release candidate deletes the session handshake for a stateless HTTP core and hardens OAuth. What changes for your agents, and when.
-
The AI-native SRE stack — a 2026 reference guide
A practitioner's map of the AI-native SRE stack in 2026: six layers from telemetry to bounded remediation, and an honest read on where AI pays off.
-
Context engineering: the window is a budget, not a bucket
The context window is your agent's working memory, not a junk drawer. Four operations — write, select, compress, isolate — and a token budget you allocate.
-
Agent sprawl is your next production incident
Teams shipping AI agents are recreating 2015's microservices sprawl with worse observability. The governance surface that contains it before it pages you.
-
No anonymous inference endpoints — the MCP security principle you're probably violating
The NSA and NIST put MCP on notice: agents are a funnel for prompt injection and privilege abuse. Why 'no anonymous inference endpoints' — and how to comply.
-
Observability for AI systems — what changes when your service calls an LLM
Golden signals miss the failure that pages you: a confident, well-formed, wrong answer. What AI observability adds — context as a span, quality as a signal.
-
Harness engineering: the third phase of AI maturity
Agent = Model + Harness, and in 2026 the harness is the bottleneck. What a production-grade SRE harness contains, with a ~40-line reference implementation.
-
What are vector embeddings?
A primer on vector embeddings — how meaning becomes something you can search, cluster, and compare, and the failure modes you only see in evaluation.
-
What is function calling (tool use)?
A primer on function calling — the JSON-schema contract that lets an LLM invoke your code. The request/response loop, parallel calls, and forced tools.
-
What is prompt caching?
Prompt caching cuts repeated-prompt cost 50–90% and halves latency. How prefix matching works, TTL economics by provider, and what decides your hit rate.
-
The CAP theorem in AI-native distributed systems
CAP didn't get repealed when LLMs showed up. How the C/A/P trade-offs shift when the datastore is a vector index, context graph, or retrieval layer.
-
NAS vs SAN for GPU workloads — what changed when AI showed up
File vs block was the old NAS-vs-SAN question. GPU training rewrote it. How the calculus shifts when storage has to keep an H100 cluster fed.
-
What is an AI agent? A primer for cloud engineers
A primer on AI agents — the perceive-reason-act loop, what separates an agent from a one-shot LLM call, and the classical agent types SREs now operate.
-
What is Model Context Protocol (MCP)?
A primer on Model Context Protocol — the open standard that lets AI applications talk to tools through one interface. Hosts, clients, servers, transports.
-
What is Retrieval-Augmented Generation (RAG)?
A primer on Retrieval-Augmented Generation — grounding an LLM's answer in documents you trust. Indexing, serving, and the failure modes that bite.
-
Mental models for applying AI to infrastructure
Tutorials answer how; mental models answer whether. Seven I use as the front gate before any LLM goes near a production system.
-
Prompt engineering for SRE: patterns that actually work in production
Prompt advice is written for chatbots; SRE workloads are different. Six patterns I've shipped to production for SRE LLM tools, and why each earned its place.
-
The MCP gateway pattern: five jobs your agent runtime can't skip
Letting agents call MCP servers directly repeats the no-API-gateway mistake. The five jobs an MCP gateway must do, with reproducible patterns for each.
-
Skills for AI agents that do SRE work
Most agent skills are chatbot prompts in disguise. Three operator-grade SRE skills — opinionated, output-contracted, portable across agent runtimes.
-
Alert fatigue? Let AI triage.
How I built alert-explainer — an open-source service that turns every Prometheus alert into a plain-English brief in 1–4 seconds for under a cent.
-
When NOT to Use AI in Production SRE
Most AI-for-SRE writing tells you where AI helps. Here are seven places it actively hurts — and the operational rule of thumb I use to decide.
-
Building incident-scribe: Slack Thread to Incident Report with Claude
How I built an open-source tool that turns messy Slack incident threads into blameless, structured incident reports in under 30 seconds.
-
Why AI is the Next SRE Superpower
After 15 years in cloud infrastructure and SRE, why I believe AI is the most significant shift in how we operate systems since Kubernetes.