Platform Engineering
72 posts — newest first.
-
Your Agent's Safety Argument Has an Expiry Date. Write Down the Defeaters.
An assurance case says why an agent may act unattended. Defeaters say when that stops holding. Wire them to live state and authority drops on its own.
-
Claude Code Hooks: Guardrails That Don't Depend on the Model Remembering
CLAUDE.md is advice the model can reinterpret. Hooks run every time. Four working hooks, the exit-code gotcha that fails open, and where hooks stop.
-
LiteLLM 1.107.0: The Key Was Deleted in May. Nobody Revoked It.
The malicious release was live for two hours. Getting there took five months, an agent platform holding an owner's token, and a relaxed release gate.
-
Your Pod Uses Half Its CPU Limit and Still Gets Throttled
CPU limits are enforced per 100 ms, not per minute. A simulation of CFS throttling at 54% average use, the metric that shows it, and how to size limits.
-
gRPC vs REST: Smaller Bytes Are the Least Interesting Difference
Protobuf is smaller than JSON until you gzip it. The real gRPC trade-offs are contracts, deadlines, status codes in trailers, and load balancing on Kubernetes.
-
The Agent Didn't Turn Off the Monitor. It Turned Off the Pager.
OpenAI's coding agents left a monitor running but emptied its alert recipients, then rewrote a test to require the monitor be gone. What to change.
-
Liveness, Readiness, Startup: The Probe That Restarts Your Healthy Pods
Each Kubernetes probe answers a different question. Put the database in the wrong one and a 90-second blip becomes twelve restarts. A simulation and the fix.
-
Webhooks: Verify, Store, Say 200, Then Think
A webhook is a message queue whose producer belongs to someone else. The receiver pattern that survives retries, duplicates, replays and out-of-order events.
-
MCP Server Cards Are Final. The Best Decision Was What They Left Out.
SEP-2127 merged on 6 October: a card says who an MCP server is and where to connect, but never what tools it has. Why that's right, and where it's too relaxed.
-
The Mitigation Was the Outage: Reading Google's us-central1 Storage Report Like an SRE
Google's preliminary report on the 8 Oct Cloud Storage incident: an optimisation exhausted the nodes it meant to relieve, and emergency capacity wasn't there.
-
Every HTTP Tool You Give an Agent Is an SSRF Primitive
Zenity's AgentCorruption went from one prompt to every AgentCore agent in a region. Every link was a default. Here's the control that breaks each one.
-
Nine Agentic Exploits, One Sentence Worth Memorising
OWASP's Q3 2026 roundup reads as a containment failure report: agents escaped lab boundaries and reached real third parties. The prompt was the boundary.
-
You've Been Governing the Agents You Run. Here Come the Ones You Don't.
Sierra and Meta propose a front door for agents acting on a customer's behalf. The hard questions are the ones the spec has not answered yet.
-
Agent Sandboxes Start 6× Faster Now. Isolation Just Got Cheaper Than the Workaround.
Cloudflare cut median time-to-interactive from 4.0s to 648ms. The number matters because cold start is why teams share sandboxes they shouldn't share.
-
NVIDIA Moved the Agent Watchdog to a Different Chip
The Open Agent Safety Platform puts enforcement on a DPU, outside the host the agent runs on. The instinct is decades old. The shape is the question.
-
Secrets Management: Scope, Lifetime, and the Blast Radius of a Leak
Storing secrets safely is the easy half. The damage is decided by what one credential can do, for how long, and whether you can revoke it safely.
-
Load Balancing and Health Checks: The Three States Your Checks Don't Have
Balancing decides where load goes; health checks decide where it stops. Most outages here come from a check that can't tell busy apart from broken.
-
Idempotency Keys: The Thing That Makes a Retry Safe
A timeout never tells you whether the work happened. Idempotency keys resolve that — if you get the concurrency and scope right.
-
Rate Limiting Algorithms: Token Bucket, Leaky Bucket, and Sliding Window, Compared
Five algorithms, one trade-off: burst tolerance vs memory vs precision. Plus why LLM APIs limit tokens instead of requests.
-
Agent browsing got 3–7× cheaper and 1.8× slower. Both numbers change your design.
Cloudflare's Kitesurf is a browser built for agents, not people. The measured trade-off, what it can't do, and why cheap browsing rewrites your threat model.
-
A reasoning model just became a platform default. The lock-in isn't where you're looking.
Salesforce made Claude the default model across Agentforce and Slack. Where model dependency actually accrues, and the five exits worth keeping open.
-
Your newest platform user is an agent. Build it a golden path.
Agents are becoming first-class platform citizens. That is an IDP problem: identity, quotas, golden paths, and a portal agents consume as an API.
-
The Model Hardware Standard is the first agent spec where rollback isn't an option
Anthropic's MHS gives agents read/write control of lab and factory hardware. What it guards, what it doesn't, and the controls you have to build yourself.
-
MCP has a standards body now. That changes your procurement, not your code.
MCP now lives under the Agentic AI Foundation at the Linux Foundation. What vendor-neutral governance actually buys a platform team.
-
Identity was the easy half. Agent authorization is becoming an audit-trail problem.
NIST, a Senate bill and the MCP roadmap converge on one requirement: prove which agent did what, for whom, under whose grant. Your traces do not.
-
Your top AI failure mode is saturation, and the SLI is in a header you throw away
Datadog found ~5% of production AI requests fail, nearly 60% on capacity limits. Provider saturation is an SRE problem you already have the primitives for.
-
MCP's next release deletes your polling loop. Start building for it now.
The MCP roadmap published 22 August 2026 names five priorities. Three change how you operate agent infrastructure — and you can prepare for all three today.
-
Queueing Theory for SREs: Little's Law and the Utilisation Knee
Latency doesn't rise with load, it rises with 1/(1−utilisation). Two formulas explain the p99 cliff and size every pool in your system.
-
Cascading Failure: Backpressure, Load Shedding, and Graceful Degradation
A cascading failure is a feedback loop, not a row of dominoes. Removing the trigger does not stop it — which is why the controls have to be in place beforehand.
-
An agent breached Hugging Face in 4.5 days. The controls that would have stopped it are boring.
Hugging Face's July 2026 postmortem is the clearest agent-intrusion writeup we have. The failures were IMDS, admission policy, and one shared credential.
-
Your cluster has GPUs. That doesn't make it AI-ready.
CNCF's Kubernetes AI Conformance program is a portability standard. Read it as a self-audit instead and it becomes a genuinely useful platform checklist.
-
Timeouts and Deadline Propagation: The Budget Nobody Sets
A timeout is a local guess; a deadline is a shared fact. The difference decides whether your system sheds doomed work or grinds on abandoned requests.
-
Compaction is a reliability event, and you are not measuring it
New research on long-horizon agents finds context compression degrades execution, not knowledge. Treat every compaction as a state transition and instrument it.
-
Progressive Delivery: Canary, Blue-Green, and the Rollback You Never Tested
Every deployment strategy answers one question: who pays for the bug? Rolling, blue-green, and canary answer differently — only one limits blast radius.
-
Model end-of-life is a certificate expiry problem, and you already know how to solve it
Models retire on a schedule, the same model retires on different dates per platform, and 60 days' notice is a floor. Treat model EOL as inventory, not as news.
-
The Deployment Pipeline: Build Once, Promote Everywhere
If your artifact is rebuilt per environment, staging never tested what production runs. Build once, promote everywhere is the rule the rest of CD rests on.
-
AI made your team faster. Your change failure rate noticed.
DORA's 2026 ROI research names the verification tax and the J-curve. For platform teams the lesson is blunt: codegen moved the bottleneck, it didn't remove it.
-
One VM, one GPU, 100 customers: what it actually takes to sell access to an open-weight model
An open-weight model serving HTTP takes twenty minutes. Making it something 100 customers can share takes a control plane. The whole path, with numbers.
-
Infrastructure as Code: State, Drift, and Why the Plan Is the Product
IaC is a three-way comparison: config, state, reality. Every confusing Terraform outcome is two of them disagreeing — and the plan tells you which.
-
The Reconciliation Loop: Why Kubernetes Self-Heals and Your Script Doesn't
Level-triggered control loops are the one idea behind Kubernetes, Terraform, GitOps, and every operator. See it once and half of platform work collapses.
-
What a Container Actually Is: Namespaces, cgroups, and Layers
A container is not a lightweight VM. It is one ordinary process with three kernel restrictions applied. Knowing which three explains almost every container bug.
-
Agent workflows belong in code, not just visual builders
OpenAI is winding down Agent Builder and platform Evals. Production agent workflows need code, tests, traces, and reviewable contracts.
-
Your agent installs Markdown from the internet and runs it. We spent 20 years learning not to do that.
Agent skills are dependencies with none of the controls. What the 2026 research found, why scanners miss it, and seven fixes for platform teams.
-
Round-robin is malpractice for LLM traffic: what the Inference Gateway actually fixes
The Gateway API Inference Extension is GA. Why a normal Kubernetes Service balances model servers badly, and how to tell if you need an InferencePool.
-
GPU utilization is a lying metric: the unit economics of self-hosted inference
You can run a GPU at 100% utilization and waste most of it. The cost model that predicts your inference bill, and the five levers in payback order.
-
Most of your AI platform's traffic doesn't need a frontier model
Small models got good enough in 2026. But the win isn't using them — it's making model choice a platform tier with a router, eval gate, and demotion path.
-
MCP 2026-07-28 shipped: the stateless core is the headline, header routing is the story
The final MCP spec deletes sessions, mirrors requests into HTTP headers, and turns elicitation into multi-round-trip calls. What breaks, and what to audit.
-
AI SRE is officially a category. Buy it like an SRE, not like a CIO.
Gartner's first Market Guide for AI SRE tooling projects 85% adoption by 2029. Six evaluation questions for cutting through the category hype.
-
The biggest outages still start with a config change — and now agents write them
Fresh postmortems keep confirming it: config changes, not code, cause the largest incidents. Agentic ops multiplies the volume. Here's the defense.
-
The EU AI Act blinked — your logging requirements didn't
The EU AI Act omnibus moved high-risk deadlines to 2027–28. The logging, oversight, and inventory work is still yours — and reliability needs it anyway.
-
MCP is the hands, A2A is the handshake: the two-layer agent protocol stack
A2A hit v1.0 with 150+ orgs behind it. MCP connects agents to tools; A2A connects agents to each other. What the two-layer stack means for platform teams.
-
The OWASP Agentic Top 10, translated for platform teams
OWASP's Agentic Top 10 reads like a security doc, but the mitigations are platform controls. Mapping all ten risks to infrastructure you already run.
-
Agentic Resource Discovery: I Read the Spec, Then Published a Catalog
Google, Microsoft, and Hugging Face shipped Agentic Resource Discovery. The real ai-catalog.json schema, a working catalog, and the gotchas that break it.
-
The Five Types of Agentic Memory (and When to Use Each)
Agentic memory is five things — working, episodic, semantic, procedural, entity — each with its own storage, eviction, and failure mode. A decision guide.
-
Agentic AI Patterns: The Decision Guide (Part 1 of 3)
Six agentic AI patterns — ReAct, Plan-and-Execute, Critic, fan-out, HITL gate, Supervisor — with a decision flowchart for picking one before you build.
-
Agentic AI Patterns: The Maturity Model (Part 3 of 3)
A five-level agentic AI maturity model, from manual to multi-agent mesh — with a self-assessment and where regulated industries should draw the line.
-
Agentic AI Patterns: Where They Break in Production (Part 2 of 3)
Every agentic AI pattern looks clean in a demo. Where each one breaks in production, the signals you're hitting them, and mitigations that actually work.
-
Making Claude Code Work with Locally Deployed Models
Claude Code can point at Ollama, vLLM, LM Studio, or any private endpoint. When that's the right call, exactly how to configure it, and what you give up.
-
OKF: The Missing Context Layer for AI Agents
The Open Knowledge Format gives agents a structured vocabulary for what data they're touching and where it came from — auditable, not blind, reasoning.
-
MCP goes stateless — what the 2026 release candidate means for your SRE tooling
The 2026-07-28 MCP release candidate deletes the session handshake for a stateless HTTP core and hardens OAuth. What changes for your agents, and when.
-
Context engineering: the window is a budget, not a bucket
The context window is your agent's working memory, not a junk drawer. Four operations — write, select, compress, isolate — and a token budget you allocate.
-
Agent sprawl is your next production incident
Teams shipping AI agents are recreating 2015's microservices sprawl with worse observability. The governance surface that contains it before it pages you.
-
No anonymous inference endpoints — the MCP security principle you're probably violating
The NSA and NIST put MCP on notice: agents are a funnel for prompt injection and privilege abuse. Why 'no anonymous inference endpoints' — and how to comply.
-
What is Site Reliability Engineering (SRE)?
A primer on Site Reliability Engineering — where it came from, how it differs from DevOps and Platform Engineering, and what changes as AI joins on-call.
-
What are vector embeddings?
A primer on vector embeddings — how meaning becomes something you can search, cluster, and compare, and the failure modes you only see in evaluation.
-
What is function calling (tool use)?
A primer on function calling — the JSON-schema contract that lets an LLM invoke your code. The request/response loop, parallel calls, and forced tools.
-
What is prompt caching?
Prompt caching cuts repeated-prompt cost 50–90% and halves latency. How prefix matching works, TTL economics by provider, and what decides your hit rate.
-
What is an AI agent? A primer for cloud engineers
A primer on AI agents — the perceive-reason-act loop, what separates an agent from a one-shot LLM call, and the classical agent types SREs now operate.
-
What is Model Context Protocol (MCP)?
A primer on Model Context Protocol — the open standard that lets AI applications talk to tools through one interface. Hosts, clients, servers, transports.
-
What is Retrieval-Augmented Generation (RAG)?
A primer on Retrieval-Augmented Generation — grounding an LLM's answer in documents you trust. Indexing, serving, and the failure modes that bite.
-
Mental models for applying AI to infrastructure
Tutorials answer how; mental models answer whether. Seven I use as the front gate before any LLM goes near a production system.
-
The MCP gateway pattern: five jobs your agent runtime can't skip
Letting agents call MCP servers directly repeats the no-API-gateway mistake. The five jobs an MCP gateway must do, with reproducible patterns for each.