· 8 min read ReliabilityAIPlatform Engineering

Your top AI failure mode is saturation, and the SLI is in a header you throw away

Datadog found ~5% of production AI requests fail, nearly 60% on capacity limits. Provider saturation is an SRE problem you already have the primitives for.

Diagram: provider capacity as a saturation problem, showing rate limit headroom exported as an SLI, the divergent handling of retryable and non-retryable 429s, and a tiered fallback path under saturation

Here is a number worth sitting with.

Datadog’s State of AI Engineering 2026 report, published on 21 April 2026, found that around 5% of AI model requests fail in production, and that nearly 60% of those failures are caused by capacity limits.

Read that as an SRE and the priority ordering falls out immediately. The dominant failure class in production AI systems is not hallucination, not a bad prompt, not a schema mismatch in a tool call. It is saturation — running into a ceiling on a resource you do not own.

Which is good news, in a way. Saturation is not a new problem. We have thirty years of practice at it. The problem is that most AI stacks are not instrumented as though saturation were the main event, so it shows up as an unexplained error rate rather than as a capacity signal you can act on ahead of time.

The signal is already in your responses

Start with the thing that costs nothing.

Every response from a rate-limited model API carries headroom information. On the Claude API, the documented response headers include:

anthropic-ratelimit-requests-limit
anthropic-ratelimit-requests-remaining
anthropic-ratelimit-requests-reset
anthropic-ratelimit-input-tokens-limit
anthropic-ratelimit-input-tokens-remaining
anthropic-ratelimit-input-tokens-reset
anthropic-ratelimit-output-tokens-limit
anthropic-ratelimit-output-tokens-remaining
anthropic-ratelimit-output-tokens-reset

That is a saturation ratio, delivered on every successful request, for free. remaining / limit is exactly the metric you would build if you owned the resource.

Almost nobody exports it.

The usual instrumentation for an LLM call captures latency, token counts, cost, and status code. Those are RED metrics — rate, errors, duration — and they are genuinely useful. But RED tells you about work that already happened. The rate-limit headers are the U and S of USE: utilization and saturation, on the one dependency most likely to break you.

The gap this creates is the classic one. Without the headers, your first indication of a capacity problem is a 429. With them, you have a gauge that crosses 80% minutes earlier, on a Tuesday, while someone is at a desk.

Export three gauges per model, per workspace:

  • llm.ratelimit.requests.utilization = 1 − (requests-remaining / requests-limit)
  • llm.ratelimit.input_tokens.utilization
  • llm.ratelimit.output_tokens.utilization

Alert on sustained high-water marks, not instantaneous peaks — bursty traffic will touch the ceiling briefly and that is fine. What you care about is the trend line that says you will be capacity-bound by Thursday.

Two subtleties worth knowing. The headers reflect the most restrictive limit currently in effect, so if a workspace-level limit binds before the organization limit, that is what you see — which is what you want. And the anthropic-workspace-id response header tells you which workspace a request counted against, which is how you attribute saturation to a team rather than to “the AI bill.”

Not all 429s are the same, and one of them is a trap

Now the part that actually bites people.

You would reasonably assume a 429 means “back off and retry.” Most of the time it does. The Claude API returns 429 with a retry-after header when you exceed RPM, ITPM, or OTPM, and honouring it is correct.

But there is a second 429 with the same error type and the opposite correct behaviour. When an organization reaches its monthly spend cap, requests return HTTP 429 with error type rate_limit_error — and, per the documentation, “the response has no retry-after header. Retrying, including the SDKs’ automatic retries, fails until access resumes.”

Access resumes at 00:00 UTC on the first day of the next month.

Sit with the failure mode. Your service starts returning 429s. Your client library, doing exactly what a good client library does, retries with exponential backoff. Your dashboards show elevated error rate and elevated retry volume. Every retry fails. This continues until someone raises the tier — and if it is the third of the month, that is a long outage driven by a control that was supposed to prevent overspend.

The distinguishing field is error.details.error_code, which is enforced_spend_limit_reached on the Messages API. There is a third variant too: a self-imposed spend limit below the tier cap returns HTTP 400 with invalid_request_error, which most retry logic will correctly not retry but most alerting will file under “client error, probably a bad request” and ignore.

Three failure classes that look alike and need different handling:

SymptomMeaningCorrect response
429, rate_limit_error, has retry-afterGenuine rate limitBack off, retry, shed load if sustained
429, rate_limit_error, no retry-after, enforced_spend_limit_reachedTier spend cap hitStop retrying. Page a human. Fail over or degrade
400, invalid_request_error, “reached your specified API usage limits”Self-set limit hitRaise the limit — a config decision, not an incident

If your integration collapses these into “LLM call failed, retry,” you have a retry storm waiting for a bad month. Classify the error at the client boundary and emit a distinct metric for each. This is twenty lines of code and it converts a multi-hour outage into a page with an obvious remediation.

Your own autoscaling can trigger this

A detail that deserves more attention than it gets. The same documentation notes:

You might also encounter 429 errors because of acceleration limits on the API if your organization has a sharp increase in usage. To avoid hitting acceleration limits, ramp up your traffic gradually and maintain consistent usage patterns.

So a sharp ramp can be rate limited even below the nominal ceiling.

Now think about when sharp ramps happen in your system. A scale-out event. A retry storm from a downstream recovery. A backfill job someone kicked off. An AI SRE agent responding to an incident by fanning out fifty parallel investigations.

That last one is the uncomfortable one, and it is a genuine feedback loop: incident happens → agent fleet scales up its investigation → provider acceleration limit trips → agent calls start failing → the agents retry → the limit stays tripped. Your remediation system has become a load generator against the dependency it needs.

The mitigations are the ordinary ones, which is reassuring. Rate-limit your own egress to the provider below the provider’s ceiling, so you shed load in a controlled way rather than having it shed for you. Cap concurrency per agent session. And give the agent fleet a token budget per incident, which is an argument I have made before in the context of error budgets for autonomy — the budget is what stops a well-intentioned agent from consuming the capacity that human responders need.

Cache hit rate is a throughput lever

This one is genuinely underrated, because caching is filed under cost and stops there.

For most Claude models, only uncached input tokens count toward the input-tokens-per-minute limit. Cache reads do not count. Anthropic’s own worked example: with a 2,000,000 ITPM limit and an 80% cache hit rate, you can effectively process 10,000,000 total input tokens per minute — 2M uncached plus 8M cached.

That is a 5× throughput increase from a caching strategy, without a tier change, without a support ticket, without negotiating anything.

For agent workloads specifically the ceiling is high, because agent traffic is unusually cacheable: a large stable system prompt, a stable tool catalog, and a conversation history that grows by append. That is close to the ideal shape for prefix caching.

Which reframes cache hit rate. It is not a cost metric that also helps latency. It is a capacity metric — arguably the cheapest capacity lever available to you — and it belongs on the same dashboard as the saturation gauges above. The unit economics of inference argument applies here in reverse: the same lever that lowers your cost per request raises your ceiling on requests per minute.

Caveat worth respecting: the rules are model-specific. Claude Haiku 3.5 counts cache reads toward ITPM. Check before you build a capacity plan on an assumption.

Designing for a ceiling you do not control

Instrumentation tells you where you are. The architecture question is what happens when you get there.

Tier your fallbacks. Rate limits apply separately per model, so a model class that is saturated does not saturate the others. That makes a fallback ladder real capacity, not just cost engineering — the SLM tier model argument, applied to availability. When the frontier tier is at its ceiling, degrading to a smaller model is a better outcome than a 429, for many request classes.

Classify requests before you need to. Under saturation you want to shed the deferrable work and protect the interactive path. That requires a classification that already exists in the request — interactive versus batch, user-facing versus background — because you cannot invent one during an incident.

Move deferrable work off the synchronous path. Batch APIs have their own separate limit pools. Anything that does not need an answer in seconds should not be competing for synchronous capacity with something that does.

Route through one place. All of the above — classification, shedding, fallback, egress rate limiting, header export — wants a single chokepoint. This is the same case as model-aware routing in the inference gateway: capacity policy implemented in eleven services is capacity policy you do not have.

Put it in the SLO. If capacity failures are ~60% of your AI error budget consumption, they belong in the error budget explicitly. A capacity_saturation failure category, tracked and reviewed, changes the conversation from “the AI was flaky again” to “we spent 40% of the budget on provider saturation this month, here are the three levers.”

The uncomfortable summary

The most common way production AI systems fail is the most boring one, and it is the one your team is best equipped to handle. It just is not on the dashboard.

  • The saturation signal ships on every response. Export it.
  • Two 429s mean opposite things. Classify them at the client.
  • Your own scale-out can trip acceleration limits. Rate limit your egress.
  • Cache hit rate is throughput, not just cost.
  • Fallback tiers are availability, not just savings.
  • Capacity belongs in the error budget, named.

None of this requires a new discipline. It requires treating a model provider as what it is: a shared dependency with a ceiling somebody else sets, which is a thing SRE has known how to manage since long before any of this was called AI.

Sources: Datadog, State of AI Engineering 2026 press release, 21 April 2026 · Anthropic API rate limits documentation

Frequently asked questions

How often do production AI requests actually fail?

Datadog's State of AI Engineering 2026 report, published 21 April 2026, found that around 5% of AI model requests fail in production, and that nearly 60% of those failures are caused by capacity limits. That makes saturation the single largest named failure class in production AI systems — larger than the model-quality problems most teams spend their attention on.

Why is provider saturation different from ordinary rate limiting?

Because you do not control the resource, cannot provision more of it on demand, and often cannot see how close you are to the edge unless you deliberately instrument for it. It behaves like a shared dependency with an externally-set ceiling, which is a familiar SRE shape — but most teams treat it as an occasional error to retry rather than as a saturation signal to manage.

What is the saturation signal I am not exporting?

Rate limit response headers. Anthropic's API, for example, returns anthropic-ratelimit-input-tokens-remaining, anthropic-ratelimit-output-tokens-remaining, anthropic-ratelimit-requests-remaining, and matching limit and reset fields on every response. Remaining over limit is a saturation ratio available on every successful call, which means you can alert on approaching the ceiling instead of discovering it when requests start failing.

Is retrying a 429 always the right response?

No, and this is a trap worth knowing about. On the Claude API a rate limit 429 carries a retry-after header and retrying works. The 429 returned when an organization hits its monthly spend cap has the same rate_limit_error type but no retry-after, and retrying — including the SDKs' automatic retries — fails until access resumes at the start of the next month. Two identical-looking errors with opposite correct responses.

Can autoscaling cause its own rate limiting?

Yes. Anthropic documents acceleration limits: an organization with a sharp increase in usage can receive 429s even below its nominal limits, and the documented mitigation is to ramp traffic gradually and maintain consistent usage patterns. A scale-out event that triples request rate in thirty seconds is exactly the pattern this penalizes, which means your own remediation can be the trigger.

Does prompt caching help with capacity, or only cost?

Both, and the capacity effect is underrated. For most Claude models only uncached input tokens count toward the input-tokens-per-minute limit — cache reads do not. Anthropic's own worked example is that a 2,000,000 ITPM limit with an 80% cache hit rate lets you process roughly 10,000,000 total input tokens per minute. Cache hit rate is therefore a throughput lever, not just a billing one.

Comments