There is a specific moment in every self-hosted inference rollout where the graphs stop making sense.
Your GPU utilization looks fine. Aggregate throughput looks fine. Nothing is crash-looping. And yet p99 time-to-first-token is four times p50, users are complaining, and adding a replica makes it slightly worse. You stare at it long enough and eventually notice the thing that was true the whole time: your load balancer has no idea what it’s balancing.
That’s the problem the Gateway API Inference Extension exists to solve, and it’s now GA as an official Kubernetes project. I want to explain what it actually changes, because most of the coverage frames it as “AI-aware routing” — which is true and tells you nothing about whether you need it.
The assumption that breaks
Kubernetes load balancing is built on three assumptions that hold beautifully for web services:
- Requests are short and roughly equal in cost.
- Any healthy backend can serve any request equally well.
- Backend state doesn’t affect which replica you should pick.
Inference violates all three, and it violates them badly enough that the standard tooling doesn’t degrade gracefully — it degrades invisibly.
Requests are not equal. A 200-token classification and a 60k-token document summarization are the same thing to kube-proxy: one connection. They are not the same thing to a GPU. Cost varies by orders of magnitude, and it varies along two axes (prompt length and generated length) that you can only partly observe up front.
Backends are not interchangeable. This is the one that surprises people coming from stateless-service land. A vLLM replica that just served a request sharing your conversation prefix is holding that prefix in its KV cache. Routing you there means it skips prefill for those tokens — the same economics as prompt caching, decided by your load balancer instead of your provider. Routing you to its identical-looking sibling means recomputing all of it. Same image, same resources, same readiness probe — wildly different latency. If you run multi-turn chat or agent loops (where every step resends a growing transcript), this is not a marginal effect. It is the dominant one.
Backend state absolutely matters. KV cache utilization determines whether the server is about to start evicting and recomputing. Queue depth determines how long you wait before your prefill even begins. Both are visible in the model server’s metrics and invisible to a Service.
So round-robin doesn’t just leave performance on the table. It actively destroys the caching behavior your inference stack spent enormous engineering effort building. Every production stack in 2026 ships paged attention by default — vLLM, SGLang, TensorRT-LLM — and then we put a load balancer in front that scatters requests across replicas as if the cache didn’t exist.
What the Inference Gateway does instead
The design is pleasantly boring, which I mean as praise.
It doesn’t replace your gateway. It upgrades one — any ext-proc-capable Gateway API implementation (Envoy Gateway, kgateway, GKE Gateway, Istio, Alibaba’s ACK Gateway, and agentgateway, which was the first to pass GIE v1.4.0 conformance). You keep your existing routing, TLS, and policy stack.
Two pieces are new:
InferencePool — the core resource. Where a Service says “here is a set of pods,” an InferencePool says “here is a set of model-serving endpoints, and here is the router that picks among them.” It’s the noun that makes model servers a first-class thing in the API rather than generic backends.
The Endpoint Picker (EPP) — a data-plane component speaking Envoy’s external processing protocol. It intercepts each inference request before it’s forwarded and selects the replica. Not the next replica. The right one.
The selection uses signals the model servers already export:
| Signal | Question it answers |
|---|---|
| Prefix cache status | Does a replica already hold this prompt’s prefix? |
| KV-cache utilization | Is this replica near eviction pressure? |
| Queue depth | How long until this request actually starts? |
| LoRA adapter availability | Is the requested adapter loaded, or does routing here force a swap? |
| Request cost | How expensive is this request likely to be? |
The scheduling algorithm is pluggable — the project treats “which scorers, weighted how” as configuration rather than doctrine, which is the correct call given nobody has a universal answer yet.
That LoRA row deserves a note. If you serve many fine-tuned variants off shared base weights, adapter-aware routing is arguably a bigger win than cache-aware routing, because an adapter swap is a hard cost you can often avoid entirely by sending adapter-A traffic to the replicas that already have adapter A resident. This is the multi-tenant serving pattern, and it’s close to unworkable without routing that understands adapters.
The honest caveat about the reference EPP
The Endpoint Picker in the upstream repo is described as a lightweight implementation provided for conformance testing. The project points production users at llm-d-router or their own implementation, and the InferenceObjective and InferenceModelRewrite APIs have already moved out to llm-d.
I read that as a healthy split rather than a warning: the Kubernetes project owns the interface (InferencePool, ext-proc contract, conformance tests), and the ecosystem competes on scheduling quality. That’s how Gateway API itself worked, and it worked well. But it does mean “adopt the Inference Gateway” is really two decisions — adopt the API surface, then choose a router — and only the first one is settled.
How to tell if you need this
I’d rather give you a decision rule than a recommendation.
You don’t need it if: every token you consume comes from a hosted API. Your routing problems are real, but they’re provider failover, per-tenant quotas, cost attribution, and prompt-cache hit rates. That’s an LLM proxy or an MCP gateway job, not an InferencePool job.
You don’t need it yet if: you self-host, but at one replica per model. There’s nothing to route between. Fix your batching and quantization first — that’s where the money is.
You probably need it if two or more of these are true:
- You run more than one replica of a self-hosted model.
- Your traffic is multi-turn — chat, agent loops, anything that resends a growing transcript.
- You serve multiple LoRA adapters off shared base weights.
- Your p99/p50 TTFT ratio is worse than 3x and you can’t explain it from GPU utilization.
- You’re being asked to cut GPU spend and have already done the easy things.
That last one is worth sitting with. Cache-aware routing is one of the few remaining optimizations that improves latency and reduces cost at the same time, because a prefill you skip is compute you don’t buy. Most levers trade one against the other.
What to measure before you migrate
Do not adopt this on faith. The whole argument is empirical, so measure it.
Before you touch anything, capture a baseline for a representative window:
# TTFT distribution — the metric the routing change should move
histogram_quantile(0.99,
sum(rate(vllm:time_to_first_token_seconds_bucket[5m])) by (le)
)
/
histogram_quantile(0.50,
sum(rate(vllm:time_to_first_token_seconds_bucket[5m])) by (le)
)
# Prefix cache hit rate — the mechanism
sum(rate(vllm:prefix_cache_hits_total[5m]))
/
sum(rate(vllm:prefix_cache_queries_total[5m]))
# Per-replica queue depth spread — the smoking gun for bad balancing
max(vllm:num_requests_waiting) - min(vllm:num_requests_waiting)That third one is the tell. If your replicas are well balanced by a naive load balancer, the spread stays tight. If you see one replica at 40 waiting requests while another sits at 2, you have a distribution problem that no amount of horizontal scaling fixes — you’ll just add more idle replicas next to the hot one.
The cache hit rate is the mechanism you expect to improve, and TTFT ratio is the outcome. If hit rate climbs and p99 doesn’t move, something else is your bottleneck — probably prefill compute or memory bandwidth — and you should go look at how your storage and interconnect are set up before blaming the router.
Roll it out on one model, shadow or canary, and compare the same three numbers. This is the same discipline I’d apply to evaluating any AI infrastructure vendor claim: run it against your own traffic, because the benchmark that matters is yours.
The part I find more interesting than the performance
For a couple of years the AI platform conversation has been stuck in a slightly embarrassing place, where “the platform” meant a pile of Python glue nobody wanted to own and the actual infrastructure people had no API surface to work with.
An InferencePool is a small thing. But it’s a Kubernetes resource with a conformance suite behind it, which means model serving is now something you can express in the same vocabulary as everything else you run — routing policy, RBAC, quotas, GitOps, admission control. The same shift happened when MCP made tool calls visible to L7 infrastructure: the interesting part wasn’t the feature, it was that a class of AI behavior stopped being special and became ordinary config.
That’s the direction I’d bet on generally. The AI infrastructure that survives contact with real organizations won’t be the stuff that demands its own parallel universe of tooling. It’ll be the stuff that shows up as a CRD, exports Prometheus metrics, fails a conformance test when it’s wrong, and can be reviewed by someone who has never read a paper about attention.
Round-robin for LLM traffic was always a placeholder. It’s good that we finally have the thing it was holding a place for.
If you’re implementing this: start by graphing the queue-depth spread across your replicas for a week. You’ll know within a day whether this post is about your cluster or somebody else’s.
Sources: Gateway API Inference Extension docs · kubernetes-sigs/gateway-api-inference-extension · Introducing Gateway API Inference Extension (Kubernetes blog) · agentgateway on GIE conformance · KV cache optimization techniques, 2026
Comments