There is a conversation I have had maybe a dozen times in the last two years, and it always goes the same way.
Someone says their platform is ready for AI workloads. I ask what that means. They say they have GPU nodes. And then it turns out that what they have is nvidia.k8s.io/gpu: 1 in a resource request, a device plugin somebody installed in 2024, and a shared node pool where one team’s fine-tuning job starves another team’s inference service every Thursday.
That is not an AI platform. That is a cluster with expensive hardware in it.
CNCF has been quietly building the thing that names the gap, and I think most platform teams should read it — not to get certified, but to find out what they are missing.
The program, briefly
The Kubernetes AI Conformance program defines a baseline of capabilities a Kubernetes platform needs to run AI and ML workloads. Its one-line pitch is a portability promise:
if your AI application works on one conformant platform, it should work on others too
It targets Kubernetes v1.33 through v1.36. It groups requirements across accelerators, networking, scheduling, observability, security, and operator support. And it recognizes three workload classes:
| Class | What it needs |
|---|---|
| Training | Distributed jobs, accelerator access, predictable scheduling |
| Inference | Model and LLM serving, latency targets, scale behavior |
| Agentic | Multi-step workflows combining tool calls and long-running tasks |
Certification is a pull request: review the requirements, document compliance with evidence, submit, and CNCF reviews it in roughly ten business days. Certifications last a year.
Certification today is self-assessment. Automated conformance tests are on the roadmap for 2026. I will come back to why that matters.
The third row is the interesting one
Training and inference on Kubernetes are well-trodden. There are opinions, there are operators, there are people who have been arguing about gang scheduling since 2019. Fine.
Agentic workloads are new as a scheduling category, and I think the significance of putting them in a conformance baseline is being undersold.
Here is why they break assumptions. A training job is long-lived and compute-saturated — you give it accelerators and it uses them until it is done. An inference service is short-lived per request and latency-bound — you scale replicas against queue depth. An agentic workload is neither:
- Long-lived but bursty. A run might last twenty minutes and spend eighteen of them blocked on a tool call, an API, or a human approval.
- Stateful across steps. It carries context between calls. Rescheduling it mid-run is not free the way restarting a stateless inference pod is.
- Spending wall-clock on things that are not compute. Your utilisation metrics will look terrible and be lying to you.
- Partially failing. Step 7 of 12 fails, and the correct response is not “restart the pod.”
If you are provisioning agentic workloads with the mental model you use for inference services, the symptoms are predictable: aggressive HPA thrash, pods evicted mid-run, accelerators reserved and idle, and cost dashboards that say you are wasting money on GPUs you are actually blocked on Slack waiting for.
I wrote about the operational side of this in agent sprawl — the point there was that the third agent is the signal to build a platform surface. This is the substrate half of the same argument. The workload class is different enough to need its own scheduling story, and now there is a standards body saying so.
Read it as an audit, not a certification
Here is my actual recommendation, and it is a bit against the grain of what a conformance program wants from you.
Do not certify. Do the gap analysis.
Certification is a real project — evidence collection, documentation, a PR, a review cycle, then annual renewal. If you are a vendor selling a managed Kubernetes platform, that cost is obviously worth paying, because the badge is a procurement asset.
If you are an internal platform team, the badge buys you nothing. Nobody is shopping for your cluster. What you want is the list.
So take the requirement categories and turn them into questions you can answer honestly in an afternoon:
Accelerators. Can a workload request a specific accelerator type, or just “a GPU”? Do you expose topology — which devices share an NVLink domain, which are on the same PCIe root? Can you fractionally share a device, and do you know which teams need to?
Scheduling. Do you have gang scheduling for distributed training, so a job that needs 8 workers does not get 6 and deadlock? Do you have priority and preemption classes that distinguish “batch fine-tune, restartable” from “inference, customer-facing”? What happens when both want the same node?
Networking. Is high-throughput inter-node networking exposed to workloads, or are your multi-node jobs quietly running over the default CNI? For serving, can you route on model identity rather than just service name — which is the thing an inference gateway exists to solve?
Observability. Are accelerator utilisation and memory pressure in the same place as your ordinary pod metrics, or in a separate dashboard nobody opens? Can you attribute GPU-seconds to a team? For agentic workloads, do you have traces that survive the whole multi-step run, per the OTel GenAI conventions?
Security. Where do model weights come from, and is that path verified? Who can schedule onto accelerator nodes? What identity does a workload use to reach a model endpoint — and if the answer is a shared API key, that is the thing to fix.
Operators. Which of your AI-serving components are operator-managed with a real reconciliation loop, and which are a Helm chart somebody installed once and has been hand-patching since?
That last question is usually where the room goes quiet.
The self-assessment problem
I want to be straight about the weak point, because the program is straight about it: certification today is self-assessment, and automated tests are still planned rather than shipped.
That means a conformance badge currently tells you a vendor read the requirements and asserted they meet them. It does not tell you a test suite verified it.
That is not nothing. Assertions have accountability attached — a vendor claiming conformance in a public PR has staked something. As a procurement filter it beats a marketing page. But it is a different artifact from Kubernetes conformance proper, which is enforced by a test suite that either passes or does not, and I would not want a platform team to conflate the two when evaluating a managed offering.
If you are evaluating vendors on this: ask which requirements they self-assessed as met, and ask them to show you the evidence they submitted. It is a public PR. You can read it.
What is happening in November
CNCF announced the KubeCon + CloudNativeCon North America 2026 schedule on August 10, and added a new AI Inference + Agentic track — Kubernetes, AI inference, agentic workflows, GPU scheduling, model serving, observability. Sessions feature vLLM, KServe, Ray, and OpenTelemetry. November 9–12, Salt Lake City.
Jonathan Bryce, CNCF’s executive director, framed it this way:
AI is quickly becoming one of the largest compute workloads the industry has ever seen, and the shift from training models to running them in production is where the real engineering challenge lives now.
That matches what I see. The interesting problems moved from “can we train it” to “can we run it on Tuesday without paging someone,” and that is squarely platform engineering work.
The Monday co-located day includes Cloud Native AI + Inference Day, alongside ArgoCon, BackstageCon, CiliumCon, and WasmCon. If you are only going for one day and you own an AI platform, that Monday is probably the highest-density option.
There is also a session called “Kubernetes Solutions for Agent-Shaped Problems” from Tim Hockin and Dmitry Berkovich at Google, which — given that Hockin has been shaping Kubernetes scheduling since before most of us had a GPU node — is the one I would clear the calendar for.
The uncomfortable summary
A conformance standard existing means the industry has agreed there is a baseline to fall below. That is genuinely useful, and it is also slightly embarrassing, because most of the requirements are things a competent platform team would have said they already had.
You probably have GPU nodes. The question the conformance list asks is whether you have a platform — whether a team can express what their workload needs, get it scheduled without a Slack negotiation, see what it cost, and have it behave the same way next quarter when you change the node pool.
Go read the requirement list. Answer the six questions above honestly. Then decide what to fix.
The certification is optional. The gap is not.
Related
- Agent sprawl is your next production incident — the operational half of the agentic-workload argument
- The inference gateway pattern: model-aware routing — what the networking requirement is really asking for
- Tracing the agent loop with OTel GenAI conventions — observability for multi-step runs
- Agents need identities, not API keys — the workload identity question
Sources: cncf/k8s-ai-conformance · CNCF: KubeCon NA 2026 schedule announcement
Comments