fidzu : k8s

In this SIG Architecture spotlight we talked with Janet Kuo (Google) and Yuan Tang (TensorScale AI) about the journey of AI Conformance in Kubernetes, from the initial Working Group to the integration SIG Architecture as a subproject.

Introductions

Frederico Muñoz and Kirti Goyal (FK): Hi Janet, Yuan, and thank you for the opportunity to learn a bit more about AI Conformance. Can you tell us a bit about yourselves and how you got involved in Kubernetes?

Janet Kuo: I'm a Senior Staff Software Engineer at Google and have been a Kubernetes maintainer since 2015, joining the community just as we were racing toward the 1.0 launch. I started my journey writing the core workload controllers that standardized how applications are orchestrated on Kubernetes. Today, I lead SIG Apps as a Co-chair & TL, and also lead the AI Conformance program. My current focus is to ensure Kubernetes remains the best platform for the next generation of workloads, such as AI.

Yuan Tang: I'm a co-founder of TensorScale AI where we build AI infrastructure for world models. Prior to that, I was a Senior Principal Software Engineer at Red Hat, where I led the development of Red Hat AI platform and upstream open source strategy. I became involved in Kubernetes early on by fostering collaborations between the Kubernetes community and projects I was leading, including Kubeflow, Argo, and KServe. I later helped found and lead WG Serving and the AI Conformance program, advancing innovation and strengthening community collaboration around running AI workloads on Kubernetes.

Program, Working Group, Subproject: the many shapes of AI Conformance

FK: AI Conformance started as a CNCF Program, was it then you got involved?

Janet Kuo: I was actually involved from day one since I pitched the original idea. It became obvious then that everyone was starting to run AI on Kubernetes, but vendors were building their infrastructure stacks in different ways. I got worried about ecosystem fragmentation. So I wrote up the initial Kubernetes AI Conformance Proposal, took it to the CNCF leadership to get their buy-in, and started the AI Conformance Program. Once we had the green light, I reached out to my fellow co-chairs Yuan Tang, Rita Zhang, and Mario Fahlandt to get the community rallied around it.

Yuan Tang: Yes. I was a co-chair of WG Serving at that time and some of the initiatives such as the Gateway AI Inference Extension quickly gained traction. There was a strong interest among the community to include inference-related requirements in the Kubernetes AI Conformance program.

FK: How would you describe the motivation behind the creation of the AI Conformance Working Group?

Janet and Yuan: The motivation behind creating the Kubernetes AI Conformance Working Group (AI WG) was to address the rapid convergence of AI workloads onto Kubernetes while preventing ecosystem fragmentation.

As AI adoption accelerated, organizations began running increasingly complex workloads, including distributed training, large-scale inference, GPU-intensive pipelines, and emerging agentic AI systems across many different Kubernetes platforms. However, vendors were implementing AI infrastructure capabilities differently, especially around GPUs, networking, scheduling, and inference orchestration. That created portability challenges, operational inconsistency, and growing vendor lock-in concerns.

The AI WG was formed to bring the community together around:

• vendor-neutral AI infrastructure standards,
• interoperable Kubernetes-based AI platforms,
• consistent operational behavior for AI workloads,
• and common requirements for scalable AI deployment.

FK: WGs are temporary by nature, based on outcomes: would you say the main goals of the WG were achieved?

Janet and Yuan: Yes, the WG achieved its primary initial goals.

The WG has succeeded in moving the ecosystem from: fragmented, vendor-specific AI infrastructure approaches to a shared, community-defined baseline for AI workloads on Kubernetes.

Some concrete indicators of success include:

• Launching the Certified Kubernetes AI Conformance Program
• Defining formal Kubernetes AI Requirements (KARs)
• Getting broad participation from major ecosystem players
• Growing certified platforms from 18 to 31 within a few months
• Aligning around newer Kubernetes primitives needed for AI workloads
• Establishing a roadmap for automated conformance verification

Most importantly, the WG created consensus that AI workloads are now a first-class Kubernetes use case.

FK: AI Conformance was specifically highlighted at the latest KubeCon: is this a reflection of the growing importance of AI throughout the project?

Janet and Yuan: Yes, the prominence of AI Conformance at the latest KubeCon + CloudNativeCon is a strong reflection of how central AI has become to the Kubernetes ecosystem overall.

A few years ago, AI workloads on Kubernetes were still viewed as somewhat specialized, mostly tied to ML platforms or GPU-heavy research environments. What's changed is that AI is now becoming a mainstream infrastructure concern for nearly every large-scale platform operator.

That shift naturally elevates AI from an application running on Kubernetes to a core workload category Kubernetes must optimize for.

The AI Conformance initiative reflects that evolution. It signals that the community now sees AI infrastructure requirements, things like accelerator scheduling, high-performance networking, workload-aware orchestration, and low-latency inference, as important enough to standardize at the ecosystem level.

Its visibility at KubeCon also shows a broader strategic realization: if Kubernetes is going to remain the universal control plane for modern infrastructure, it needs to provide a consistent foundation for AI workloads across clouds, on-prem, and edge environments.

So the emphasis on AI Conformance is not just about chasing AI hype; it's about Kubernetes adapting to a major shift in workload patterns and ensuring the ecosystem evolves cohesively rather than fragmenting into vendor-specific AI stacks.

From Working Group to SIG Architecture subproject

FK: How would you describe the change from WG to SIG? Is it a reflection of AI as something that requires ongoing architectural governance?

Janet and Yuan: In Kubernetes, Working Groups are meant to be temporary. Our WG had a specific goal to get the initial Kubernetes AI Conformance requirements published and get the certification process off the ground. Once we hit those milestones, we archived the WG and moved AI Conformance to be a permanent subproject under SIG Architecture.

Moving to SIG Architecture definitely proves that AI needs ongoing governance. AI isn't just a passing trend or a one-off workload. It has become a core part of what Kubernetes runs every day. Because SIG Architecture already owns overall Kubernetes Conformance and API design principles, housing the AI Conformance subproject there makes perfect sense. It makes sure our AI Conformance requirements stay aligned with the rest of the Kubernetes ecosystem for the long haul.

FK: And with that change, how would you describe the main goals and areas of intervention of the subproject?

Janet and Yuan: Now that we're a subproject, our main goal is to keep the AI Conformance standards current, vendor-neutral, and standard-setting. We want to make sure Kubernetes captures the latest advancements in AI infrastructure while keeping the ecosystem fair for everyone. When we find gaps where no clear community pattern exists, we help foster and form new standards to solve them.

AI Conformance and Kubernetes

FK: AI Infrastructure is evolving very quickly, while Kubernetes APIs are designed to remain stable for years. Is this something you consider a challenge?

Janet and Yuan: It's definitely a tricky balance. AI moves very fast, while Kubernetes is designed to be stable and boring. But we actually structured the program around this exact tension.

First, we use a tiered graduation model. When new AI infrastructure concepts emerge, we introduce them as recommended SHOULD requirements. That gives the ecosystem room to experiment and agree on best practices without locking Kubernetes into rigid APIs too early. Once those APIs stabilize and with ecosystem adoption, we graduate them to mandatory MUST requirements.

Second, we intentionally keep our focus on the infrastructure layer. We test for core capabilities like accelerator support through Dynamic Resource Allocation (DRA), rather than getting bogged down in specific AI frameworks or model servers. That keeps the base platform stable while letting the AI tooling evolve as fast as it needs to.

FK: DRA has evolved, also due to the work of WG Device Management, as an important part of Kubernetes (and especially so for AI workloads), with SIG Architecture having an important role in how it's integrated in Kubernetes: do you find any parallel in this? Specifically, do you think there will be a heavier focus in the integration with the evolving Kubernetes architecture as part of SIG Architecture?

Janet and Yuan: There is a huge parallel there. Whenever we face a massive, complicated challenge that touches multiple parts of the codebase and involves multiple SIGs, a Working Group is the best place to start. It lets experts from different SIGs get together to figure out the problem, and build consensus.

Being part of SIG Architecture means there will be a much heavier focus on how AI capabilities weave into the evolving Kubernetes architecture. Instead of treating AI as an add-on layer, we're looking at how core Kubernetes primitives, like scheduling and resource allocation, need to adapt at a foundational level to support the next decade of AI workloads.

Final remarks

FK: For interested contributors, what's the best way to get involved with the AI Conformance subproject?

Janet and Yuan: We're always looking for new contributors. The easiest way to get started is by checking out our main repo at kubernetes-sigs/ai-conformance and join our meetings. If you're interested in defining new AI Conformance requirements, you can jump into PR reviews or propose your own using our KAR template. If you want to write code, we always need help expanding our automated Go test suites in the test/ directory.

For vendors looking to certify their platforms, all the details are at cncf/k8s-ai-conformance.

FK: Any final comments you would like to share?

Janet and Yuan: We've been blown away by how fast the community has rallied around this. When we first announced the AI Conformance program during the keynote at KubeCon NA 2025, we weren't sure what the response would be. But by the time we gave our update keynote at KubeCon EU 2026, we had grown from 18 certified platforms to 31 in just a few months.

That kind of growth shows just how much the industry wanted a unified, vendor-neutral standard. It gives organizations the confidence that they can run their AI workloads on any certification Kubernetes platform without getting locked into a proprietary stack.

If you're a platform vendor, we'd love to see you get certified. If you're a developer interested in the intersection of AI and cloud-native infrastructure, come join us in SIG Architecture!

FK: Excellent, thank you both for guiding us in this journey on AI and AI Conformance in Kubernetes.

29 Oct 2026 5:00am GMT

We are excited to announce that the Kubernetes community now has a Workload Conformance Working Group. The working group is writing a shared standard for how workloads behave at runtime on Kubernetes. To get it right, we need input from the people who build those workloads and from the people who run the clusters they're deployed on.

Why workloads need a shared standard

Certified Kubernetes tests the cluster, and Kubernetes AI Conformance tests whether a cluster can run AI and ML workloads. Nothing comparable exists for the workloads themselves, so cluster operators still check each application by hand before routine maintenance or a Kubernetes upgrade.

The problems they find are familiar. A routine node drain is often how you discover that an application can't tolerate disruption. An upgrade can break an application that still depends on a removed API version. Others show up in day-to-day operation: workloads that ask for privileges they don't need, or reserve resources they never use.

Kubernetes already documents much of this, including security and handling disruptions. Even so, every cluster operator ends up repeating the same assessment, and application maintainers get the same questions from each environment they support. A common specification with repeatable tests would give both groups the same baseline to start from.

What the working group will deliver

The charter sets out three deliverables:

• A specification of workload behavior for each Kubernetes minor version.
• A method for testing each requirement against a running workload.
• Upstream conformance tooling that implements those tests.

The first step is deciding which workloads and behaviors the specification covers. The scope includes security, resilience during disruptions, resource use, networking, storage, API usage, and observability, but the group hasn't yet agreed on specific requirements or on how to test them. However, one rule is already set: every requirement needs an objective test that a test binary can run, so nothing subjective goes into the specification.

The specification covers how a workload runs, and importantly not what it does. Consequently, application logic is out of scope of workload conformance. The binary won't retest anything Certified Kubernetes already covers, and it won't require a particular workload packaging format, or vendor tool like policy engine, security scanner, etc.

How the work is organized

Two Special Interest Groups (SIGs) are sponsors for the working group.

• SIG Architecture will own the specification and tooling through a subproject, which will be created once the first artifacts are proposed by the Working Group.
• SIG Apps will bring the application perspective.

Whenever a requirement touches another Kubernetes SIG's area, such as node behavior or networking, the working group will initiate conversations with those SIGs as well.

The working group is a forum where SIGs can get involved and shape the direction of Workload Conformance while the specification is being written. We're aiming for a stable first specification and working tests within two to three Kubernetes release cycles.

Get involved

The specification will and needs to be built from real examples, so we want to hear your experiences. If you run clusters, tell us how you assess workloads today and what broke during your last upgrade or maintenance window. If you maintain an application, tell us what it needs from the cluster and where the existing guidance left you guessing.

• Beyond examples, the work needs reviewers for proposed requirements and contributors to write the tests and tooling.
• When early versions of the tests are ready, we also want early testers to run the binary against your own workloads and tell us where the results differ between runs or environments.
• If a requirement touches your SIG's area, help us check its scope and the test behind it.
• And because the subproject will outlive the working group, we're also looking for people who want to maintain it for the long term.

You can find us in #wg-workload-conformance on Kubernetes Slack (visit slack.k8s.io for an invitation) and on the mailing list. The working group meets every other Wednesday at 15:00 UTC. The Zoom link and the agenda are in the meeting notes. If you want to raise a topic, add it to the agenda. Joining the mailing list usually adds the meeting invite to your calendar.

Thanks to everyone who reviewed and supported the working group proposal.

27 Oct 2026 4:30am GMT

You submit a Kubernetes Job and wait for its Pods to appear. A minute passes. Then another. The Job is still suspended, there are no application logs, and there may not even be a Pod to inspect.

In a cluster managed by Kueue, this can be completely normal. Kueue may still be deciding whether the Job can use the requested resources. The useful question is not only "Why is the Pod pending?" It is first "Has Kueue admitted the Workload?"

The Kueue plugin for Headlamp brings that investigation into the Headlamp UI. Instead of reconstructing the admission path from several commands and large YAML objects, you can begin with the Workload, read its condition message, and follow links through the LocalQueue, ClusterQueue, ResourceFlavor, and Cohort.

This article follows that Headlamp-first path. The command line appears only in a short optional section for reproducing the example.

Before you begin

You need a Kubernetes cluster with Kueue installed, a Headlamp installation, and the 0.1.0-alpha Kueue plugin. Your Kubernetes identity must be allowed to read the Kueue resources used in this article.

The alpha release provides read-only views for Kueue v1beta2 Cohorts, ClusterQueues, LocalQueues, ResourceFlavors, and Workloads. It does not add its own create, edit, stop, resume, or delete workflows. Headlamp still provides its standard YAML and resource actions where your permissions allow them.

First, separate admission from Pod scheduling

Kueue and the Kubernetes scheduler answer different questions.

• Kueue admission: Should this Workload start now, considering queue policy, quota, ResourceFlavors, admission checks, and other waiting Workloads?

• Pod scheduling: After the Workload is admitted and the Job resumes, which nodes can run its Pods?

That boundary changes the investigation. If the Workload is not admitted, stay in the Kueue section of Headlamp. If it is admitted but its Pods are still Pending, move to Headlamp's normal Job and Pod views and inspect scheduler events.

The Headlamp-first troubleshooting path

The quickest investigation starts with the Workload, not with a shell.

Step 1: Open the Workloads view

In Headlamp, open Kueue → Workloads. Select the namespace that contains the Job, then find the Workload created for it.

Kueue creates a Workload to represent the resource request and admission state of a Job. In the alpha plugin, owner references are shown as Kind/name text but are not yet clickable. The Workload name normally contains the Job name, so the namespace filter and table search are the simplest ways to locate it.

Before opening the detail page, the list already gives you a useful first scan:

• LocalQueue

• priority and priority class

• active state

• admitted state

• completion state

• a readable current status such as Admitted, Finished, or Inadmissible

• age

If there is no matching Workload, check three things before going further: the selected namespace, whether the Job was submitted to a LocalQueue, and whether your Kubernetes account can list Workloads. The plugin performs permission checks and displays an access message when the current identity cannot read a resource.

Step 2: Read status, conditions, and events

Open the Workload. Start with its current status and conditions, not with the full YAML.

For admission troubleshooting, the most important condition types are QuotaReserved and Admitted.

• Admitted=True means Kueue has admitted the Workload. The Job can resume, so any remaining Pending Pods belong to the scheduling branch of the investigation.

• QuotaReserved=False with a reason such as Pending means Kueue has not reserved quota. The condition message explains what prevented the reservation.

• An Inadmissible status means the Workload currently cannot fit the available queue configuration. A missing covered resource, unsuitable ResourceFlavor, stopped queue, or admission check can all lead you here.

For example, a condition message may report that Kueue could not assign a ResourceFlavor because the ClusterQueue has insufficient CPU quota. That message is the first real answer to "Why is my Job still pending?" The rest of the Headlamp journey explains the configuration behind it.

Also check the Events section. Conditions describe the current state; events help show what recently changed or what Kueue attempted.

Step 3: Compare the Workload's request with its assignment

Stay on the Workload detail page and inspect the PodSets and resource requests. The plugin shows the requested Pod count, CPU, memory, and other resources that Kueue evaluates together.

This matters because a Job with parallelism: 3 and a one-CPU request per Pod asks Kueue for three CPUs in total. Looking at only one Pod's template would underestimate the Workload.

If quota has already been reserved, the same page also shows the assigned ClusterQueue, ResourceFlavors, and admitted resource usage. If those admission assignments are absent, the Workload has not reached that point yet.

In our example, the Workload requests:

• three Pods

• three CPUs in total

• 600 MiB of memory in total

The next question is where that request entered Kueue.

Step 4: Follow the linked LocalQueue

Use the LocalQueue link on the Workload detail page. A LocalQueue is the namespace-level entry point used by the Job. It connects namespaced Workloads to a cluster-scoped ClusterQueue.

The LocalQueue page shows:

• its namespace and connected ClusterQueue

• stop policy and current status

• pending, reserving, and admitted Workload counts

• Kueue conditions and Kubernetes events

• related Workloads in the same namespace

This page answers two practical questions: Did the Job enter the queue you expected, and is it the only Workload waiting there?

If the LocalQueue is inactive or stopped, that may already explain the wait. If it is active, follow the linked ClusterQueue to inspect the capacity behind it.

Step 5: Inspect ClusterQueue capacity and usage

A ClusterQueue represents the cluster-level quota and admission policy available to one or more LocalQueues.

On its Headlamp detail page, compare the Workload request with:

• covered resources

• nominal quota for each ResourceFlavor

• current reservations and usage

• borrowing and lending limits

• pending, reserving, and admitted Workload counts

• queueing strategy and stop policy

• preemption, fair-sharing, flavor-fungibility, and admission-check settings

• Kueue conditions and Kubernetes events

For the example in this article, the ClusterQueue provides a nominal quota of two CPUs and two Pod slots. The Workload asks for three of each. The condition message and the ClusterQueue page now tell the same story: the Workload does not fit, so Kueue leaves the Job suspended.

This is an admission-capacity problem. It is not evidence that kube-scheduler failed to find a node.

Step 6: Check ResourceFlavor and Cohort when needed

Quota is not always a single pool. A ResourceFlavor can represent a class of compute through node labels, taints, tolerations, or topology settings. Use the ResourceFlavor link from the ClusterQueue when the condition mentions flavor assignment or when the Workload requires a particular hardware class, such as GPU nodes.

If the ClusterQueue belongs to a Cohort, follow that link too. Cohorts let ClusterQueues share unused quota. The Headlamp page shows parent and child Cohorts, member ClusterQueues, referenced ResourceFlavors, resource groups, borrowing and lending limits, and fair-sharing information.

These pages help answer a more subtle question: Is the Workload blocked because the queue has no quota of its own, because it cannot borrow enough, or because no suitable flavor can be assigned?

What we learned without leaving Headlamp

For the sample pending Job, the UI gives us a complete explanation:

1. The Workload is not admitted.

2. Its QuotaReserved condition is false and the message points to insufficient quota.

3. The Workload requests three CPUs and three Pod slots.

4. Its LocalQueue is active and points to the expected ClusterQueue.

5. The ClusterQueue provides only two CPUs and two Pod slots.

6. Kueue therefore keeps the Job suspended until quota becomes available or the queue configuration changes.

The important part is not that Headlamp invents a new status. It reads the same Kubernetes API objects as other clients, then places the related resources and human-readable values in one connected investigation.

If the Workload is admitted, switch views

Suppose the Workload detail page shows Admitted=True, but the application is still not running. Kueue has completed its part of the decision. Continue in Headlamp's standard Kubernetes views:

1. Open Workloads → Jobs and inspect the Job state and events.

2. Open Workloads → Pods and filter by the Job name or namespace.

3. Select a Pending Pod and read its Events section.

At this stage, look for scheduler concerns such as insufficient node capacity, node affinity, taints and tolerations, topology constraints, persistent-volume binding, or image-pull failures.

This handoff is the most useful troubleshooting rule in the article:

• Not admitted: investigate Kueue resources in Headlamp.

• Admitted, but Pods are Pending: investigate Job, Pod, node, and scheduler information in Headlamp.

A repeatable Headlamp checklist

When the next Job does not start, use this short path:

1. Open Kueue → Workloads and select the correct namespace.

2. Find the Workload associated with the Job.

3. Check the readable status, Admitted, and QuotaReserved conditions.

4. Read the condition message and recent events.

5. Compare PodSet resource requests with any admission assignments.

6. Follow the LocalQueue link and confirm its status and ClusterQueue.

7. Follow the ClusterQueue link and compare quota, reservations, usage, and policy.

8. Inspect ResourceFlavor or Cohort details when the message points to flavor selection or shared quota.

9. If admitted, move to Headlamp's Job and Pod views.

What Headlamp replaces during this investigation

The plugin does not replace kubectl or kueuectl for scripts, exact object output, or changes. It replaces the repetitive UI-less navigation needed during an interactive investigation.

Question	Headlamp path	Command-line equivalent
Is the Workload admitted?	Kueue → Workloads → status and conditions	kubectl describe workload <workload-name> -n <namespace>
Which LocalQueue did it enter?	Workload → linked LocalQueue	kubectl get localqueue <localqueue-name> -n <namespace> -o yaml
Which ClusterQueue provides quota?	LocalQueue → linked ClusterQueue	kubectl get clusterqueue <clusterqueue-name> -o yaml
Which flavor or shared quota matters?	ClusterQueue → ResourceFlavor or Cohort	kubectl get resourceflavor <flavor-name> -o yaml or kubectl get cohort <cohort-name> -o yaml
What happened after admission?	Workloads → Jobs and Pods → events	kubectl describe job <job-name> -n <namespace> or kubectl describe pod <pod-name> -n <namespace>

For scripts and GitOps workflows, the command line remains the right tool. For a person asking "why is this waiting?", the linked Headlamp path is usually faster to read.

Optional: reproduce the pending scenario

The plugin repository includes sample Cohort, ClusterQueue, LocalQueue, ResourceFlavor, and Job manifests for local testing. Apply those examples to a cluster where Kueue is installed, then open the Kueue section in Headlamp.

To reproduce the specific quota mismatch used in this article, configure a ClusterQueue with two CPUs and two Pod slots, then submit a three-Pod Job that requests one CPU per Pod. The Job enters its LocalQueue, Kueue creates a Workload, and the Workload remains pending because the combined request is larger than the available quota.

Keeping this setup separate from the main walkthrough is intentional: once the resources exist, the investigation itself should happen in Headlamp.

Current alpha limitations

The 0.1.0-alpha release is an initial read-only view of the Kueue admission path.

• The plugin does not add its own create, edit, pause, resume, or delete flows.

• Workload owner references are displayed as text and are not currently navigable.

• The UI depends on the Kubernetes permissions of the identity used by Headlamp.

• Metrics and a visual relationship map are possible follow-up areas rather than features of this release.

These limits are useful to state clearly: the plugin's current purpose is to make admission state and resource relationships easier to inspect.

Developed during LFX Mentorship

The plugin was developed during CNCF LFX Mentorship 2026 Term 2 using TypeScript, React, Kubernetes APIs, and Headlamp's plugin system.

Utkarsh Raj implemented the plugin with guidance and review from René Dudfield (@illume) and Kevin Hannon (@kannon92). Feedback from the Headlamp and Kueue communities shaped the resource views, status presentation, navigation, documentation, tests, and first alpha release.

Try it and share feedback

Explore the plugin source and README, read the 0.1.0-alpha release documentation, or share feedback in the release issue.

If a pending Workload is still difficult to explain after following the Headlamp path, that is exactly the kind of feedback that can improve the plugin.

14 Oct 2026 6:35pm GMT