27 Oct 2026

feedKubernetes Blog

Announcing the Workload Conformance Working Group

We are excited to announce that the Kubernetes community now has a Workload Conformance Working Group. The working group is writing a shared standard for how workloads behave at runtime on Kubernetes. To get it right, we need input from the people who build those workloads and from the people who run the clusters they're deployed on.

Why workloads need a shared standard

Certified Kubernetes tests the cluster, and Kubernetes AI Conformance tests whether a cluster can run AI and ML workloads. Nothing comparable exists for the workloads themselves, so cluster operators still check each application by hand before routine maintenance or a Kubernetes upgrade.

The problems they find are familiar. A routine node drain is often how you discover that an application can't tolerate disruption. An upgrade can break an application that still depends on a removed API version. Others show up in day-to-day operation: workloads that ask for privileges they don't need, or reserve resources they never use.

Kubernetes already documents much of this, including security and handling disruptions. Even so, every cluster operator ends up repeating the same assessment, and application maintainers get the same questions from each environment they support. A common specification with repeatable tests would give both groups the same baseline to start from.

What the working group will deliver

The charter sets out three deliverables:

The first step is deciding which workloads and behaviors the specification covers. The scope includes security, resilience during disruptions, resource use, networking, storage, API usage, and observability, but the group hasn't yet agreed on specific requirements or on how to test them. However, one rule is already set: every requirement needs an objective test that a test binary can run, so nothing subjective goes into the specification.

The specification covers how a workload runs, and importantly not what it does. Consequently, application logic is out of scope of workload conformance. The binary won't retest anything Certified Kubernetes already covers, and it won't require a particular workload packaging format, or vendor tool like policy engine, security scanner, etc.

How the work is organized

Two Special Interest Groups (SIGs) are sponsors for the working group.

Whenever a requirement touches another Kubernetes SIG's area, such as node behavior or networking, the working group will initiate conversations with those SIGs as well.

The working group is a forum where SIGs can get involved and shape the direction of Workload Conformance while the specification is being written. We're aiming for a stable first specification and working tests within two to three Kubernetes release cycles.

Get involved

The specification will and needs to be built from real examples, so we want to hear your experiences. If you run clusters, tell us how you assess workloads today and what broke during your last upgrade or maintenance window. If you maintain an application, tell us what it needs from the cluster and where the existing guidance left you guessing.

You can find us in #wg-workload-conformance on Kubernetes Slack (visit slack.k8s.io for an invitation) and on the mailing list. The working group meets every other Wednesday at 15:00 UTC. The Zoom link and the agenda are in the meeting notes. If you want to raise a topic, add it to the agenda. Joining the mailing list usually adds the meeting invite to your calendar.

Thanks to everyone who reviewed and supported the working group proposal.

27 Oct 2026 4:30am GMT

14 Oct 2026

feedKubernetes Blog

Why Is My Kubernetes Job Still Pending? Tracing Kueue Admission with Headlamp

You submit a Kubernetes Job and wait for its Pods to appear. A minute passes. Then another. The Job is still suspended, there are no application logs, and there may not even be a Pod to inspect.

In a cluster managed by Kueue, this can be completely normal. Kueue may still be deciding whether the Job can use the requested resources. The useful question is not only "Why is the Pod pending?" It is first "Has Kueue admitted the Workload?"

The Kueue plugin for Headlamp brings that investigation into the Headlamp UI. Instead of reconstructing the admission path from several commands and large YAML objects, you can begin with the Workload, read its condition message, and follow links through the LocalQueue, ClusterQueue, ResourceFlavor, and Cohort.

This article follows that Headlamp-first path. The command line appears only in a short optional section for reproducing the example.

Before you begin

You need a Kubernetes cluster with Kueue installed, a Headlamp installation, and the 0.1.0-alpha Kueue plugin. Your Kubernetes identity must be allowed to read the Kueue resources used in this article.

The alpha release provides read-only views for Kueue v1beta2 Cohorts, ClusterQueues, LocalQueues, ResourceFlavors, and Workloads. It does not add its own create, edit, stop, resume, or delete workflows. Headlamp still provides its standard YAML and resource actions where your permissions allow them.

First, separate admission from Pod scheduling

Kueue and the Kubernetes scheduler answer different questions.

That boundary changes the investigation. If the Workload is not admitted, stay in the Kueue section of Headlamp. If it is admitted but its Pods are still Pending, move to Headlamp's normal Job and Pod views and inspect scheduler events.

A Kubernetes Job becomes a Kueue Workload. Kueue evaluates its LocalQueue, ClusterQueue, ResourceFlavor, and optional Cohort before resuming the Job. Headlamp reads these resources through the Kubernetes API.

Kueue admission architecture

The Headlamp-first troubleshooting path

The quickest investigation starts with the Workload, not with a shell.

A Headlamp troubleshooting flow that starts with Kueue Workloads, checks admission conditions, follows LocalQueue and ClusterQueue links when admission is blocked, and switches to Job and Pod views after admission.

Tracing a pending Job through Headlamp

Step 1: Open the Workloads view

In Headlamp, open Kueue → Workloads. Select the namespace that contains the Job, then find the Workload created for it.

Kueue creates a Workload to represent the resource request and admission state of a Job. In the alpha plugin, owner references are shown as Kind/name text but are not yet clickable. The Workload name normally contains the Job name, so the namespace filter and table search are the simplest ways to locate it.

Before opening the detail page, the list already gives you a useful first scan:

If there is no matching Workload, check three things before going further: the selected namespace, whether the Job was submitted to a LocalQueue, and whether your Kubernetes account can list Workloads. The plugin performs permission checks and displays an access message when the current identity cannot read a resource.

Step 2: Read status, conditions, and events

Open the Workload. Start with its current status and conditions, not with the full YAML.

For admission troubleshooting, the most important condition types are QuotaReserved and Admitted.

For example, a condition message may report that Kueue could not assign a ResourceFlavor because the ClusterQueue has insufficient CPU quota. That message is the first real answer to "Why is my Job still pending?" The rest of the Headlamp journey explains the configuration behind it.

Also check the Events section. Conditions describe the current state; events help show what recently changed or what Kueue attempted.

Step 3: Compare the Workload's request with its assignment

Stay on the Workload detail page and inspect the PodSets and resource requests. The plugin shows the requested Pod count, CPU, memory, and other resources that Kueue evaluates together.

This matters because a Job with parallelism: 3 and a one-CPU request per Pod asks Kueue for three CPUs in total. Looking at only one Pod's template would underestimate the Workload.

If quota has already been reserved, the same page also shows the assigned ClusterQueue, ResourceFlavors, and admitted resource usage. If those admission assignments are absent, the Workload has not reached that point yet.

In our example, the Workload requests:

The next question is where that request entered Kueue.

Step 4: Follow the linked LocalQueue

Use the LocalQueue link on the Workload detail page. A LocalQueue is the namespace-level entry point used by the Job. It connects namespaced Workloads to a cluster-scoped ClusterQueue.

The LocalQueue page shows:

This page answers two practical questions: Did the Job enter the queue you expected, and is it the only Workload waiting there?

If the LocalQueue is inactive or stopped, that may already explain the wait. If it is active, follow the linked ClusterQueue to inspect the capacity behind it.

Step 5: Inspect ClusterQueue capacity and usage

A ClusterQueue represents the cluster-level quota and admission policy available to one or more LocalQueues.

On its Headlamp detail page, compare the Workload request with:

For the example in this article, the ClusterQueue provides a nominal quota of two CPUs and two Pod slots. The Workload asks for three of each. The condition message and the ClusterQueue page now tell the same story: the Workload does not fit, so Kueue leaves the Job suspended.

This is an admission-capacity problem. It is not evidence that kube-scheduler failed to find a node.

Step 6: Check ResourceFlavor and Cohort when needed

Quota is not always a single pool. A ResourceFlavor can represent a class of compute through node labels, taints, tolerations, or topology settings. Use the ResourceFlavor link from the ClusterQueue when the condition mentions flavor assignment or when the Workload requires a particular hardware class, such as GPU nodes.

If the ClusterQueue belongs to a Cohort, follow that link too. Cohorts let ClusterQueues share unused quota. The Headlamp page shows parent and child Cohorts, member ClusterQueues, referenced ResourceFlavors, resource groups, borrowing and lending limits, and fair-sharing information.

These pages help answer a more subtle question: Is the Workload blocked because the queue has no quota of its own, because it cannot borrow enough, or because no suitable flavor can be assigned?

What we learned without leaving Headlamp

For the sample pending Job, the UI gives us a complete explanation:

  1. The Workload is not admitted.

  2. Its QuotaReserved condition is false and the message points to insufficient quota.

  3. The Workload requests three CPUs and three Pod slots.

  4. Its LocalQueue is active and points to the expected ClusterQueue.

  5. The ClusterQueue provides only two CPUs and two Pod slots.

  6. Kueue therefore keeps the Job suspended until quota becomes available or the queue configuration changes.

The important part is not that Headlamp invents a new status. It reads the same Kubernetes API objects as other clients, then places the related resources and human-readable values in one connected investigation.

If the Workload is admitted, switch views

Suppose the Workload detail page shows Admitted=True, but the application is still not running. Kueue has completed its part of the decision. Continue in Headlamp's standard Kubernetes views:

  1. Open Workloads → Jobs and inspect the Job state and events.

  2. Open Workloads → Pods and filter by the Job name or namespace.

  3. Select a Pending Pod and read its Events section.

At this stage, look for scheduler concerns such as insufficient node capacity, node affinity, taints and tolerations, topology constraints, persistent-volume binding, or image-pull failures.

This handoff is the most useful troubleshooting rule in the article:

A repeatable Headlamp checklist

When the next Job does not start, use this short path:

  1. Open Kueue → Workloads and select the correct namespace.

  2. Find the Workload associated with the Job.

  3. Check the readable status, Admitted, and QuotaReserved conditions.

  4. Read the condition message and recent events.

  5. Compare PodSet resource requests with any admission assignments.

  6. Follow the LocalQueue link and confirm its status and ClusterQueue.

  7. Follow the ClusterQueue link and compare quota, reservations, usage, and policy.

  8. Inspect ResourceFlavor or Cohort details when the message points to flavor selection or shared quota.

  9. If admitted, move to Headlamp's Job and Pod views.

What Headlamp replaces during this investigation

The plugin does not replace kubectl or kueuectl for scripts, exact object output, or changes. It replaces the repetitive UI-less navigation needed during an interactive investigation.

Question Headlamp path Command-line equivalent
Is the Workload admitted? Kueue → Workloads → status and conditions kubectl describe workload <workload-name> -n <namespace>
Which LocalQueue did it enter? Workload → linked LocalQueue kubectl get localqueue <localqueue-name> -n <namespace> -o yaml
Which ClusterQueue provides quota? LocalQueue → linked ClusterQueue kubectl get clusterqueue <clusterqueue-name> -o yaml
Which flavor or shared quota matters? ClusterQueue → ResourceFlavor or Cohort kubectl get resourceflavor <flavor-name> -o yaml or kubectl get cohort <cohort-name> -o yaml
What happened after admission? Workloads → Jobs and Pods → events kubectl describe job <job-name> -n <namespace> or kubectl describe pod <pod-name> -n <namespace>

For scripts and GitOps workflows, the command line remains the right tool. For a person asking "why is this waiting?", the linked Headlamp path is usually faster to read.

Optional: reproduce the pending scenario

The plugin repository includes sample Cohort, ClusterQueue, LocalQueue, ResourceFlavor, and Job manifests for local testing. Apply those examples to a cluster where Kueue is installed, then open the Kueue section in Headlamp.

To reproduce the specific quota mismatch used in this article, configure a ClusterQueue with two CPUs and two Pod slots, then submit a three-Pod Job that requests one CPU per Pod. The Job enters its LocalQueue, Kueue creates a Workload, and the Workload remains pending because the combined request is larger than the available quota.

Keeping this setup separate from the main walkthrough is intentional: once the resources exist, the investigation itself should happen in Headlamp.

Current alpha limitations

The 0.1.0-alpha release is an initial read-only view of the Kueue admission path.

These limits are useful to state clearly: the plugin's current purpose is to make admission state and resource relationships easier to inspect.

Developed during LFX Mentorship

The plugin was developed during CNCF LFX Mentorship 2026 Term 2 using TypeScript, React, Kubernetes APIs, and Headlamp's plugin system.

Utkarsh Raj implemented the plugin with guidance and review from René Dudfield (@illume) and Kevin Hannon (@kannon92). Feedback from the Headlamp and Kueue communities shaped the resource views, status presentation, navigation, documentation, tests, and first alpha release.

Try it and share feedback

Explore the plugin source and README, read the 0.1.0-alpha release documentation, or share feedback in the release issue.

If a pending Workload is still difficult to explain after following the Headlamp path, that is exactly the kind of feedback that can improve the plugin.

14 Oct 2026 6:35pm GMT

06 Oct 2026

feedKubernetes Blog

The Shift to cgroup v2 in Kubernetes: What You Need to Know

In Linux, cgroups (control groups) are a kernel feature used for managing system resources. Kubernetes uses cgroups to allocate resources like CPU and memory to containers, ensuring that applications run smoothly without interfering with each other. With the release of Kubernetes v1.31, support for v1 cgroup management moved into maintenance mode. Support for v2 cgroup management has been stable since Kubernetes v1.25.

Compared with cgroup v1, cgroup v2 provides a single unified hierarchy, a more consistent interface, and a stronger foundation for resource isolation and modern resource-management features.

Deprecation of cgroup v1

Kubernetes has deprecated cgroup v1. Starting with Kubernetes v1.35, failCgroupV1 defaults to true, so the kubelet does not start on a cgroup v1 node by default. Administrators can temporarily set failCgroupV1: false in the kubelet configuration file, but removal will follow the Kubernetes deprecation policy. Further removal work is tracked in KEP-5573: Remove cgroup v1 support.

If you are still on a release older than v1.35, migrate every Linux node to cgroup v2 before upgrading, or plan to set the temporary failCgroupV1: false override. If you are already on v1.35 or later, confirm that every Linux node runs cgroup v2 (or that you intentionally keep the override). Under the default configuration, a remaining cgroup v1 node fails during kubelet startup.

For kubeadm-managed clusters, Kubernetes v1.35 also makes this an earlier, stricter check. The SystemVerification preflight check, provided by k8s.io/system-validators, returns an error during kubeadm init, kubeadm join, and kubeadm upgrade when it detects cgroup v1 with kubelet v1.35 or later; with an older kubelet, the check remains a warning. See kubernetes/system-validators#1.12.1 release notes for details.

The top FAQs cover three main areas: why to migrate, the benefits and drawbacks, and key points to keep in mind when using cgroup v2.

Limitations of cgroup v1 and Improvements with cgroup v2

The Linux kernel documentation describes both interfaces:

Let's enumerate some known issues.

active_file memory is not considered available memory

The kubelet treats active_file memory as not reclaimable. For I/O-intensive workloads, a large page cache can therefore make the kubelet report memory pressure and evict Pods. This is a known kubelet issue (kubernetes/kubernetes#43916); migrating to cgroup v2 does not by itself change that calculation. The documented workaround is to set equal memory requests and limits for containers that perform intensive I/O, after measuring an appropriate value.

Memory QoS updates in Kubernetes v1.36

Memory QoS was introduced as an alpha feature in Kubernetes v1.22 and updated in v1.27. It remains alpha in v1.36, but now separates memory throttling from memory reservation and adds tiered memory protection:

Memory QoS is available only on Linux nodes that use cgroup v2. It relies on the cgroup v2 memory controller: memory.high provides throttling, while memory.min and memory.low provide hard and soft protection when tiered reservation is enabled. cgroup v1 cannot provide this protection model.

For example, to opt in to tiered protection:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
 MemoryQoS: true
memoryReservationPolicy: TieredReservation
memoryThrottlingFactor: 0.9

The overall Kubernetes recommendation is not to enable Alpha features in production; however, if you judge that memory QoS with tiered reservations is useful for your platform in production, make sure to test the configuration and account for hard-reserved memory before you enable the TieredReservation feature gate. See the Pod QoS documentation for the current mapping of Kubernetes QoS classes to cgroup v2 controls.

Container-aware OOM handling

On cgroup v2 nodes, the kubelet defaults singleProcessOOMKill to false. It therefore sets memory.oom.group for each container cgroup so that an OOM event kills all processes in that container together, rather than leaving a partially functioning multi-process container. Set singleProcessOOMKill: true only if you need the cgroup v1-compatible behavior where the kernel may kill one process at a time. See the KubeletConfiguration reference for this setting.

This behavior is scoped to a container cgroup, not the whole Pod. Also, cgroup.kill is a separate administrative interface: writing 1 to it sends SIGKILL to every process in that cgroup and its descendants; it does not configure OOM behavior. The cgroup v2 memory controller additionally provides memory.events counters that monitoring systems and userspace OOM managers can observe.

Rootless support

In cgroup v1, delegating controllers to less privileged containers may be dangerous.

Unlike cgroup v1, cgroup v2 officially supports delegation. Most implementations of rootless containers rely on systemd for delegating v2 controllers to non-root users.

This delegation mechanism is separate from Kubernetes Pod user namespaces, which map container users to unprivileged host users. Pod user namespace support graduated to stable in Kubernetes v1.36; check its filesystem, kernel, CRI runtime, and OCI runtime prerequisites before enabling it.

What else?

  1. eBPF stories:
    • In cgroup v1, device access controls are exposed through interface files.
    • The cgroup v2 device controller has no interface files, and is implemented on top of cgroup BPF.
    • Cilium attaches BPF cgroup programs for socket-based load balancing. Its default cgroup root is /run/cilium/cgroupv2.
  2. Pressure Stall Information (PSI) reports CPU, memory, and I/O contention at node, Pod, and container level. On supported clusters, the kubelet exposes PSI by default (KubeletPSI is stable and locked on). PSI requires cgroup v2, Linux 4.20 or later, CONFIG_PSI=y, and a kernel not booted with psi=0. The kubelet surfaces the data through the Summary API and /metrics/cadvisor.
  3. When migrating, update software that reads the cgroup filesystem directly. The migration guidance recommends cAdvisor v0.43.0 or later and lists compatible Java, Node.js, and automaxprocs versions.

CPU weight conversion in newer OCI runtimes

cgroup v1 uses cpu.shares, whereas cgroup v2 uses cpu.weight. Newer OCI runtimes use an improved non-linear conversion that preserves the default priority and gives small CPU requests more usable granularity. The change is implemented in the OCI runtime rather than Kubernetes: it is available in crun v1.23 and runc v1.3.2. After upgrading a runtime, monitoring or policy tools that predict exact cpu.weight values may need updates. Read New Conversion from cgroup v1 CPU Shares to v2 CPU Weight for the formula, examples, and compatibility considerations.

In-place resource updates

In-place Pod vertical scaling graduated to stable in Kubernetes v1.35. Kubernetes v1.36 then [enabled] (/blog/2026/04/30/kubernetes-v1-36-inplace-pod-level-resources-beta/) in-place vertical scaling for Pod-level resources by default, as a Beta feature. The kubelet coordinates changes between the Pod-level and container cgroups so that increases create headroom before container limits grow, while decreases constrain containers before shrinking the Pod-level boundary. Accurate aggregate enforcement for this v1.36 feature requires cgroup v2.

Adopting cgroup version 2

Requirements

Here's what you need to use cgroup v2 with Kubernetes. First up, you need to be using a version of Kubernetes with support for v2 cgroup management; that's been stable since Kubernetes v1.25 and all supported Kubernetes releases include this support.

For now, you can opt back in to use cgroup v1; the Kubernetes project recommends using cgroup v2, but in Kubernetes 1.36 (the current release) the cgroup v1 option remains supported as a fallback. That fallback is scheduled for removal in Kubernetes v1.38. If you are running an older cluster, plan to migrate; if you are setting up a new cluster with Linux nodes, you should prefer cgroup v2. In either case, review both the Kubernetes runtime documentation and (if relevant) the containerd compatibility matrix.

kernel updates around cgroup v2

When Kubernetes was first announced, in 2014, only v1 cgroup existed. Version 2 cgroup management first appeared in Linux kernel 4.5, released in 2016.

cgroup driver configuration

Configure the kubelet's cgroup driver to match the container runtime cgroup driver.

If you use kubeadm to manage your cluster, Kubernetes recommends that you use the systemd cgroup driver, because kubeadm manages the kubelet as a systemd service. For other management tooling, check the documentation for the tool you're using to manage your cluster.

If you can pick either option, I recommend using the systemd driver.

Whatever tooling you've chosen, the kubelet automatically tries to detect the runtime's recommended cgroup driver. This automatic detection relies on using a runtime that implements the RuntimeConfig CRI RPC (for example: containerd v2.0+ or CRI-O v1.28+).

If you're using a container runtime that supports cgroup v2 but doesn't support automatic cgroup driver detection, you can manually configure an override by editing the kubelet configuration file. For example:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

Automatic discovery of the runtime's cgroup driver through the CRI, tracked by KEP-4033, graduated to stable in Kubernetes v1.34. It requires a runtime that implements the RuntimeConfig CRI RPC (containerd v2.0+ or CRI-O v1.28+). When available, the kubelet uses the value reported by the runtime instead of its configured cgroupDriver value.

Tools and commands for troubleshooting

Tools and commands that you should know about cgroups:

How to check if a Pod CPU or memory limit is successfully applied to the cgroup file?

Work from the API object down to the node. Identify the node, then compare desired resources in the spec with the enacted values in status (in-place resize can leave those out of sync):

kubectl get pod <pod-name> -n <namespace> \
 -o jsonpath='{.spec.nodeName}{"\n"}'

kubectl get pod <pod-name> -n <namespace> \
 -o jsonpath='{range .spec.containers[*]}{.name}{" spec\t"}{.resources}{"\n"}{end}'

kubectl get pod <pod-name> -n <namespace> \
 -o jsonpath='{range .status.containerStatuses[*]}{.name}{" status\t"}{.resources}{"\n"}{end}'

On that node, as root, with the systemd cgroup driver and cgroup v2 (these snippets also need jq). Pin the container with kubelet labels; a container name alone is not unique on a node. Use crictl ps -a if the container is not running:

CONTAINER_ID=$(crictl ps \
 --label io.kubernetes.pod.namespace=<namespace> \
 --label io.kubernetes.pod.name=<pod-name> \
 --name <container-name> -q | head -n1)

# OCI view (not the CRI protobuf field names)
crictl inspect "$CONTAINER_ID" | jq '.info.runtimeSpec.linux.resources'

# Kernel cgroup path for this container
PID=$(crictl inspect "$CONTAINER_ID" | jq -r '.info.pid')
CGROUP="/sys/fs/cgroup$(awk -F: '$1=="0"{print $3}' /proc/$PID/cgroup)"

cat "$CGROUP/cpu.weight" # request.cpu → shares → weight; not a limit
cat "$CGROUP/cpu.max" # limit.cpu as "quota period"; unlimited is "max <period>"
cat "$CGROUP/memory.max" # limit.memory; unlimited is "max"

# Pod-level cgroup (parent slice), used by Pod-level resources
cat "$(dirname "$CGROUP")/cpu.max" "$(dirname "$CGROUP")/memory.max"

# Present when MemoryQoS is enabled
cat "$CGROUP/memory.high" "$CGROUP/memory.min" "$CGROUP/memory.low"

SCOPE=$(basename "$CGROUP")
systemctl show "$SCOPE" \
 -p CPUWeight -p CPUQuotaPerSecUSec -p CPUQuotaPeriodUSec -p MemoryMax

Do not treat info.runtimeSpec.linux.cgroupsPath as a filesystem path when the runtime uses the systemd driver; that value is a systemd unit path (slice:runtime:id), not a directory under /sys/fs/cgroup.

Expected mapping at each layer:

Further reading

06 Oct 2026 6:00pm GMT