fidzu : Kubernetes

You submit a Kubernetes Job and wait for its Pods to appear. A minute passes. Then another. The Job is still suspended, there are no application logs, and there may not even be a Pod to inspect.

In a cluster managed by Kueue, this can be completely normal. Kueue may still be deciding whether the Job can use the requested resources. The useful question is not only "Why is the Pod pending?" It is first "Has Kueue admitted the Workload?"

The Kueue plugin for Headlamp brings that investigation into the Headlamp UI. Instead of reconstructing the admission path from several commands and large YAML objects, you can begin with the Workload, read its condition message, and follow links through the LocalQueue, ClusterQueue, ResourceFlavor, and Cohort.

This article follows that Headlamp-first path. The command line appears only in a short optional section for reproducing the example.

Before you begin

You need a Kubernetes cluster with Kueue installed, a Headlamp installation, and the 0.1.0-alpha Kueue plugin. Your Kubernetes identity must be allowed to read the Kueue resources used in this article.

The alpha release provides read-only views for Kueue v1beta2 Cohorts, ClusterQueues, LocalQueues, ResourceFlavors, and Workloads. It does not add its own create, edit, stop, resume, or delete workflows. Headlamp still provides its standard YAML and resource actions where your permissions allow them.

First, separate admission from Pod scheduling

Kueue and the Kubernetes scheduler answer different questions.

• Kueue admission: Should this Workload start now, considering queue policy, quota, ResourceFlavors, admission checks, and other waiting Workloads?

• Pod scheduling: After the Workload is admitted and the Job resumes, which nodes can run its Pods?

That boundary changes the investigation. If the Workload is not admitted, stay in the Kueue section of Headlamp. If it is admitted but its Pods are still Pending, move to Headlamp's normal Job and Pod views and inspect scheduler events.

The Headlamp-first troubleshooting path

The quickest investigation starts with the Workload, not with a shell.

Step 1: Open the Workloads view

In Headlamp, open Kueue → Workloads. Select the namespace that contains the Job, then find the Workload created for it.

Kueue creates a Workload to represent the resource request and admission state of a Job. In the alpha plugin, owner references are shown as Kind/name text but are not yet clickable. The Workload name normally contains the Job name, so the namespace filter and table search are the simplest ways to locate it.

Before opening the detail page, the list already gives you a useful first scan:

• LocalQueue

• priority and priority class

• active state

• admitted state

• completion state

• a readable current status such as Admitted, Finished, or Inadmissible

• age

If there is no matching Workload, check three things before going further: the selected namespace, whether the Job was submitted to a LocalQueue, and whether your Kubernetes account can list Workloads. The plugin performs permission checks and displays an access message when the current identity cannot read a resource.

Step 2: Read status, conditions, and events

Open the Workload. Start with its current status and conditions, not with the full YAML.

For admission troubleshooting, the most important condition types are QuotaReserved and Admitted.

• Admitted=True means Kueue has admitted the Workload. The Job can resume, so any remaining Pending Pods belong to the scheduling branch of the investigation.

• QuotaReserved=False with a reason such as Pending means Kueue has not reserved quota. The condition message explains what prevented the reservation.

• An Inadmissible status means the Workload currently cannot fit the available queue configuration. A missing covered resource, unsuitable ResourceFlavor, stopped queue, or admission check can all lead you here.

For example, a condition message may report that Kueue could not assign a ResourceFlavor because the ClusterQueue has insufficient CPU quota. That message is the first real answer to "Why is my Job still pending?" The rest of the Headlamp journey explains the configuration behind it.

Also check the Events section. Conditions describe the current state; events help show what recently changed or what Kueue attempted.

Step 3: Compare the Workload's request with its assignment

Stay on the Workload detail page and inspect the PodSets and resource requests. The plugin shows the requested Pod count, CPU, memory, and other resources that Kueue evaluates together.

This matters because a Job with parallelism: 3 and a one-CPU request per Pod asks Kueue for three CPUs in total. Looking at only one Pod's template would underestimate the Workload.

If quota has already been reserved, the same page also shows the assigned ClusterQueue, ResourceFlavors, and admitted resource usage. If those admission assignments are absent, the Workload has not reached that point yet.

In our example, the Workload requests:

• three Pods

• three CPUs in total

• 600 MiB of memory in total

The next question is where that request entered Kueue.

Step 4: Follow the linked LocalQueue

Use the LocalQueue link on the Workload detail page. A LocalQueue is the namespace-level entry point used by the Job. It connects namespaced Workloads to a cluster-scoped ClusterQueue.

The LocalQueue page shows:

• its namespace and connected ClusterQueue

• stop policy and current status

• pending, reserving, and admitted Workload counts

• Kueue conditions and Kubernetes events

• related Workloads in the same namespace

This page answers two practical questions: Did the Job enter the queue you expected, and is it the only Workload waiting there?

If the LocalQueue is inactive or stopped, that may already explain the wait. If it is active, follow the linked ClusterQueue to inspect the capacity behind it.

Step 5: Inspect ClusterQueue capacity and usage

A ClusterQueue represents the cluster-level quota and admission policy available to one or more LocalQueues.

On its Headlamp detail page, compare the Workload request with:

• covered resources

• nominal quota for each ResourceFlavor

• current reservations and usage

• borrowing and lending limits

• pending, reserving, and admitted Workload counts

• queueing strategy and stop policy

• preemption, fair-sharing, flavor-fungibility, and admission-check settings

• Kueue conditions and Kubernetes events

For the example in this article, the ClusterQueue provides a nominal quota of two CPUs and two Pod slots. The Workload asks for three of each. The condition message and the ClusterQueue page now tell the same story: the Workload does not fit, so Kueue leaves the Job suspended.

This is an admission-capacity problem. It is not evidence that kube-scheduler failed to find a node.

Step 6: Check ResourceFlavor and Cohort when needed

Quota is not always a single pool. A ResourceFlavor can represent a class of compute through node labels, taints, tolerations, or topology settings. Use the ResourceFlavor link from the ClusterQueue when the condition mentions flavor assignment or when the Workload requires a particular hardware class, such as GPU nodes.

If the ClusterQueue belongs to a Cohort, follow that link too. Cohorts let ClusterQueues share unused quota. The Headlamp page shows parent and child Cohorts, member ClusterQueues, referenced ResourceFlavors, resource groups, borrowing and lending limits, and fair-sharing information.

These pages help answer a more subtle question: Is the Workload blocked because the queue has no quota of its own, because it cannot borrow enough, or because no suitable flavor can be assigned?

What we learned without leaving Headlamp

For the sample pending Job, the UI gives us a complete explanation:

1. The Workload is not admitted.

2. Its QuotaReserved condition is false and the message points to insufficient quota.

3. The Workload requests three CPUs and three Pod slots.

4. Its LocalQueue is active and points to the expected ClusterQueue.

5. The ClusterQueue provides only two CPUs and two Pod slots.

6. Kueue therefore keeps the Job suspended until quota becomes available or the queue configuration changes.

The important part is not that Headlamp invents a new status. It reads the same Kubernetes API objects as other clients, then places the related resources and human-readable values in one connected investigation.

If the Workload is admitted, switch views

Suppose the Workload detail page shows Admitted=True, but the application is still not running. Kueue has completed its part of the decision. Continue in Headlamp's standard Kubernetes views:

1. Open Workloads → Jobs and inspect the Job state and events.

2. Open Workloads → Pods and filter by the Job name or namespace.

3. Select a Pending Pod and read its Events section.

At this stage, look for scheduler concerns such as insufficient node capacity, node affinity, taints and tolerations, topology constraints, persistent-volume binding, or image-pull failures.

This handoff is the most useful troubleshooting rule in the article:

• Not admitted: investigate Kueue resources in Headlamp.

• Admitted, but Pods are Pending: investigate Job, Pod, node, and scheduler information in Headlamp.

A repeatable Headlamp checklist

When the next Job does not start, use this short path:

1. Open Kueue → Workloads and select the correct namespace.

2. Find the Workload associated with the Job.

3. Check the readable status, Admitted, and QuotaReserved conditions.

4. Read the condition message and recent events.

5. Compare PodSet resource requests with any admission assignments.

6. Follow the LocalQueue link and confirm its status and ClusterQueue.

7. Follow the ClusterQueue link and compare quota, reservations, usage, and policy.

8. Inspect ResourceFlavor or Cohort details when the message points to flavor selection or shared quota.

9. If admitted, move to Headlamp's Job and Pod views.

What Headlamp replaces during this investigation

The plugin does not replace kubectl or kueuectl for scripts, exact object output, or changes. It replaces the repetitive UI-less navigation needed during an interactive investigation.

For scripts and GitOps workflows, the command line remains the right tool. For a person asking "why is this waiting?", the linked Headlamp path is usually faster to read.

Optional: reproduce the pending scenario

The plugin repository includes sample Cohort, ClusterQueue, LocalQueue, ResourceFlavor, and Job manifests for local testing. Apply those examples to a cluster where Kueue is installed, then open the Kueue section in Headlamp.

To reproduce the specific quota mismatch used in this article, configure a ClusterQueue with two CPUs and two Pod slots, then submit a three-Pod Job that requests one CPU per Pod. The Job enters its LocalQueue, Kueue creates a Workload, and the Workload remains pending because the combined request is larger than the available quota.

Keeping this setup separate from the main walkthrough is intentional: once the resources exist, the investigation itself should happen in Headlamp.

Current alpha limitations

The 0.1.0-alpha release is an initial read-only view of the Kueue admission path.

• The plugin does not add its own create, edit, pause, resume, or delete flows.

• Workload owner references are displayed as text and are not currently navigable.

• The UI depends on the Kubernetes permissions of the identity used by Headlamp.

• Metrics and a visual relationship map are possible follow-up areas rather than features of this release.

These limits are useful to state clearly: the plugin's current purpose is to make admission state and resource relationships easier to inspect.

Developed during LFX Mentorship

The plugin was developed during CNCF LFX Mentorship 2026 Term 2 using TypeScript, React, Kubernetes APIs, and Headlamp's plugin system.

Utkarsh Raj implemented the plugin with guidance and review from René Dudfield (@illume) and Kevin Hannon (@kannon92). Feedback from the Headlamp and Kueue communities shaped the resource views, status presentation, navigation, documentation, tests, and first alpha release.

Try it and share feedback

Explore the plugin source and README, read the 0.1.0-alpha release documentation, or share feedback in the release issue.

If a pending Workload is still difficult to explain after following the Headlamp path, that is exactly the kind of feedback that can improve the plugin.

14 Oct 2026 6:35pm GMT

In Linux, cgroups (control groups) are a kernel feature used for managing system resources. Kubernetes uses cgroups to allocate resources like CPU and memory to containers, ensuring that applications run smoothly without interfering with each other. With the release of Kubernetes v1.31, support for v1 cgroup management moved into maintenance mode. Support for v2 cgroup management has been stable since Kubernetes v1.25.

Compared with cgroup v1, cgroup v2 provides a single unified hierarchy, a more consistent interface, and a stronger foundation for resource isolation and modern resource-management features.

Deprecation of cgroup v1

Kubernetes has deprecated cgroup v1. Starting with Kubernetes v1.35, failCgroupV1 defaults to true, so the kubelet does not start on a cgroup v1 node by default. Administrators can temporarily set failCgroupV1: false in the kubelet configuration file, but removal will follow the Kubernetes deprecation policy. Further removal work is tracked in KEP-5573: Remove cgroup v1 support.

If you are still on a release older than v1.35, migrate every Linux node to cgroup v2 before upgrading, or plan to set the temporary failCgroupV1: false override. If you are already on v1.35 or later, confirm that every Linux node runs cgroup v2 (or that you intentionally keep the override). Under the default configuration, a remaining cgroup v1 node fails during kubelet startup.

For kubeadm-managed clusters, Kubernetes v1.35 also makes this an earlier, stricter check. The SystemVerification preflight check, provided by k8s.io/system-validators, returns an error during kubeadm init, kubeadm join, and kubeadm upgrade when it detects cgroup v1 with kubelet v1.35 or later; with an older kubelet, the check remains a warning. See kubernetes/system-validators#1.12.1 release notes for details.

The top FAQs cover three main areas: why to migrate, the benefits and drawbacks, and key points to keep in mind when using cgroup v2.

Limitations of cgroup v1 and Improvements with cgroup v2

The Linux kernel documentation describes both interfaces:

• cgroup v1 documentation
• cgroup v2 documentation

Let's enumerate some known issues.

active_file memory is not considered available memory

The kubelet treats active_file memory as not reclaimable. For I/O-intensive workloads, a large page cache can therefore make the kubelet report memory pressure and evict Pods. This is a known kubelet issue (kubernetes/kubernetes#43916); migrating to cgroup v2 does not by itself change that calculation. The documented workaround is to set equal memory requests and limits for containers that perform intensive I/O, after measuring an appropriate value.

Memory QoS updates in Kubernetes v1.36

Memory QoS was introduced as an alpha feature in Kubernetes v1.22 and updated in v1.27. It remains alpha in v1.36, but now separates memory throttling from memory reservation and adds tiered memory protection:

Memory QoS is available only on Linux nodes that use cgroup v2. It relies on the cgroup v2 memory controller: memory.high provides throttling, while memory.min and memory.low provide hard and soft protection when tiered reservation is enabled. cgroup v1 cannot provide this protection model.

• Enabling the MemoryQoS feature gate applies memory.high throttling to Burstable containers. The threshold is derived from the request, limit, and memoryThrottlingFactor (default 0.9).

• memoryReservationPolicy: None is the default. It does not write memory.min or memory.low.

• memoryReservationPolicy: TieredReservation maps Guaranteed Pod memory requests to memory.min (hard protection) and Burstable Pod requests to memory.low (soft protection). BestEffort Pods receive neither protection.

• The kubelet exposes Alpha metrics for the total memory.min and memory.low reservations on a node.

• Kernel 5.9 or later is recommended. On older kernels, memory.high reclaim can trigger a known livelock; from v1.36 the kubelet logs a warning when Memory QoS is enabled on an affected kernel.

For example, to opt in to tiered protection:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
 MemoryQoS: true
memoryReservationPolicy: TieredReservation
memoryThrottlingFactor: 0.9

The overall Kubernetes recommendation is not to enable Alpha features in production; however, if you judge that memory QoS with tiered reservations is useful for your platform in production, make sure to test the configuration and account for hard-reserved memory before you enable the TieredReservation feature gate. See the Pod QoS documentation for the current mapping of Kubernetes QoS classes to cgroup v2 controls.

Container-aware OOM handling

On cgroup v2 nodes, the kubelet defaults singleProcessOOMKill to false. It therefore sets memory.oom.group for each container cgroup so that an OOM event kills all processes in that container together, rather than leaving a partially functioning multi-process container. Set singleProcessOOMKill: true only if you need the cgroup v1-compatible behavior where the kernel may kill one process at a time. See the KubeletConfiguration reference for this setting.

This behavior is scoped to a container cgroup, not the whole Pod. Also, cgroup.kill is a separate administrative interface: writing 1 to it sends SIGKILL to every process in that cgroup and its descendants; it does not configure OOM behavior. The cgroup v2 memory controller additionally provides memory.events counters that monitoring systems and userspace OOM managers can observe.

Rootless support

In cgroup v1, delegating controllers to less privileged containers may be dangerous.

Unlike cgroup v1, cgroup v2 officially supports delegation. Most implementations of rootless containers rely on systemd for delegating v2 controllers to non-root users.

This delegation mechanism is separate from Kubernetes Pod user namespaces, which map container users to unprivileged host users. Pod user namespace support graduated to stable in Kubernetes v1.36; check its filesystem, kernel, CRI runtime, and OCI runtime prerequisites before enabling it.

What else?

1. eBPF stories:
   • In cgroup v1, device access controls are exposed through interface files.
   • The cgroup v2 device controller has no interface files, and is implemented on top of cgroup BPF.
   • Cilium attaches BPF cgroup programs for socket-based load balancing. Its default cgroup root is /run/cilium/cgroupv2.
2. Pressure Stall Information (PSI) reports CPU, memory, and I/O contention at node, Pod, and container level. On supported clusters, the kubelet exposes PSI by default (KubeletPSI is stable and locked on). PSI requires cgroup v2, Linux 4.20 or later, CONFIG_PSI=y, and a kernel not booted with psi=0. The kubelet surfaces the data through the Summary API and /metrics/cadvisor.
3. When migrating, update software that reads the cgroup filesystem directly. The migration guidance recommends cAdvisor v0.43.0 or later and lists compatible Java, Node.js, and automaxprocs versions.

CPU weight conversion in newer OCI runtimes

cgroup v1 uses cpu.shares, whereas cgroup v2 uses cpu.weight. Newer OCI runtimes use an improved non-linear conversion that preserves the default priority and gives small CPU requests more usable granularity. The change is implemented in the OCI runtime rather than Kubernetes: it is available in crun v1.23 and runc v1.3.2. After upgrading a runtime, monitoring or policy tools that predict exact cpu.weight values may need updates. Read New Conversion from cgroup v1 CPU Shares to v2 CPU Weight for the formula, examples, and compatibility considerations.

In-place resource updates

In-place Pod vertical scaling graduated to stable in Kubernetes v1.35. Kubernetes v1.36 then [enabled] (/blog/2026/04/30/kubernetes-v1-36-inplace-pod-level-resources-beta/) in-place vertical scaling for Pod-level resources by default, as a Beta feature. The kubelet coordinates changes between the Pod-level and container cgroups so that increases create headroom before container limits grow, while decreases constrain containers before shrinking the Pod-level boundary. Accurate aggregate enforcement for this v1.36 feature requires cgroup v2.

Adopting cgroup version 2

Requirements

Here's what you need to use cgroup v2 with Kubernetes. First up, you need to be using a version of Kubernetes with support for v2 cgroup management; that's been stable since Kubernetes v1.25 and all supported Kubernetes releases include this support.

• You need at least one Linux node; cgroup is a Linux-only concept
• Your OS install must run with cgroup v2 enabled
• The kernel version must be 5.8 or later (5.9 or later is recommended when using memory QoS)
• The container runtime must support cgroup v2. For example:
  • containerd v1.4 or later supports cgroup v2; use containerd v2.0 or later for automatic cgroup-driver discovery
  • CRI-O v1.20 or later
• The kubelet and the container runtime must both be configured to use the correct cgroup driver. See Configure the kubelet's cgroup driver to match the container runtime cgroup driver.

For now, you can opt back in to use cgroup v1; the Kubernetes project recommends using cgroup v2, but in Kubernetes 1.36 (the current release) the cgroup v1 option remains supported as a fallback. That fallback is scheduled for removal in Kubernetes v1.38. If you are running an older cluster, plan to migrate; if you are setting up a new cluster with Linux nodes, you should prefer cgroup v2. In either case, review both the Kubernetes runtime documentation and (if relevant) the containerd compatibility matrix.

kernel updates around cgroup v2

When Kubernetes was first announced, in 2014, only v1 cgroup existed. Version 2 cgroup management first appeared in Linux kernel 4.5, released in 2016.

• In Linux 4.5, the cgroup v2 io, memory, and pids controllers were supported.
• Linux 4.15 added support for the cgroup v2 cpu controller.
• Pressure Stall Information (PSI) support began with Linux 4.20.
• The Kubernetes project does not recommend using cgroup v2 with a Linux kernel older than 5.2 due to lack of cgroup-level task freezer support.
• Kubernetes documents 5.8 as the minimum kernel version for cgroup v2; the root cgroup's system-level cpu.stat file was added in Linux 5.8.
• The memory.high livelock fix used by Memory QoS is present in Linux 5.9 and later.
• memory.peak was added in Linux 5.19.

cgroup driver configuration

Configure the kubelet's cgroup driver to match the container runtime cgroup driver.

If you use kubeadm to manage your cluster, Kubernetes recommends that you use the systemd cgroup driver, because kubeadm manages the kubelet as a systemd service. For other management tooling, check the documentation for the tool you're using to manage your cluster.

If you can pick either option, I recommend using the systemd driver.

Whatever tooling you've chosen, the kubelet automatically tries to detect the runtime's recommended cgroup driver. This automatic detection relies on using a runtime that implements the RuntimeConfig CRI RPC (for example: containerd v2.0+ or CRI-O v1.28+).

If you're using a container runtime that supports cgroup v2 but doesn't support automatic cgroup driver detection, you can manually configure an override by editing the kubelet configuration file. For example:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

Automatic discovery of the runtime's cgroup driver through the CRI, tracked by KEP-4033, graduated to stable in Kubernetes v1.34. It requires a runtime that implements the RuntimeConfig CRI RPC (containerd v2.0+ or CRI-O v1.28+). When available, the kubelet uses the value reported by the runtime instead of its configured cgroupDriver value.

Tools and commands for troubleshooting

Tools and commands that you should know about cgroups:

• stat -fc %T /sys/fs/cgroup/: Check whether cgroup v2 is enabled; it returns cgroup2fs.
• systemctl list-units 'kube*' --type=slice or --type=scope: List Kubernetes-related units that systemd currently has in memory.
• bpftool cgroup list /sys/fs/cgroup/*: List all programs attached to the cgroup CGROUP.
• systemd-cgls /sys/fs/cgroup/*: Recursively show control group contents.
• systemd-cgtop: Show top control groups by their resource usage.
• tree -L 2 -d /sys/fs/cgroup/kubepods.slice: Show Pods' related cgroups directories.

How to check if a Pod CPU or memory limit is successfully applied to the cgroup file?

Work from the API object down to the node. Identify the node, then compare desired resources in the spec with the enacted values in status (in-place resize can leave those out of sync):

kubectl get pod <pod-name> -n <namespace> \
 -o jsonpath='{.spec.nodeName}{"\n"}'

kubectl get pod <pod-name> -n <namespace> \
 -o jsonpath='{range .spec.containers[*]}{.name}{" spec\t"}{.resources}{"\n"}{end}'

kubectl get pod <pod-name> -n <namespace> \
 -o jsonpath='{range .status.containerStatuses[*]}{.name}{" status\t"}{.resources}{"\n"}{end}'

On that node, as root, with the systemd cgroup driver and cgroup v2 (these snippets also need jq). Pin the container with kubelet labels; a container name alone is not unique on a node. Use crictl ps -a if the container is not running:

CONTAINER_ID=$(crictl ps \
 --label io.kubernetes.pod.namespace=<namespace> \
 --label io.kubernetes.pod.name=<pod-name> \
 --name <container-name> -q | head -n1)

# OCI view (not the CRI protobuf field names)
crictl inspect "$CONTAINER_ID" | jq '.info.runtimeSpec.linux.resources'

# Kernel cgroup path for this container
PID=$(crictl inspect "$CONTAINER_ID" | jq -r '.info.pid')
CGROUP="/sys/fs/cgroup$(awk -F: '$1=="0"{print $3}' /proc/$PID/cgroup)"

cat "$CGROUP/cpu.weight" # request.cpu → shares → weight; not a limit
cat "$CGROUP/cpu.max" # limit.cpu as "quota period"; unlimited is "max <period>"
cat "$CGROUP/memory.max" # limit.memory; unlimited is "max"

# Pod-level cgroup (parent slice), used by Pod-level resources
cat "$(dirname "$CGROUP")/cpu.max" "$(dirname "$CGROUP")/memory.max"

# Present when MemoryQoS is enabled
cat "$CGROUP/memory.high" "$CGROUP/memory.min" "$CGROUP/memory.low"

SCOPE=$(basename "$CGROUP")
systemctl show "$SCOPE" \
 -p CPUWeight -p CPUQuotaPerSecUSec -p CPUQuotaPeriodUSec -p MemoryMax

Do not treat info.runtimeSpec.linux.cgroupsPath as a filesystem path when the runtime uses the systemd driver; that value is a systemd unit path (slice:runtime:id), not a directory under /sys/fs/cgroup.

Expected mapping at each layer:

• Kubernetes: spec.containers[*].resources (desired) and status.containerStatuses[*].resources (enacted after in-place resize). Pod-level resources are on spec.resources and the parent pod slice.
• CRI (kubelet → runtime; these protobuf names stay cgroup v1-style even on v2): cpu_shares, cpu_quota, cpu_period, memory_limit_in_bytes, plus unified for Memory QoS. crictl inspect does not show these names; it shows the OCI spec the runtime derived from them.
• OCI spec: linux.resources.cpu.{shares,quota,period}, linux.resources.memory.limit, and linux.resources.unified. The runtime converts shares, quota, and period into cgroup v2 files. After upgrading crun or runc, the shares-to-weight formula may change; see CPU weight conversion.
• systemd scope: CPUWeight, CPUQuotaPerSecUSec, CPUQuotaPeriodUSec, MemoryMax. Unlimited or unset values often appear as [not set] or infinity.
• cgroupfs: cpu.weight (from CPU request; an unset request still yields the default of 2 shares), cpu.max, and memory.max on the container scope. Memory QoS adds memory.high, memory.min, and memory.low.

Further reading

• Kubernetes 1.31: Moving cgroup v1 Support into Maintenance Mode
• Kubernetes v1.36: Tiered Memory Protection with Memory QoS
• Kubernetes v1.36: PSI Metrics for Kubernetes Graduates to GA
• Kubernetes 1.25: cgroup v2 graduates to GA
• KubeCon NA 2022 cgroup v2: Before You Jump In by Tony Gosselin & Mike Tougeron, Adobe Systems
• KubeCon NA 2022 Cgroupv2 Is Coming Soon To a Cluster Near You - David Porter, Google & Mrunal Patel, RedHat
• KubeCon EU 2020 Kubernetes On cgroup v2 by Giuseppe Scrivano, Red Hat.
• This blog only covers the basic requirements and configuration of Kubernetes components. It will not include how to enable cgroup fs in OS distributions. For migration, you can refer to migrating cgroup v2

06 Oct 2026 6:00pm GMT

Memory is often the first hard limit a Kubernetes cluster hits. Nodes run out of RAM long before they run out of CPU, and the new wave of agentic AI workloads makes this worse. These workloads demand large memory footprints to start up and run untrusted code, then sit idle waiting for the next prompt. That idle but resident memory is expensive, and it caps how many pods a node can hold. This is where swap helps. Kubernetes support for running nodes with swap enabled reached General Availability in v1.34, and by backing that swap with fast NVMe solid state drives (SSDs), a node can page out dormant memory and pack in far more pods. This post explains how we benchmarked that approach across three workloads, including CI/CD kernel builds, sandboxed headless browsers, and isolated Python runtimes; we found density gains of up to 3×, often with little or no latency cost.

The node density problem

The Kubernetes ecosystem has reached a fundamental physical resource constraint: the strict limits of hardware memory versus the growing demand for dynamic, bursty workloads in the new agentic era.

Historically, administrators provisioning memory-intensive workloads encountered a persistent dilemma: set memory limits too high and you waste expensive infrastructure on idle RAM; set them too low and you risk Out-Of-Memory (OOM) kills.

This conflict is amplified when deploying autonomous AI agents using secure execution environments like the agent-sandbox framework. These agentic pods require large memory footprints to initialize and execute untrusted code. However, after their burst of activity, they typically enter long-tail idle phases waiting for user prompts. Keeping this idle state in physical RAM caps cluster density and makes AI infrastructure expensive to run.

The solution: Kubernetes node swap

With the introduction of Kubernetes' support for running nodes with swap enabled (which reached General Availability in v1.34), this paradigm shifts. By enabling the Linux kernel to page out anonymous memory to disk, node swap acts as a shock absorber during traffic spikes or periods of heavy memory oversubscription.

Historically, swap was discouraged in Kubernetes for two reasons. The first was memory accounting. Under cgroup v1, the controls treated memory and swap as a single combined limit rather than letting operators set an independent limit for disk swap. Without independent tracking, a process could page large amounts of anonymous memory out to disk, which made a container's real memory usage unpredictable and hard to isolate. Kubernetes' swap support resolves this by relying on cgroup v2, whose separate swap accounting tracks disk swap on its own. The second reason was the latency penalty of paging to slow spinning disks, which fast NVMe Local SSDs largely eliminate. Together, these make it practical to increase pod density and buffer against memory spikes without sacrificing cluster stability.

The benchmark data

To quantify the performance boundaries and cost-saving potential of Local SSD-backed node swap, this analysis covers three distinct workload categories: a Traditional Build Workload for CI/CD pipelines, High-Density Browser Sandboxes, and Isolated Python Sandboxes.

1. Traditional workload: Linux kernel build

Before exploring specialized agentic architectures, swap was validated against classic batch workloads by running a complete Linux 6.1.1 kernel build. The kernel compilation process leverages concurrent worker threads, balloons in memory to hold compiled object files, and requires a large memory spike during the brief linking phase.

This workload mirrors the memory behavior of enterprise CI/CD pipelines. Because earlier compiled objects sit inactive in memory while the pipeline progresses, CI/CD jobs frequently hoard unused physical RAM, which makes them well suited to node swap compression.

On a baseline node without swap, the minimum memory limit to prevent an OOM crash during compilation was 600 MB. Routing swap to a Local SSD cut the container memory limit by 50% to 300 MB without incurring any execution slowdown (in fact, it ran cleanly in 374s vs the baseline 433s). However, as an explicit tradeoff, compressing the limit further to 200 MB forced the active working set into swap, causing long I/O wait times and increasing execution time by over 40%. This reinforces that swap serves as an insurance policy for burst memory, not a replacement for active RAM.

2. High-density agent workloads: headless browser runtimes

AI agent workloads frequently require manipulating headless browsers via Chromium. However, trusting external code execution often requires stricter security isolation than standard Linux namespaces. This benchmark cross-evaluated several container runtime environments. The raw logs and testing methodologies for the default runtime are available in the Agent Sandbox GKE Swap directory.

• Unsandboxed baseline limits (runc): To test the limits of the environment without the overhead of security runtimes, plain runc containers were swept on a c4-standard-32 node (32 vCPU, 120 GB RAM). Without swap, the node exhausted physical memory and failed past 512 pods. Enabling Local SSD swap allowed the node to support 768 concurrent pods.
• Advanced security runtimes (for example: gVisor, Kata Containers): Enabling strict security sandboxing increases memory overhead and normally reduces pod density. However, memory swap naturally absorbs this overhead penalty. Without swap, a gVisor environment hit a hard limit at 80 pods. Local SSD swap doubled that capacity, which allowed 160 concurrent gVisor pods on a single node. Similarly, Kata Containers microVMs exhausted physical RAM at 40 concurrent pods without swap, but using GCP Local SSD swap expanded this to 50 stable Kata microVMs before CPU saturation.

At these maximum densities, the per-pod latency increase is driven mainly by pods competing for CPU, not by swap I/O. An operator tuning for a specific latency target would run at a lower density than the peak numbers here and see a proportionally smaller latency cost.

For a comprehensive architectural breakdown and density metrics for gVisor and Kata, see the Agent Sandbox GKE Swap Runtimes directory.

3. Beyond browsers: sandboxed Python runtimes

The advantages of node swap also extend to untrusted, isolated code-execution environments. This sweep deployed simultaneous Python sandbox sessions analyzing 5 million rows of data from the MovieLens 20M dataset, requiring a ≃375 MiB resident memory footprint per execution. The in-depth scaling results and deployment code for this sweep can be reviewed in the Agent Sandbox GKE Swap Python Density directory.

Without swap, heavy concurrent bursts exhausted physical memory, causing the node to hit a hard RAM limit and fail at 80 concurrent sessions. Enabling Local SSD swap offloaded dormant anonymous memory, freeing up physical RAM and preserving the node's page cache. This allowed the node to scale to 240 concurrently isolated Python sandboxes-a 3× density improvement. As with the browser workloads, the latency rise at peak density comes mainly from the sandboxes competing for CPU rather than from swap itself.

How to use it

If you manage Kubernetes infrastructure for developer environments, browser testing farms, JVM applications, or AI execution runtimes, leveraging Local SSD swap can multiply your density efficiency.

In Kubernetes v1.34+, node swap is Generally Available. You enable it via the kubelet configuration:

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
failSwapOn: false
memorySwap:
 swapBehavior: LimitedSwap

Pairing this upstream configuration with your cloud provider's high-speed local disk gives you dynamic memory balancing. For example, this is natively supported on Google Kubernetes Engine via Node Memory Swap configured on Local SSD profiles.

To get the benefits, configure your workloads with Burstable QoS: set your container's memory limits higher than its requests. The node automatically rations fast swap space based on idle application memory usage while keeping active processes responsive.

Conclusion

As the Kubernetes ecosystem transitions into the agentic era, administrators face a growing conflict between finite hardware memory limits and the bursty behavior of AI workloads. Frameworks like Agent Sandbox provide the security isolation required for running untrusted agents, but that isolation traditionally demands large amounts of idle memory overhead.

By configuring the kubelet with LimitedSwap and routing it to Local SSDs, you can mitigate this conflict. Fast swap offloads the dormant states of idle agents, allowing you to increase pod density and node utilization on the same infrastructure without compromising security boundaries.

05 Oct 2026 6:00pm GMT

As Kubernetes adoption has grown, the conversation has shifted beyond running containers to managing increasingly complex application lifecycles. Modern platforms support stateless web services, stateful databases, batch processing, AI workloads, and platform services. At the same time, they must remain reliable during upgrades, scaling events, and infrastructure failures.

Every Kubernetes user relies on SIG Apps, whether they realize it or not. Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs form the foundation of how applications are deployed, updated, scaled, and operated across the Kubernetes ecosystem.

SIG Apps is focused on improving workload resilience, refining application lifecycle management, and addressing the operational challenges that emerge when applications encounter node failures, rollout disruptions, and increasingly complex infrastructure environments.

In this spotlight, we sit down with two of the three SIG Apps chairs Janet Kuo and Maciej Szulik to discuss the evolution of Kubernetes workload management, the challenges of balancing application reliability with operational simplicity, and the future of application lifecycle management within one of Kubernetes' most influential Special Interest Groups.

Introducing SIG Apps

Natalie Fisher: Can you introduce yourself, your role, and how you got involved in SIG Apps?

Janet Kuo: I'm a Senior Staff Software Engineer at Google and have been a Kubernetes maintainer since 2015, joining the community just as we were racing toward the 1.0 launch. In those early days, my focus was on building the core Workloads API, specifically developing controllers like Deployment, ReplicaSet, StatefulSet, and DaemonSet, defining their rollout behaviors, and bringing them from initial designs to GA. That hands-on work was my entry point into SIG Apps.

Since then, I've stayed deeply involved in both the technical and community sides of Kubernetes. I have led SIG Apps as Co-Chair and Tech Lead since 2019. Currently, in addition to maintaining the workloads API, I am driving new subprojects like the Agent Sandbox to ensure Kubernetes is ready for next-generation agentic and AI workloads.

Maciej Szulik: I started contributing to Kubernetes all the way back in 2014. Since then, I've worked across various areas of the project: controllers, kubectl, and apimachinery, which eventually led me to become one of the Chairs and Tech Leads for SIG Apps. My current focus is reliability of the workload controllers under the SIG Apps umbrella and stability and ease of use of kubectl as part of my SIG CLI Tech Lead role. I also care about overall community health and growth as part of my Steering Committee role. Outside of Kubernetes, I work as a Staff Platform Engineer at Defense Unicorns, where I'm helping make Kubernetes more airgap-native with a project called zarf.

The problem and the solution

SIG Apps is responsible for the core workload APIs that power how applications run on Kubernetes. From Deployments and StatefulSets to Jobs and CronJobs, these controllers determine how workloads are created, updated, scaled, and recovered when things go wrong.

As Kubernetes expands to support increasingly diverse workloads - including AI, batch processing, and large-scale distributed applications - SIG Apps continues to evolve these APIs while balancing reliability, backward compatibility, and operational simplicity.

NF: For readers who may not be familiar, what is SIG Apps, and what role does it play within the broader Kubernetes ecosystem?

MS: SIG Apps is the Kubernetes Special Interest Group responsible for the workloads APIs. CronJob and Job help running batch workloads, whereas DaemonSet, Deployment, ReplicaSet, and StatefulSet serve the majority of other applications. More broadly, SIG Apps owns the layer most developers actually touch day-to-day: the controllers that turn a workload specification into running, self-healing pods. It's the group deciding how Deployments roll out, how Jobs retry, how DaemonSets place a pod per node.

JK: Adding to what Maciej described, as the industry shifts, we are seeing a massive demand to run complex, non-traditional workloads like distributed AI training, batch computing, and dynamic agent environments. Our role is expanding: we aren't just maintaining the classic workloads API, but we are actively evolving it and establishing new patterns (like the Agent Sandbox) to make sure Kubernetes remains the best platform for the next generation of workloads, such as AI.

NF: Looking at the workload APIs owned by SIG Apps (Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs), which areas are receiving the most attention from maintainers and contributors?

MS: After a long stretch focused on making batch workloads run smoothly on Kubernetes, we've shifted attention to make sure serving workloads (DaemonSets, StatefulSets, etc) aren't left behind. This means performance and high-scale improvements to rollout and scaling behavior, plus working through our backlog of user-reported issues, prioritizing the ones with the strongest support from the user base.

Current focus areas

As Kubernetes workloads grow in scale and complexity, the challenges facing workload controllers evolve as well. We asked the SIG Apps chairs where contributors are focusing their efforts today and which resilience problems they believe are the highest priorities.

NF: From your perspective, what are the most important workload resilience problems SIG Apps is trying to solve today?

MS: Node lifecycle challenges have come up repeatedly across SIG Apps, SIG Node, and SIG Autoscaling discussions. DaemonSets and Jobs are just where the pain is most visible, since they're the workloads most directly bound to node state. Rather than solve it piecemeal within one SIG, we've settled on spinning up a dedicated Node Lifecycle Working Group to focus on this properly and hopefully land long-term solutions instead of one-off patches.

JK: From an AI perspective, resilience is critical. When you are running a massive distributed LLM training job that spans hundreds of GPUs, a single node failure can halt the entire pipeline. Similarly, if a DaemonSet that runs your logging or GPU monitoring agent gets stuck on a bad node, it impacts the entire cluster's health.

In addition to the work in the Node Lifecycle WG to handle infrastructure-level degradation, SIG Apps is addressing this at the orchestration layer through subprojects like JobSet (for distributed training) and LeaderWorkerSet (LWS) (for sharded LLM inference). These APIs introduce patterns like "all-or-nothing" failure handling, where a single pod or job failure triggers a coordinated group-level restart to resume from the last clean checkpoint, rather than letting stuck workloads hang in an inconsistent state.

Real-world impact

The work happening within SIG Apps extends far beyond controller implementations and API design. We wanted to understand what these improvements mean in practice for platform teams operating Kubernetes clusters in production.

NF: For platform teams operating Kubernetes in production, what practical improvements would they notice if the node lifecycle and workload resilience work currently under discussion is successfully delivered?

MS: I'm mostly looking from the sidelines, the folks actually in the Node Lifecycle Working Group would give you a sharper answer. But from where I sit, I'm hoping their work translates into fewer 3am pages that turn out to be "a DaemonSet rollout got stuck because node X was flaky, and someone had to manually cordon/delete/restart to unstick it."

JK: +1 to what Maciej said, and beyond reducing manual intervention, platform teams will also see much better resource predictability and cost efficiency. For example, in AI workloads where GPU idle time is extremely expensive, having Kubernetes automatically detect a degraded node and reschedule the training coordinator or agent before the job crashes means less wasted compute and more stable job execution.

Challenges and trade-offs

Evolving APIs that millions of workloads rely on requires careful engineering and even more careful decision-making. We asked the SIG Apps chairs about the technical and operational trade-offs they weigh when introducing changes to Kubernetes' core workload controllers.

NF: What are some of the hardest technical or operational trade-offs SIG Apps encounters when evolving core workload controllers?

MS: Honestly, a few tensions keep coming up: how aggressively a controller should give up on stuck pods, and what signals it actually needs to make that call correctly. At the same time, we always have to think about backward compatibility. Deployment, DaemonSet, and Job behavior has been depended on for a decade [by Kubernetes users, tooling, automation, and higher-level controllers], so even a change that's clearly "more correct" can break automation people built around the old behavior without meaning to.

JK: One of our hardest trade-offs is resisting the urge to make "elegant" design changes that break backward compatibility. Instead, we have to design opt-in features that let users adopt new behaviors without forcing them on legacy workloads. When we need to support completely new paradigms, we prefer introducing them as CRDs first rather than bloating the core APIs, like we are doing with Agent Sandbox, JobSet, and LWS.

Looking ahead

While much of SIG Apps' work focuses on maintaining the stability of existing workload APIs, the group is also shaping the future of Kubernetes through new enhancements and proposals. We concluded by asking about one proposal that recently returned to active development and what it represents for the future of workload management.

NF: The SIG recently discussed reviving KEP-4443 with a target release of Kubernetes 1.38. What opportunities or challenges does this proposal aim to address, and why is now the right time to revisit it?

KEP-4443 addresses a small but real gap in the Job API: a PodFailurePolicy can be configured to add a condition reason to the JobFailed condition, but different pod failure policy rules targeting different container exit codes all produce that same generic reason. The proposal is simple: an optional Name field on each PodFailurePolicyRule, which gets appended to the JobFailed condition reason, so higher-level tools like JobSet can finally react differently depending on which rule triggered the failure.

As for timing, the answer is as simple as it always is in open source: we lost the original contributor who was driving this. Now we've got someone new interested in picking it up, that's why we're targeting the next release.

Getting Involved

NF: For someone interested in contributing to SIG Apps, where would you recommend they start, especially if they are not yet a Kubernetes maintainer?

MS: The best place to start is the #sig-apps slack channel and our regular SIG Apps meetings. We've all started there, and if it feels intimidating, or nobody replies right away, that's completely normal. Everyone's busy. It's not personal.

JK: In addition to what Maciej answered, I'd suggest looking at our newer subprojects and initiatives. Contributing to stable APIs like Deployment or StatefulSet can be daunting because the barrier for making changes is very high due to backward compatibility, and there is much less low-hanging fruit.

If you are new to the community, projects like the Agent Sandbox are fantastic entry points. They are actively evolving, have a friendly group of maintainers, and offer plenty of greenfield development opportunities where you can make a significant impact quickly.

Summary

SIG Apps has shaped how Kubernetes applications are deployed and operated since the project's earliest days. While users often interact with Deployments, StatefulSets, Jobs, and DaemonSets without thinking about the controllers behind them, the work within SIG Apps continues to shape the reliability and scalability of workloads across the Kubernetes ecosystem.

From improving workload resilience and node lifecycle behavior to enabling new patterns for AI and distributed computing, the SIG is evolving Kubernetes while remaining committed to one of the project's core principles: preserving the stability and backward compatibility that users depend on. Whether you're interested in core workload APIs, emerging projects like Agent Sandbox, or helping improve the operational experience of Kubernetes users everywhere, SIG Apps offers many opportunities to get involved.

22 Sep 2026 6:00pm GMT

Kubernetes v1.37 promotes the PersistentVolumeClaimUnusedSinceTime feature gate to Beta (enabled by default). With this feature, the PersistentVolumeClaim (PVC) protection controller adds an Unused condition to each PVC, telling you whether any running pod currently references it - no custom tooling or cross-referencing required.

For the API definition of PVC conditions, see the PersistentVolumeClaim API reference. Read on to learn how the Unused condition works and how to use it.

Why track PVC usage?

In large-scale Kubernetes clusters, it is common for users to create PVCs and then delete the associated pods without cleaning up the storage, because Kubernetes does not automatically delete PVCs when their pods are removed (to protect against accidental data loss). Over time, these orphaned PVCs may accumulate, silently consuming storage capacity and driving up cloud costs.

Before Kubernetes v1.37, it was easy to identify an unused PersistentVolume, but much harder to determine whether a PVC was still being used. Doing so required cross-referencing pods, PersistentVolumes, and PVCs over a potentially large window of time. Administrators often resorted to custom monitoring pipelines or scripts to answer a seemingly simple question: "Is anything actually using this volume?"

The PersistentVolumeClaimUnusedSinceTime feature solves this by making the answer available natively in the PVC status. Once the feature is enabled, every PVC gets an Unused condition managed by the PVC protection controller.

User stories

• Storage administrator: "I want to know which PVCs in my cluster are not being used by any pod so I can safely identify orphaned volumes and schedule them for deletion."
• DevOps engineer: "I want to list PVCs that have the Unused condition set to True so I can automate cleanup in development environments."

How does it work?

The PVC protection controller - which already watches pods to enforce the storage object in use protection - now also manages a new Unused condition on PVCs.

The condition works as follows:

A few details worth noting:

• Terminated pods don't count: A pod that has completed (phase Succeeded or Failed) does not keep the PVC marked as in use. This means batch jobs with restartPolicy: Never won't prevent the PVC from becoming Unused=True after they finish.
• Pending pods do count: Even an unschedulable pod (for example, one with an impossible node selector) still counts as using the PVC. The intent to use the volume is enough.
• Multiple pods: If several pods reference the same PVC, the condition transitions to Unused=True only after the *last" non-terminated pod is removed or terminates.

Using lastTransitionTime to find when a PVC became idle

Like every Kubernetes condition, the Unused condition carries a standard lastTransitionTime field. This means you get a useful bonus for free: when the condition transitions from False to True, the lastTransitionTime records exactly when the PVC became idle. You can use this timestamp to answer questions like "how long has this PVC been sitting unused?" - for example, to find PVCs that have been idle for more than 30 days (see the example query below).

What changed from Alpha to Beta?

Kubernetes v1.36 introduced this feature as Alpha, where you had to enable the PersistentVolumeClaimUnusedSinceTime feature gate explicitly. For Beta in v1.37, the feature gate is enabled by default, and the feature has full end-to-end test coverage.

How to use it

Since the feature is Beta and enabled by default in Kubernetes v1.37, the Unused condition will appear on PVCs automatically. Here is a walkthrough to see it in action:

1. Create a PVC:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
    name: my-data
   spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
    storage: 1Gi
   

2. After a short time, inspect the PVC conditions:

   kubectl get pvc my-data -o jsonpath='{.status.conditions[*]}' | jq .
   

   You should see an Unused condition with status True and reason NoPodsUsingPVC:

   {
    "lastProbeTime": null,
    "lastTransitionTime": "2026-09-14T12:03:11Z",
    "message": "No pods are currently referencing this PVC",
    "reason": "NoPodsUsingPVC",
    "status": "True",
    "type": "Unused"
   }
   

3. Create a pod that uses the PVC:

   apiVersion: v1
   kind: Pod
   metadata:
    name: my-app
   spec:
    containers:
    - name: app
    image: busybox
    command: ["sleep", "3600"]
    volumeMounts:
    - name: data
    mountPath: /data
    volumes:
    - name: data
    persistentVolumeClaim:
    claimName: my-data
   

4. Check the condition again - it should now show Unused=False:

   kubectl get pvc my-data -o jsonpath='{.status.conditions[?(@.type=="Unused")].status}'
   

   Output:

   False
   

5. Delete the pod and wait for the condition to transition back to Unused=True:

   kubectl delete pod my-app
   kubectl get pvc my-data -o jsonpath='{.status.conditions[?(@.type=="Unused")]}'
   

   The condition should show Unused=True with reason NoPodsUsingPVC again.

Finding unused PVCs across the cluster

To list all PVCs that have been unused for more than 30 days, you can use a command like:

kubectl get pvc -A -o json | jq -r '
 .items[]
 | select(.status.conditions[]? | select(.type=="Unused" and .status=="True"))
 | select(
 (.status.conditions[] | select(.type=="Unused") | .lastTransitionTime) as $t
 | (now - ($t | fromdateiso8601)) > (30 * 86400)
 )
 | "\(.metadata.namespace)/\(.metadata.name) unused since \(.status.conditions[] | select(.type=="Unused") | .lastTransitionTime)"
'

What's next?

Depending on feedback and adoption, the Kubernetes project intends to graduate this feature to General Availability (GA) in a future release. If you have feedback on this feature, please open an issue in the kubernetes/kubernetes repository.

To learn more about this enhancement, refer to KEP-5541: PersistentVolumeClaim last used time.

Getting involved

The Kubernetes project always welcomes new contributors. If you would like to get involved, you can join us at SIG Storage.

If you would like to share feedback, you can do so on our public Slack channel (visit https://slack.k8s.io/ for an invitation if you need one).

Special thanks to the contributors who helped design and implement this feature (alphabetical order):

• Arvind Parekh (ArvindParekh)
• Hemant Kumar (gnufied)
• Jan Šafránek (jsafrane)
• Kevin Hannon (kannon92)
• Roman Bednář (RomanBednar)

21 Sep 2026 6:30pm GMT

Kubernetes v1.37 brings important storage security features: emptyDir permission modes and bind mount options. They help application programmers and security professionals implement rigorous security policies, for example, prohibiting deletion of files across containers or execution of arbitrary binaries from writable volumes, directly in Kubernetes without any complicated circumvention.

Linux storage and permission fundamentals

Before diving into the new Kubernetes features, let us briefly review the low-level Linux security mechanisms that make them possible.

Bind mount flags

When Linux mounts or remounts a directory, Virtual File System (VFS) flags control what actions are permitted on that filesystem:

• noexec: Do not permit direct execution of any binaries on the mounted filesystem.
• nosuid: Do not allow set-user-identifier or set-group-identifier bits to take effect.
• nodev: Do not interpret character or block special devices on the file system.

Directory permissions and the sticky bit

Standard Unix permissions regulate access across three scopes: Owner, Group, and Others (e.g., 0755 or 0777).

Beyond standard read, write, and execute bits, Linux supports the sticky bit (as in mode 01777). When applied to a directory, the sticky bit ensures that a file inside that directory can only be deleted or renamed by the file's owner or root. This is essential for shared writable directories like /tmp.

Motivation for the improvements

Why does Kubernetes need bind mount options and emptyDir permissions?

The primary goal of these features is to increase the security of Kubernetes workloads by allowing security-related bind mount options on volume mounts. By default, volumes are bind-mounted into containers by the container runtime and kubelet without noexec, nosuid, or nodev flags. This default can undermine security. For example, with noexec missing, a compromised process can use any writable volume (emptyDir, PersistentVolume, etc.) to download, chmod +x, and execute arbitrary binaries even when the container has a read-only root filesystem (readOnlyRootFilesystem: true). Supporting noexec, nodev, and nosuid gives users a native way to harden volume mounts to match security benchmarks and policy.

The gap is most visible with emptyDir volumes, which are the most common writable volume type and have been the subject of multiple security findings:

• Issue #48912: Recognized security gap - the inability to set mount options on emptyDir was flagged in an audit but remained unresolved until now.
• Issue #119627: Kubernetes 1.24 Security Audit (Finding NCC-E003660-7HM) - external auditors specifically noted that the inability to mount emptyDir with noexec represents a security failure.

However, the same gap applies to all volume types. PersistentVolumes have a mountOptions field, but those options are filesystem-level flags applied by the CSI driver at the node, so they do not reliably translate into bind mount flags inside the container. Previously, there was no mechanism to set noexec, nosuid, or nodev on the bind mount that the container runtime creates for any volume type.

Additionally, the emptyDir volume type defaults to creating directories with a hardcoded mode of 0777. This previously meant that any process that can discover the volume could read, write, and delete anything in the volume, regardless of who created it.

You could - and still can - use an initial container to set a different access mode, but this is more complex, and hard to verify for compliance.

This causes real problems:

• Multi-container pods sharing an emptyDir could not prevent one container from deleting another's files. The sticky bit (01777) solves this, but there was no native way to set it.
• Some applications and security frameworks expect /tmp directories to have the sticky bit set (mode 01777). Without native support for setting the emptyDir mode, users had to use init containers or alternative volume types to meet this requirement.
• Platform engineers who want tighter permissions (e.g., 0750 for owner and group only) have to use init containers running chmod, which adds unnecessary complexity.

The emptyDir volume type was a notable gap. As one of the most common writable volume types in Kubernetes, it had no way to control its creation permissions.

Real-world use cases

Application developers, working closely with security engineers, are responsible for maintaining the security posture of their applications and ensuring workloads do not pose risks to the wider infrastructure. These features allow development teams to confidently address critical security scenarios:

Preventing Privilege Escalation on Writable Mounts: An application developer configuring temporary workspace volumes (like emptyDir or /tmp mounts) can ensure they are mounted with nosuid and noexec. This guarantees that even if the application is compromised and a malicious payload is downloaded, the workload cannot execute the payload or use it to escalate privileges on the node.

Securing Shared Scratch Space in Multi-Container Pods: A developer configuring CI/CD pipeline pods often needs multiple containers (e.g., a builder container and a sidecar logger) to share a workspace. By setting mode: 01777 on an emptyDir, the developer ensures the shared workspace behaves like a traditional Unix /tmp directory. Each container can write files independently, but a compromised process in one container cannot delete the build artifacts produced by another.

Enforcing Principle of Least Privilege for Application Data: An application developer deploying a database pod can lock down access to the database's temporary storage. By setting mode: 0750 on the emptyDir, the developer ensures that only the specific database user and group can read or write to the volume, explicitly denying access to any other processes or sidecars in the same pod.

Note: Both features are behind Alpha feature gates in Kubernetes v1.37. To use them, enable VolumeBindMountOptions and EmptyDirVolumeMode on the API server and kubelet.

Example 1: Enforcing bind mount options

This full Pod manifest mounts an emptyDir volume at /tmp with bindMountOptions: [noexec, nosuid].

apiVersion: v1
kind: Pod
metadata:
 name: hardened-bindmount-pod
 namespace: default
spec:
 os:
 name: linux
 containers:
 - name: hardened-app
 image: alpine:latest
 command: ["sleep", "3600"]
 securityContext:
 readOnlyRootFilesystem: true
 volumeMounts:
 - name: temp-storage
 mountPath: /tmp
 bindMountOptions:
 - noexec
 - nosuid
 volumes:
 - name: temp-storage
 emptyDir: {}

Example 2: emptyDir volume permission mode with sticky bit

This full Pod manifest creates an emptyDir volume using mode: 01777 to enforce standard Unix /tmp sticky bit protections across containers.

apiVersion: v1
kind: Pod
metadata:
 name: hardened-emptydir-pod
 namespace: default
spec:
 os:
 name: linux
 containers:
 - name: app-container
 image: alpine:latest
 command: ["sleep", "3600"]
 volumeMounts:
 - name: shared-tmp
 mountPath: /tmp
 volumes:
 - name: shared-tmp
 emptyDir:
 mode: 01777

Verifying the features in Linux

To verify that these features are actively enforcing restrictions, you can run kubectl exec into the container. The following examples simulate attempts to perform actions that are successfully blocked by these features.

Verifying noexec

Attempt to write and run a script on a volume mounted with noexec:

# 1. Exec into the pod
kubectl exec -it hardened-bindmount-pod -- sh

# 2. Create an executable script on the mounted volume
cd /tmp
echo '#!/bin/sh' > test.sh
echo 'echo "Executing untrusted code..."' >> test.sh
chmod +x test.sh

# 3. Attempt to run the script
./test.sh

Expected result:

sh: ./test.sh: Permission denied

Even if an executable file is created, the Linux kernel refuses execution because MS_NOEXEC is enforced at the bind mount level.

Verifying the sticky bit

Attempt to delete another user's file in an emptyDir with 01777 permission mode:

# 1. Exec into the pod
kubectl exec -it hardened-emptydir-pod -- sh

# 2. Verify directory permissions on /tmp
ls -ld /tmp
# Output: drwxrwxrwt 2 root root ... /tmp (Notice the 't' indicating sticky bit)

# 3. Create a file as the guest user
su -s /bin/sh -c "touch /tmp/guest_file" guest

# 4. Attempt to delete that file as nobody
su -s /bin/sh -c "rm /tmp/guest_file" nobody

Expected result:

rm: can't remove '/tmp/guest_file': Operation not permitted

The kernel blocks deletion because the sticky bit (01777) restricts file removal strictly to the owner of the file.

Things to know

Keep these key details in mind as you begin using these features. Full details are available in the official documentation for bind mount options, emptyDir volume mode, and emptyDir volumes.

• Default unchanged: If you omit bindMountOptions or do not set an emptyDir mode, you get standard default behaviors (like 0777 permissions) exactly as before.
• Broad volume support: bindMountOptions works with emptyDir, PersistentVolumes, CSI volumes, projected volumes, ConfigMaps, Secrets, and more. The only exception is image volumes, which are explicitly unsupported. The mode field works with all emptyDir medium types: default (disk-backed), Memory (tmpfs), and HugePages.
• Runtime capabilities matter (for bindMountOptions): The container runtime must support the CRI mount_options field and advertise it via runtimeFeatures. The scheduler uses node declared features to avoid placing pods on incompatible nodes. If a pod reaches such a node anyway, the kubelet rejects it. There is no silent degradation. However, using mode for an emptyDir does not require runtime support.
• Not the same as PV mountOptions: PersistentVolume mountOptions apply at the storage layer via the CSI driver. The new bindMountOptions controls bind mount flags applied inside the container by the runtime. They operate at different layers and do not conflict.
• fsGroup interaction: If fsGroup is set in the pod's security context, the group permissions applied by fsGroup will override the mode specified for the emptyDir volume. This is the same behavior that exists for defaultMode on Secret and ConfigMap volumes.
• Linux only: Flags like noexec, nosuid, nodev, and Unix permission modes are Linux concepts. bindMountOptions has no effect on Windows nodes. On Windows, the mode field is also skipped for emptyDir volumes, since Windows does not support Unix-style file permissions.
• Version skew safety: Both features are additive. For emptyDir mode: if the API server has the gate enabled but the kubelet does not, the field is accepted but ignored - the kubelet falls back to 0777. For bindMountOptions: the scheduler uses Node Declared Features to prevent placing pods on nodes without runtime support; if a pod reaches such a node, the kubelet rejects it rather than silently ignoring the options.
• Feature Gates: Both capabilities are available as Alpha features in Kubernetes v1.37:
  • VolumeBindMountOptions: Controls bind mount flags on volume mounts.
  • EmptyDirVolumeMode: Controls creation permission modes on emptyDir volumes.

How do I get involved?

These new features are driven by SIG Node and SIG Storage. You can find more details in the KEPs for these enhancements: KEP-5855 (bind mount options) and KEP-5502 (emptyDir permission mode).

Reach out to SIG Node:

• Slack: #sig-node
• Mailing list

Reach out to SIG Storage:

• Slack: #sig-storage
• Mailing list

16 Sep 2026 6:30pm GMT

With the release of Kubernetes v1.37, the Pod-Level Resource Managers feature has graduated to Beta status (disabled by default)!

First introduced as an Alpha feature in Kubernetes v1.36, this enhancement builds on Pod-Level Resources by equipping Kubelet's Topology Manager, CPU Manager, and Memory Manager to use Pod-level resource declarations (.spec.resources) directly when making hardware placement decisions.

Bringing pod-level resources to node managers

Before this feature, obtaining exclusive NUMA-aligned CPU cores or memory for latency-critical applications forced cluster operators into an all-or-nothing choice: assign integer resource requests to every container in the Pod, or forfeit exclusive NUMA alignment entirely. For modern workloads running lightweight sidecars (such as logging agents or telemetry exporters), allocating dedicated physical cores to auxiliary containers was wasteful.

Pod-Level Resource Managers solves this challenge by enabling hybrid allocation models. The Kubelet can reserve exclusive NUMA-aligned resources for primary application containers while placing non-Guaranteed sidecars into a pod-isolated shared pool. This ensures primary workloads get unthrottled, NUMA-local performance while sidecars benefit from running in a pod-isolated shared pool, enjoying local NUMA alignment and protection from external node interference without consuming dedicated physical cores.

What's new in Beta

Graduating to Beta brings key operational and API enhancements:

• Graduation to Beta: Controlled by the PodLevelResourceManagers feature gate, available to opt in (disabled by default) in Kubernetes v1.37.
• PodResources API Reporting: The v1 PodResources gRPC service (PodResourcesLister) introduces top-level cpu_ids and memory fields on PodResources responses. Monitoring tools and device plugins can query pod-level exclusive assignments directly without double-counting container allocations.

Getting started and providing feedback

For a deep dive into the technical details and configuration of this feature, check out the official documentation:

• Pod-level resource managers
• Pod-level resource managers reference

To follow a step-by-step tutorial on configuring and deploying workloads:

• Use pod-level resources with Kubelet resource managers

To learn more about how to assign resources to pods:

• Assign Pod-level CPU and memory resources

As this feature moves through Beta toward GA, your feedback is invaluable. Please report any issues or share your experiences via the standard Kubernetes communication channels:

• Slack: #sig-node
• Mailing list
• Open Community Issues/PRs

15 Sep 2026 6:30pm GMT

Memory QoS has graduated to Beta in Kubernetes v1.37 and is now enabled by default. On Linux nodes running cgroup v2, the feature uses the memory controller to give the kernel better guidance on how to treat container memory. It was first introduced as Alpha in v1.22, and expanded in v1.36 with tiered memory reservation.

This post covers what changed in v1.37, what the Beta promotion means for cluster operators, and how to configure the feature.

What changed in v1.37

Memory QoS is Beta and enabled by default

The MemoryQoS feature gate is now Beta in v1.37. This means every v1.37 kubelet has the feature gate turned on without any configuration change. Turning on the feature by default is safe because the default kubelet configuration does not enable memory throttling or memory reservation. No memory.high, memory.min, or memory.low values are written to cgroups unless you explicitly configure them.

You can opt into specific behaviors through kubelet configuration fields:

1. Set memoryThrottlingFactor (for example, 0.9) to enable memory.high throttling on Burstable and BestEffort containers. The default is null, which means no throttling.
2. Set memoryReservationPolicy to TieredReservation to enable tiered memory protection via memory.min and memory.low. The default is None, which means no memory reservation.

Default memoryThrottlingFactor changed to null

In earlier Alpha releases, memoryThrottlingFactor defaulted to 0.9, which meant enabling the feature gate caused the kubelet to set memory.high on containers. In v1.37, the default is null, so the kubelet does not set memory.high unless you configure a value.

This change was made because, with the feature gate now on by default, an automatic memory.high could throttle workloads that were previously running without throttling. Making it null ensures that upgrading to v1.37 does not change runtime behavior for existing clusters.

If your kubelet configuration file already contains an explicit memoryThrottlingFactor value, that value is preserved during the upgrade and throttling continues to work as before. If your configuration file does not include memoryThrottlingFactor, the kubelet uses the new null default and stops setting memory.high. To keep throttling in that case, add memoryThrottlingFactor explicitly:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9

How to configure MemoryQoS in v1.37

For full details on configuring Memory QoS, see Memory QoS with cgroup v2, Configuring memory reservation, and System requirements

Enable memory throttling only

Set memoryThrottlingFactor to a value between 0 and 1. The kubelet uses this factor to calculate memory.high for Burstable and BestEffort containers. See Memory throttling for how memory.high is calculated for each QoS class.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9

Enable memory throttling and tiered reservation

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9
memoryReservationPolicy: TieredReservation

Enable tiered reservation without throttling

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryReservationPolicy: TieredReservation

Disable Memory QoS entirely

To disable the feature after upgrading, set the feature gate to false and ensure a compatible kubelet configuration. The kubelet rejects the configuration if memoryThrottlingFactor is set to anything other than the former default of 0.9, or if memoryReservationPolicy is TieredReservation, so remove or adjust those fields if you set them.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
 MemoryQoS: false

When the feature gate is off, or memoryReservationPolicy is not TieredReservation, the kubelet resets stale protection at startup on cgroup v2 nodes: memory.min=0 and memory.low=0 on the root kubepods cgroup, and memory.low=0 on the Burstable QoS cgroup. For containers, stale memory.high values are reset to max on reconciliation paths such as restart or resize.

Known limitation: memory reservation is node-wide

memoryReservationPolicy applies to every pod on the node. With TieredReservation, every Guaranteed pod gets memory.min and every Burstable pod gets memory.low; there is no way to opt individual pods in or out. A node that mixes workloads needing hard reservation with workloads that should stay reclaimable has to choose one policy for all of them.

Hard reservation also covers everything charged to the container's cgroup, including page cache, so a pod that reads large files can hold memory the kernel would otherwise reclaim to serve its neighbors.

SIG Node is tracking both in kubernetes/kubernetes#140246. If this affects you, that issue is the best place to describe your workload.

What to expect next

The next milestone for Memory QoS is graduation to GA. Feedback from Beta users will shape any remaining adjustments before that step. If you run into issues, please file bugs at kubernetes/kubernetes.

How can I learn more?

• KEP-2570: Memory QoS
• Pod Quality of Service Classes
• Memory QoS with cgroup v2
• Managing Resources for Containers
• Kubernetes cgroups v2 support
• Linux kernel cgroups v2 documentation

Getting involved

This feature is driven by SIG Node. If you are interested in contributing or have feedback, you can reach out through:

• Slack: #sig-node
• Mailing list
• SIG Node meetings

14 Sep 2026 6:30pm GMT

Changed Block Tracking (CBT) support for CSI drivers shipped as Alpha in September 2025. With the March 2026 v1.0.0 release of the external-snapshot-metadata project, the feature moved to Beta.

If you aren't yet familiar with changed block tracking for storage in Kubernetes, the Alpha announcement covers the motivation, the three primary components (the CSI SnapshotMetadata gRPC service, the SnapshotMetadataService CRD, and the external-snapshot-metadata sidecar), and a walkthrough of how to use the API. CBT currently applies to block volumes; file-volume and network file-share changed-list tracking is not covered by this feature. This post focuses on what is different in Beta.

What's new in Beta

The main change in that release was the promotion of the SnapshotMetadataService CRD from v1alpha1 to v1beta1. The CRD used to advertise a driver's metadata service now serves cbt.storage.k8s.io/v1beta1. The schema itself is unchanged, but this release removed v1alpha1 (rather than serving it alongside the new version). If you are upgrading from Alpha, you need to:

• Re-apply the CRD definition shipped with v1.0.0.
• Update SnapshotMetadataService manifests to use apiVersion: cbt.storage.k8s.io/v1beta1.
• Update any client or controller code that talks to the CRD.

This is a one-time change. There is no automatic conversion between the two versions.

Compatibility

• Minimum Kubernetes version: 1.33
• CSI spec: 1.10 or newer
• Container image: registry.k8s.io/sig-storage/csi-snapshot-metadata:v1.0.0

Trying it out

The Getting Started section in the Alpha blog still applies. In short:

1. Make sure your CSI driver supports volume snapshots and ships the external-snapshot-metadata sidecar.
2. Install the SnapshotMetadataService CRD (the v1beta1 definition from the v1.0.0 release).
3. Create a SnapshotMetadataService resource for your driver.
4. Use a client - snapshot-metadata-lister, or your own implementation - to call GetMetadataAllocated and GetMetadataDelta.

If you want to see the full flow end-to-end, the hostpath driver example is a good starting point.

What's next?

The focus for the rest of the Beta cycle is wider CSI driver adoption and operational feedback before the feature moves towards GA. If you maintain a CSI driver, this is a good time to evaluate adding support. If you are building a backup application on top of the API, feedback on the streaming clients and the iterator package is very welcome.

Where can I learn more?

• The CSI developer documentation for snapshot metadata.
• KEP-3314.
• The external-snapshot-metadata repository.
• The gRPC schema.
• The snapshot-metadata-lister example client.

How do I get involved?

This work is the result of contributions from many people across SIG Storage. A big thank you to everyone who helped review, code, and test the feature through Alpha and into Beta:

• Ben Swartzlander (bswartz)
• Carl Braganza (carlbraganza)
• Daniil Fedotov (hairyhum)
• Ivan Sim (ihcsim)
• Nikhil Ladha (Nikhil-Ladha)
• Praveen M (iPraveenParihar)
• Rakshith R (Rakshith-R)
• Xing Yang (xing-yang)

If you would like to get involved with CSI or storage in Kubernetes, SIG Storage is the place to start. The Data Protection Working Group also holds regular meetings, and new attendees are always welcome.

14 Sep 2026 6:30pm GMT

I'm excited to announce that native histogram support for Kubernetes metrics is graduating to Beta and is enabled by default in Kubernetes v1.37!

Native histograms (previously introduced as Alpha in Kubernetes v1.36 under KEP-5808) bring high-resolution, low-cardinality observability to Kubernetes metrics. By adopting Prometheus Native Histograms, Kubernetes components now expose latency and duration metrics with far greater accuracy while significantly reducing telemetry storage and scraping overhead.

Why move beyond classic histograms?

Since the early days of Kubernetes observability, duration and latency metrics (such as API server request latencies or scheduling durations) have relied on classic Prometheus histograms.

Classic histograms require metric authors to define a static list of cumulative bucket boundaries (le labels), such as 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10. While familiar, this approach introduces three major challenges:

1. The Bucket Guessing Game: If a workload's latency profile changes, for example, shifting into microsecond ranges or experiencing long-tail tail latencies beyond the highest bucket, the histogram loses visibility. Specifying bucket boundaries upfront requires knowing the distribution before observing it
2. High Cardinality & Storage Cost: With classic histograms, each bucket boundary is exported as a separate time series (_bucket{le="..."}). A histogram with 10 buckets across multiple labels multiplies the number of time series by 10, increasing memory consumption in Prometheus and inflating time series database (TSDB) storage costs
3. Interpolation Error in Quantiles: Calculating percentiles using histogram_quantile() relies on linear interpolation between static bucket boundaries. When bucket spans are coarse, quantile calculations can suffer from significant estimation error

What are Prometheus native histograms?

Prometheus Native Histograms replace static user-defined buckets with dynamic, exponential buckets.

Instead of emitting a separate time series for every single bucket boundary, a native histogram is stored as a single time series containing a rich schema of positive and negative spans, zero thresholds, and exponential scaling factors.

• High Resolution Automatically: Exponential buckets dynamically adjust to any value range - from nanoseconds to hours - without requiring pre-configured bucket boundaries
• Up to 90% Fewer Time Series: By consolidating buckets into structured spans within a single time series, scraping and storage overhead are dramatically reduced
• Accurate Quantile Calculation: Quantiles can be calculated with mathematical bounds on error (≃5% worst-case relative error under default settings) across the entire spectrum of observations

How native histograms work in Kubernetes

In Kubernetes, native histogram support is implemented directly inside the shared metrics subsystem (k8s.io/component-base/metrics).

Figure 1 illustrates how native histogram metrics are processed and exposed across Kubernetes components.

1. Dual exposition for zero breaking changes

A primary design requirement for KEP-5808 was zero disruption for existing observability stacks. When the NativeHistograms feature gate is enabled, Kubernetes components use dual exposition:

• Classic buckets (h.Bucket) are still emitted alongside native spans. Existing Prometheus servers, dashboards, and alerting rules that rely on traditional text scraping or classic bucket labels continue to work unmodified
• Native spans (h.Schema, h.PositiveSpan) are included in the same Protobuf payload for collectors that understand native histograms

2. Tuned default exponential configuration

When NativeHistograms is enabled, the k8s.io/component-base/metrics package automatically applies standardized exponential options to all histogram metrics:

• BucketFactor: 1.1: Configures exponential buckets where each bucket is at most 10% wider than the preceding one. This guarantees a mathematically bounded worst-case relative error of at most ~5% for quantile calculations regardless of whether an operation takes 1 millisecond or 10 seconds.
• MaxBucketNumber: 160: Caps the maximum number of buckets per histogram to 160. Following OpenTelemetry SDK recommendations for base-2 exponential histogram aggregation, this limit protects component memory usage even under extreme outlier distributions.

3. Broad component support

Because native histograms are integrated into component-base/metrics, all major Kubernetes control plane and node components inherit support automatically, including:

• kube-apiserver (e.g., apiserver_request_duration_seconds, authentication/authorization metrics, validation latencies)
• kube-scheduler (e.g., scheduler_plugin_execution_duration_seconds, scheduler_scheduling_algorithm_duration_seconds)
• kubelet (node-level container runtime and pod lifecycle metrics)
• kube-controller-manager and kube-proxy

How to scrape native histograms

The simple answer: upgrade to Kubernetes v1.37, and it works.

Because Kubernetes v1.37 enables NativeHistograms by default, your cluster is already emitting dual-exposition metrics. How you configure Prometheus to scrape native histograms depends on your Prometheus version:

1. Prometheus scrape configuration by version

• Prometheus 3.0+ (Recommended): Use explicit per-job configuration in your scrape_configs rather than global flags (the global --enable-feature=native-histograms flag is deprecated in Prometheus 3.9+):

  scrape_configs:
   - job_name: 'kubernetes-apiservers'
   scrape_native_histograms: true
   always_scrape_classic_histograms: true # Recommended during transition
  

  You must read the caution in Migrating dashboards and alerts in the Native Histograms documentation. In summary: always set always_scrape_classic_histograms: true during your transition period. Without this setting, Prometheus will only ingest the native format and stop ingesting classic _bucket, _count, and _sum series. Setting always_scrape_classic_histograms: true ensures existing dashboards (histogram_quantile(..._bucket...)) and alerts continue to work while you migrate them to native histograms.

• Prometheus 2.40 - 2.x: Enable Native Histograms globally by starting Prometheus with the feature flag:

  prometheus --enable-feature=native-histograms
  

  Note that in Prometheus 2.x, this is an all-or-nothing setting for all scrape targets.

2. Verify Protobuf dual exposition

Standard Prometheus text scraping (application/openmetrics-text or plain text format) only transfers classic buckets. When scrape_native_histograms is enabled, Prometheus automatically negotiates Protobuf format with Kubernetes endpoints.

You can verify that a Kubernetes component is exporting native histograms using curl with an Accept header specifying Protobuf. For example:

## THIS IS NOT SECURE. ONLY DO THIS IN A TEST CONTEXT.
curl --insecure \
 -H "Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited" \
 --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
 https://localhost:6443/metrics

When decoded, the returned MetricFamily for histogram metrics (like apiserver_request_duration_seconds) will contain both traditional bucket entries and populated schema / positive_span fields.

Querying native histograms in PromQL

Once native histograms are ingested into Prometheus, you can query them using standard PromQL histogram functions without needing static le bucket labels or _bucket suffixes:

# 1. Calculating P99 latency for a single target:
# Classic histogram (requires _bucket suffix):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m]))

# Native histogram (operates directly on the metric name):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds[5m]))

# 2. Aggregating across multiple instances (e.g., all API servers):
# Classic histogram (requires sum by (le) to preserve bucket boundaries):
histogram_quantile(0.99, sum by (le) (rate(apiserver_request_duration_seconds_bucket[5m])))

# Native histogram (no grouping by le required!):
histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds[5m])))

With native histograms, functions like histogram_quantile() operate directly on the dynamic exponential spans inside the time series, producing highly accurate quantiles without static bucket interpolation error.

For official documentation on querying Native Histograms in PromQL, see:

• PromQL histogram_quantile documentation
• PromQL histogram_fraction documentation
• PromQL histogram_sum documentation
• PromQL histogram_count documentation
• PromQL histogram_count documentation
• PromQL histogram_stddev and histogram_stdvar documentation

Dashboard migration & rollback strategy

Recommended migration workflow

To safely transition your monitoring infrastructure to Native Histograms without breaking existing alerts or dashboards, I recommend a four-step migration workflow:

1. Enable Both Formats: In your Prometheus 3.x scrape config, set scrape_native_histograms: true AND always_scrape_classic_histograms: true so both formats are collected safely during transition
2. Migrate Queries: Update your Grafana dashboards and Prometheus alerting rules from classic quantile queries (histogram_quantile(..._bucket...)) to native histogram queries (histogram_quantile(...)), and replace references to classic _count and _sum series with histogram_count(...) and histogram_sum(...)
3. Verify in Staging/Production: Validate that all dashboards and SLO alerts fire and graph correctly using the new native histogram queries
4. Unlock ~10x Storage Savings: Once migration is complete, set always_scrape_classic_histograms: false. Prometheus will stop ingesting the static _bucket, _count, and _sum time series, reducing your histogram time series count by up to 90%!

Opt-out and rollback flexibility

Because native histograms are dual-exposed, using them is entirely opt-in from a collector perspective:

• Instant Collector Rollback: If you need to stop ingesting native histograms, simply set scrape_native_histograms: false in your Prometheus job configuration. No Kubernetes restart is required, and Prometheus will immediately resume scraping only the classic format without data loss
• Component Feature Gate Rollback: Administrators can also disable the feature gate on Kubernetes components using --feature-gates=NativeHistograms=false (requires component restart)

What's next & how to get involved

As native histograms progress toward General Availability (GA) in future Kubernetes releases, SIG Instrumentation will continue evaluating ecosystem readiness, performance characteristics, and long-term plans for eventually deprecating static classic buckets once native histogram adoption becomes ubiquitous across the monitoring community.

• Read the KEP-5808 page or the KEP GitHub issue to learn more.
• Read the Prometheus Native Histograms specification and PromQL querying functions documentation
• Get involved with SIG Instrumentation on Slack in #sig-instrumentation or join the weekly SIG meetings

Acknowledgements

A huge thank you to contributors across SIG Instrumentation and component owners who collaborated on the design, implementation, testing, and review of native histograms in Kubernetes!

11 Sep 2026 6:30pm GMT

In Kubernetes, resource allocation has historically been a static decision made during a Pod's initial scheduling and placement. With the graduation of the core in-Place Pod resize feature to General Availability in v1.35, application developers and cluster operators gained the powerful ability to dynamically adjust CPU and memory allocations of running containers without incurring disruptive restarts or application downtime.

However, in-place resizing introduced a unique resource scheduling gap: if a running Pod requested a resource scale-up that exceeded the host node's allocatable headroom, the Kubelet was forced to mark the request as Deferred. The Pod would remain parked in this state indefinitely, waiting for resources on the node to naturally free up.

To bridge this scheduling gap, Kubernetes v1.37 introduces scheduler preemption for in-place Pod resize (Alpha), behind the InPlacePodVerticalScalingSchedulerPreemption feature gate. This feature allows the Kubernetes scheduler to actively free up capacity on a fully-utilized node by preempting lower-priority workloads, enabling the pending in-place resizes of critical, higher-priority applications to succeed.

The "deferred" resize challenge

To understand why this preemption mechanism is needed, it is helpful to look at how Kubernetes handles running Pod resizing. When a user or controller (such as the Vertical Pod Autoscaler) updates the resource requests of an active container, the Kubelet evaluates whether the underlying node has enough spare allocatable capacity to fulfill the increase.

If the node's resources are fully utilized and cannot satisfy the new limits, the Kubelet sets the container's resizeStatus (reported in the Pod's status.containerStatuses[]) to Deferred. Unlike an Infeasible resize request (which is immediately rejected because it exceeds physical machine boundaries, namespace limit ranges, or admission quotas) a Deferred status indicates that the request is valid but is temporarily unable to be actuated, waiting until node capacity becomes available.

Before the introduction of this preemption mechanism, a Pod's in-place resize scale-up request could become permanently blocked if the node was heavily utilized. Even when a critical application (such as an in-memory database or a real-time web server) required more memory to prevent an imminent out-of-memory (OOM) crash, and the node lacked free capacity, the resize remained Deferred.

In this scenario, cluster administrators had limited choices:

1. Manually evict lower-priority Pods from the node to clear resource headroom.
2. Rely on the cluster autoscaler to eventually spin up a larger node and reschedule the Pod. However, this is an operation that is highly disruptive and violates the core "no restart" value proposition of in-place scaling.
3. Rely on a custom autoscaling solution, for example a cluster autoscaler that can trigger dynamic node resizing operations itself.

Because the kube-scheduler was unaware of deferred resizes on running Pods, it could not leverage standard priority-based preemption to evict lower-priority workloads and make room for the higher-priority running Pod's resource growth.

Why this matters

In production Kubernetes environments, cluster administrators strive to maximize resource utilization and efficiency. A common strategy is to bin-pack unused capacity on not-yet-full nodes with lower-priority workloads, such as batch jobs, background data processing, or best-effort tasks.

Without scheduler preemption for in-place resizing, this created a major operational dilemma. If lower-priority workloads consumed the remaining headroom on a node, higher-priority applications running on that same node would become blocked (Deferred) when they needed to scale up to handle sudden traffic surges or memory spikes. Operators were forced to choose between running low-utilization clusters with idle buffer capacity or risking that critical workloads could not resize when needed.

With scheduler preemption for in-place Pod resize, you can confidently bin-pack unused space across your clusters with lower-priority workloads without worrying about them degrading higher-priority Pods or blocking their scale-up requests. If a high-priority workload requires an in-place resize that exceeds available node capacity, the scheduler automatically preempts the lower-priority Pods to clear headroom. You achieve high cluster utilization and cost efficiency while preserving the responsiveness and reliability of critical services.

Architectural mechanics: How it works

Scheduler preemption for in-place Pod resize integrates directly into the core scheduling cycle to coordinate resources dynamically and safely.

Centralized scheduler tracking

The kube-scheduler monitors the cluster for running Pods with a Deferred resize status condition. Normally, Pods with spec.nodeName populated are considered successfully placed and bypass the active scheduling queue. Under this feature gate, the scheduler intercepts Pods carrying the Deferred condition, permitting them to remain in active scheduling evaluations specifically to trigger preemption. The scheduler maintains continuous tracking of these Pods until the Kubelet successfully completes the resize actuation.

Single-node preemption boundary

Unlike placement preemption, which evaluates all nodes in a cluster to find the best scheduling fit, preemption for in-place resizing is strictly localized to the Pod's currently assigned node. The scheduler identifies eligible lower-priority "victim" Pods on the same host and initiates their graceful eviction, freeing up local capacity. Preemption is strictly scoped to the same node where the deferred Pod is running; if a node cannot accommodate the resize even after evicting all eligible lower-priority workloads, the resize remains in the Deferred state.

Resource reservation safety

To prevent scheduling races and double-allocation, the scheduler treats resources requested for a resize as already consumed. This enables the Kubelet to actuate the resize once the preemption takes effect.

Separation of concerns & critical admission

When a node is under resource pressure, the Kubelet includes a local mechanism known as the critical Pod admission handler. During initial Pod admission, if a critical system Pod arrives on a node that lacks spare capacity, this local handler can directly evict lower-priority Pods on that node to guarantee admission for the critical workload.

A significant architectural benefit of this new feature is the strict separation of concerns between the Kubelet and the scheduler. Under the InPlacePodVerticalScalingSchedulerPreemption feature gate, the Kubelet's critical Pod admission handler does not perform local preemption checks or trigger local evictions for in-place resizing operations. Instead, the Kubelet defers the request and delegates the preemption decision entirely to the scheduler. This guarantees that a single, centralized orchestrator manages all resize-related preemption logic, respecting global priorities, Pod disruption budgets (PDBs), and graceful termination policies.

Managing competing updates & races

If a competing, higher-priority resize request is submitted for another running Pod on the same node during an active preemption cycle, the Kubelet prioritizes the higher-priority request. The scheduler is designed to observe these updates and will dynamically trigger a new round of preemption if more capacity is required to fulfill the new state.

Node-level preemption configuration

Administrators and automated controllers (such as a cluster autoscaler) can disable preemption specifically for in-place resizes on particular nodes. This is configured using the new spec.podPreemptionPolicy field in the Node Spec:

apiVersion: v1
kind: Node
metadata:
 name: batch-workload-node
spec:
 podPreemptionPolicy:
 disableResizePreemption:
 - "cluster-autoscaler.kubernetes.io/disable-preemption"
 - "operator.example.com/policy-override"

An example use case for this policy is when a controller would prefer to size down other pods or dynamically adjust the node capacity itself when possible, only enabling scheduler preemption as a last resort.

Try it out!

To utilize scheduler preemption for in-place Pod resize:

• Your cluster must be running Kubernetes v1.37 or later across both the control plane and all worker nodes.
• The InPlacePodVerticalScalingSchedulerPreemption feature gate must be enabled across all control plane components (kube-apiserver, kube-scheduler) and the kubelet.

Mini-tutorial: Observe resize preemption in action

To see this feature in action locally, you can test scheduler preemption on a single-node kind cluster with constrained CPU headroom.

1. Create a kind cluster with scheduler resize preemption enabled

Create a kind cluster configuration file named kind-config.yaml with the InPlacePodVerticalScalingSchedulerPreemption feature gate enabled:

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
 InPlacePodVerticalScalingSchedulerPreemption: true

Create the cluster using this configuration, passing the --image flag to ensure the cluster is running Kubernetes v1.37 (or later):

kind create cluster --config kind-config.yaml --image kindest/node:v1.37.0

Once your cluster is ready, inspect the node to check how many allocatable CPU cores it has:

kubectl get nodes -o custom-columns=NAME:.metadata.name,ALLOCATABLE_CPU:.status.allocatable.cpu

In a standard local kind environment, the output shows 8 allocatable CPU cores:

NAME ALLOCATABLE_CPU
kind-control-plane 8

2. Create PriorityClasses and deploy Pods

Create two PriorityClasses and deploy a low-priority Pod (requesting 3 CPU) alongside a high-priority Pod (requesting 4 CPU). Together, these workloads consume 7 of the 8 available CPU cores, leaving 1 CPU of free allocatable headroom on the node.

# preemption-demo.yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
 name: high-priority
value: 1000000
globalDefault: false
description: "High priority workload"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
 name: low-priority
value: 1000
globalDefault: false
description: "Low priority workload"
---
apiVersion: v1
kind: Pod
metadata:
 name: low-priority-pod
spec:
 priorityClassName: low-priority
 containers:
 - name: worker
 image: nginx
 resources:
 requests:
 cpu: "3"
 memory: "500Mi"
 limits:
 cpu: "3"
 memory: "500Mi"
---
apiVersion: v1
kind: Pod
metadata:
 name: high-priority-pod
spec:
 priorityClassName: high-priority
 containers:
 - name: app
 image: nginx
 resources:
 requests:
 cpu: "4"
 memory: "1Gi"
 limits:
 cpu: "4"
 memory: "1Gi"

Save this manifest to preemption-demo.yaml and apply it:

kubectl apply -f preemption-demo.yaml

Wait until both Pods are running on the node:

kubectl get pods

Output:

NAME READY STATUS RESTARTS AGE
high-priority-pod 1/1 Running 0 9s
low-priority-pod 1/1 Running 0 9s

3. Request an in-place scale-up

Patch the high-priority Pod to increase its CPU request from 4 to 6 (+2 CPU delta). Because only 1 CPU of headroom is free on the node, this resize request exceeds remaining allocatable capacity:

kubectl patch pod high-priority-pod --subresource resize --patch \
 '{"spec":{"containers":[{"name":"app", "resources":{"requests":{"cpu":"6"}, "limits":{"cpu":"6"}}}]}}'

4. Inspect the preemption event on the low-priority Pod

With InPlacePodVerticalScalingSchedulerPreemption enabled, the scheduler intercepts the Deferred resize condition on high-priority-pod and targets low-priority-pod for preemption.

To verify that the scheduler actively preempted the low-priority Pod, inspect its events:

kubectl get events --field-selector involvedObject.name=low-priority-pod

In the event stream (or via kubectl describe pod low-priority-pod), you will see a Preempted event emitted by the scheduler:

LAST SEEN TYPE REASON OBJECT MESSAGE
5s Normal Preempted pod/low-priority-pod Preempted by pod 97dba925-6b5f-4e2f-99f9-d51c30016586 on node kind-control-plane
5s Normal Killing pod/low-priority-pod Stopping container worker

5. Trace the resize event lifecycle on the high-priority Pod

Next, inspect the event history on high-priority-pod to observe how the resize progressed from being deferred to successfully completed:

kubectl get events --field-selector involvedObject.name=high-priority-pod

You will observe a sequence of events as the Kubelet coordinates with the scheduler:

LAST SEEN TYPE REASON OBJECT MESSAGE
33s Warning ResizeDeferred pod/high-priority-pod Pod resize OutOfcpu: {"containers":[{"name":"app","resources":{"limits":{"cpu":"6","memory":"1Gi"},"requests":{"cpu":"6","memory":"1Gi"}}}],"generation":2,"error":"Node didn't have enough resource: cpu, requested: 6000, used: 3950, capacity: 8000"}
32s Normal ResizeStarted pod/high-priority-pod Pod resize started: {"containers":[{"name":"app","resources":{"limits":{"cpu":"6","memory":"1Gi"},"requests":{"cpu":"6","memory":"1Gi"}}}],"generation":2}
32s Normal ResizeCompleted pod/high-priority-pod Pod resize completed: {"containers":[{"name":"app","resources":{"limits":{"cpu":"6","memory":"1Gi"},"requests":{"cpu":"6","memory":"1Gi"}}}],"generation":2}

1. ResizeDeferred: The Kubelet initially marks the resize request as deferred (Warning) due to insufficient CPU headroom on the node (OutOfcpu).
2. ResizeStarted: Once the scheduler preempts low-priority-pod and capacity is released, the Kubelet accepts the new allocation and begins actuating the resize.
3. ResizeCompleted: The Kubelet successfully updates container cgroup limits via the container runtime without restarting the Pod.

Finally, verify that the allocated CPU on the container reflects the new request (appending {"\n"} to the JSONPath query ensures a trailing newline in your terminal):

kubectl get pod high-priority-pod -o jsonpath='{.status.containerStatuses[0].allocatedResources.cpu}{"\n"}'

Output:

6

This confirms that the in-place resize succeeded.

Getting involved

This feature represents a major step forward for resource scheduling, bringing enterprise-grade density control and workload prioritization to dynamic resource scaling. We invite cluster operators, platform architects, and developers to enable the InPlacePodVerticalScalingSchedulerPreemption feature gate in their testing environments and share feedback.

If you want to share your experience with this feature, please get in touch with the community via SIG Scheduling or SIG Node channels!

10 Sep 2026 6:30pm GMT

Kubernetes has many ways to describe what is happening on a Node. Readiness, taints, Pod state, labels, annotations, and provider-specific APIs each expose part of the picture. What has been missing is a shared, Kubernetes-owned way to say that a Node is draining, undergoing maintenance, or undergoing Graceful Node Shutdown.

Kubernetes v1.37 introduces five well-known Node conditions that provide that description:

• DrainInProgress
• Drained
• MaintenancePlanned
• MaintenanceInProgress
• GracefulNodeShutdownInProgress

The new Node lifecycle conditions

Maintenance can include hardware or software rollout, remediation, decommissioning, or debugging. Whether maintenance requires a drain depends on its impact. A Kubernetes upgrade usually should follow a drain, while a kernel live patch might not need one.

Like other Node conditions, each lifecycle condition uses status to report whether the observation is active:

• True: the lifecycle state is currently observed.
• False: the lifecycle state is not currently observed.
• Unknown: Kubernetes cannot determine whether the lifecycle state is active.

The reason provides a stable, machine-readable cause for the current status, and message can provide additional human-readable detail.

For example, an authorized maintenance controller could publish:

# Node .status excerpt
status:
 conditions:
 - type: MaintenancePlanned
 status: "True"
 reason: MaintenanceWindow
 lastTransitionTime: "2026-12-09T12:00:00Z"
 message: "Hardware maintenance is scheduled for this Node"

What changes in Kubernetes v1.37

The v1.37 release reserves these names as well-known NodeConditionType constants and introduces the Alpha NodeLifecycleConditions feature gate, which is disabled by default. In v1.37 the gate is effectively a no-op: it does not restrict who can set these conditions, and no core component reads them. It exists so that the built-in behavior planned for future releases - controllers that consume these conditions - can be opted into when it arrives. You do not need to enable it to start publishing the conditions today.

For this release, an administrator or an administrator-authorized controller is responsible for setting and clearing the lifecycle conditions.

In this first release, no core workload controller changes its behavior based on these conditions, but an administrator can publish them to communicate maintenance and drains to cluster users.

How to use lifecycle conditions today

The immediate value is operational clarity. Administrators and lifecycle automation can use these conditions as a common status channel for Node lifecycle work that already happens today.

For example, maintenance automation can set MaintenancePlanned when a future maintenance window is scheduled, then set MaintenanceInProgress when work starts. Drain automation can set DrainInProgress when it begins evicting Pods and Drained when the administrator's selected drain criteria have been met. The GracefulNodeShutdownInProgress condition can report that Graceful Node Shutdown is in progress on the Node.

The recommended pattern is to use lifecycle conditions to report status, while lifecycle operations are managed through other mechanisms. Continue to use existing Kubernetes mechanisms such as kubectl cordon, kubectl drain, taints, and workload-specific controls to change scheduling or eviction behavior. Use lifecycle conditions to make the state of that work visible to people, dashboards, alerts, and automation that choose to consume the signal.

When setting a condition, use True while the lifecycle state is active. Set the condition to False, or remove it, when the state is no longer active. Use a stable reason value and a clear message so that both people and automation can understand why the condition changed. Cluster administrators should also decide which component owns each lifecycle condition to avoid conflicting writes.

Why a shared signal matters

Node lifecycle affects components across the cluster. The kubelet, node lifecycle controller, workload controllers, scheduler, autoscalers, storage operators, and external maintenance systems all need some understanding of what is happening to a Node.

Today, each component has to reconstruct that understanding from indirect signals. One controller might look at Node readiness, another at taints, and another at Pods that are terminating or missing. Infrastructure providers and operators often add their own labels or annotations.

Those signals remain useful for their intended purposes, but they do not answer the same question. A taint can influence scheduling or eviction, for example, but it does not attest that a drain is in progress or that an administrator's drain criteria have been met. A NotReady Node does not explain whether the cause is an unexpected failure, a graceful shutdown, or planned maintenance.

Without shared lifecycle context, independently correct components can make conflicting decisions. A DaemonSet controller can replace a Pod that the kubelet intentionally terminated during graceful shutdown. A Job controller can wait indefinitely for a terminal Pod phase on a Node that an administrator is removing. A storage operator might learn about maintenance only after drain has already started.

The new conditions provide a stable place on the Node for that missing context, as part of the larger effort to enhance Node Lifecycle management.

The foundation for lifecycle-aware Kubernetes

The value of a shared signal comes from what can consume it - core controllers, administrators, or the ecosystem of lifecycle projects. Follow-up enhancements can build on the conditions without every component inventing a different way to infer Node lifecycle state.

Consider a long-standing DaemonSet rollout edge case. A Node that is broken or undergoing maintenance can remain unavailable for reasons unrelated to the new DaemonSet revision. That Node still consumes the rollout's availability budget, which can slow or block the controller from progressing the rollout on healthy Nodes.

The DaemonSet controller knows that a Pod is unavailable, but it cannot tell whether the new revision failed or an administrator intentionally took the Node out of service. Readiness, taints, and Pod state expose pieces of the situation, but none provides authoritative maintenance context.

The MaintenanceInProgress condition creates a Kubernetes-owned place to publish that context. Future work can define how the DaemonSet controller uses it for rollout ordering, availability accounting, and status reporting. Those behaviors still require careful design, but the goal is for administrators to no longer have to manually adjust the rollout.

Future expansions and getting involved

Node lifecycle is a cross-cutting problem. Solving it starts with components sharing enough context to make compatible decisions. The next stage is to build on Node Lifecycle Conditions to improve scenarios such as Graceful Node Shutdown, drain, and maintenance. Longer-term lifecycle coordination may require explicit ownership, locking, and potentially a dedicated API.

The Kubernetes ecosystem already includes many solutions for Node maintenance, remediation, drain, autoscaling, and fleet management. The experience behind those projects is essential to building a foundation that works across different environments and operational models. The Node Lifecycle Working Group, SIG Node, and SIG Apps invite maintainers and users to share their use cases and ideas to shape the future work.

Follow the work through KEP-5683: Node Lifecycle Conditions. To participate in our discussions, join one of our groups:

• Node Lifecycle Working Group
• SIG Node
• SIG Apps

09 Sep 2026 6:30pm GMT

AI/ML and complex batch workloads continue to push the boundaries of Kubernetes scheduling. Following the foundational workload-centric enhancements introduced in previous releases, Kubernetes v1.37 delivers the next major milestone in the Workload-Aware Scheduling (WAS) journey. In this release, the core Workload and PodGroup APIs-enabling gang scheduling-along with Workload-Aware Preemption (WAP) and shared DRA ResourceClaims for PodGroups, all graduate to Beta, solidifying their role in the Kubernetes ecosystem.

To address the hierarchical scheduling requirements of modern high-performance distributed workloads, v1.37 introduces the new CompositePodGroup API. This new API allows expressing multi-level topology constraints, gang scheduling, and preemption policies for complex, heterogeneous groups of Pods. Crucially, this architectural expansion unlocks native scheduling support for advanced workload structures commonly managed by higher-order extension APIs such as JobSet and LeaderWorkerSet (LWS).

Alongside these API additions, v1.37 focuses on streamlining adoption by introducing a new set of controller integration APIs and the workloadbuilder Go library. These provide standardized building blocks that significantly simplify how out-of-tree controllers can integrate with WAS capabilities. Utilizing these new tools, the native Job controller integration has been upgraded to fully consume the expanded WAS APIs-enabling advanced scheduling policies, flexible disruption modes, and topology-aware scheduling for standard batch workloads.

Gang scheduling and Workload / PodGroup APIs

Kubernetes v1.37 delivers a major milestone: Workload / PodGroup APIs and gang scheduling are officially graduating to Beta. This graduation signals that native, "all-or-nothing" scheduling for workloads is solidifying for wider adoption.

Key updates to the API and gang scheduling algorithm in this release include:

Beta graduation and API versioning changes

The core Workload and PodGroup APIs have been promoted to v1beta1, meaning they are now one step away from General Availability (GA). For early adopters who have been testing these features, take note of the alpha versioning transition: v1alpha2 has been entirely replaced by v1alpha3. This transition introduces breaking changes designed to clean up the API structure around disruptionMode.

Native PodGroup queueing

A significant under-the-hood improvement in v1.37 makes the PodGroup a first-class citizen in the scheduling queue. Previously, even if belonging to a PodGroup, all member Pods were queued individually. Now, only the top-level PodGroup object is queued. This ensures all Pods share the same queueing behavior and lays the groundwork for more advanced PodGroup queueing strategies in the future.

Dynamic elasticity with minCount mutability

In earlier iterations, the minCount field, which dictates the minimum number of Pods required to successfully schedule a PodGroup, was strictly immutable. In v1.37, minCount is now mutable. This API change unlocks flexibility for elastic workloads. Controllers can now dynamically adjust the minimum required size of a gang on the fly, allowing workloads to gracefully degrade or expand without interrupting already-scheduled Pods.

Workload-aware preemption

In Kubernetes v1.37 the separate WorkloadAwarePreemption feature gate for workload-aware preemption was merged into the GenericWorkload feature gate, becoming a core part of the gang scheduling effort.

While the core concepts of workload-aware preemption stay the same, there are some differences between the v1.36 and v1.37 releases:

Performance and optimality

To check whether a preemptor can fit in the cluster thanks to preemption, the scheduler simulates the removal of all potential victims and re-runs the scheduling algorithm. After that it tries to reprieve as many victims as possible. In the v1.36 release, the scheduling algorithm was run for each victim reprieval, verifying whether with the victim reprieved, the algorithm can still find a valid placement for the preemptor. In v1.37, the scheduling algorithm is run only once and the preemptor Pods are assumed based on its output. Later, the reprieval checks whether a victim can still run in its place with the preemptor assumed.

PodGroup as a victim

One of the limitations of v1.36 was the fact that the default preemption for single Pods was not aware of PodGroups and was not respecting their disruptionMode fields, allowing for disruption of single Pods even when the PodGroup had disruptionMode: {all: {}} set. Kubernetes v1.37 removes this limitation; the default preemption now respects the PodGroup disruptionMode field.

Rename of the disruptionMode fields

During the promotion of the API to Beta, the disruptionMode field was changed to decouple its naming from the PodGroup object, allowing consistent naming across PodGroups and CompositePodGroups. The modes changed as follows: PodGroup became all, and Pod became single.

Support for preemptionPolicy

In v1.36, the PodGroup does not have a preemptionPolicy field. The PodGroup can perform preemption as long as none of the Pods forming it has preemptionPolicy: Never set. In v1.37, when the PodGroupPreemptionPolicy feature gate is enabled, a PodGroup also has a preemptionPolicy field. It serves as an authoritative field for whether a PodGroup can perform preemption.

CompositePodGroup API

In Kubernetes v1.36, workload-aware scheduling established a clean separation between static workload templates (Workload) and runtime group state (PodGroup), but the supported scheduling policies were limited to a single, flat group. The CompositePodGroup API, introduced in Kubernetes v1.37, extends this model to support hierarchical scheduling requirements.

This API allows its consumers to express multi-level scheduling requirements by organizing a workload in a tree-shaped hierarchy consisting of CompositePodGroup and PodGroup objects. Each CompositePodGroup carries policies and constraints that apply to other groups (CompositePodGroups and/or PodGroups), similar to how PodGroups govern scheduling behavior for a flat group of Pods. The scheduler treats such a hierarchy as a single scheduling unit and aims to satisfy the requirements specified by every group within that hierarchy.

Defining a workload hierarchy

To express multi-level scheduling requirements, you define a hierarchy of templates in a Workload object. Controllers then create the corresponding CompositePodGroup and PodGroup objects from that hierarchy.

To support this, the Workload API is extended with the spec.compositePodGroupTemplates field. Each CompositePodGroupTemplate defines a template for a parent CompositePodGroup and directly nests the templates (podGroupTemplates and/or compositePodGroupTemplates) from which its child groups derive.

Below is a sample Workload object that defines a two-level template hierarchy:

apiVersion: scheduling.k8s.io/v1beta1
kind: Workload
metadata:
 name: example-workload
 annotations:
 kubernetes.io/description: "Two-level workload hierarchy requiring 4 worker Pods and 1 driver Pod to schedule together."
spec:
 compositePodGroupTemplates:
 - name: workload-root
 schedulingPolicy:
 gang:
 minGroupCount: 2
 podGroupTemplates:
 - name: workers
 schedulingPolicy:
 gang:
 minCount: 4
 - name: driver
 schedulingPolicy:
 gang:
 minCount: 1

After creating example-workload, a controller can stamp out the corresponding runtime group objects from these templates:

1. A root CompositePodGroup that references the workload-root template in example-workload and carries its group-level scheduling policy (gang scheduling with minGroupCount: 2):

   apiVersion: scheduling.k8s.io/v1alpha3
   kind: CompositePodGroup
   metadata:
    name: example-root-group
    annotations:
    kubernetes.io/description: "Root group coordinating gang scheduling across child worker and driver PodGroups."
   spec:
    workloadRef:
    workloadName: example-workload
    templateName: workload-root
    schedulingPolicy:
    gang:
    minGroupCount: 2
   

2. Two child PodGroup objects (example-workload-workers and example-workload-driver) that reference their respective leaf templates in example-workload and link to the root group via parentCompositePodGroupName:

   apiVersion: scheduling.k8s.io/v1beta1
   kind: PodGroup
   metadata:
    name: example-workload-workers
    annotations:
    kubernetes.io/description: "Worker group requiring at least 4 Pods to be scheduled together."
   spec:
    parentCompositePodGroupName: example-root-group
    workloadRef:
    workloadName: example-workload
    templateName: workers
    schedulingPolicy:
    gang:
    minCount: 4
   ---
   apiVersion: scheduling.k8s.io/v1beta1
   kind: PodGroup
   metadata:
    name: example-workload-driver
    annotations:
    kubernetes.io/description: "Driver group requiring 1 Pod to schedule alongside the workers."
   spec:
    parentCompositePodGroupName: example-root-group
    workloadRef:
    workloadName: example-workload
    templateName: driver
    schedulingPolicy:
    gang:
    minCount: 1
   

How multi-level gang scheduling works

To schedule a hierarchical workload, kube-scheduler evaluates the entire group tree as a unified scheduling unit:

• Recursive evaluation: The scheduler traverses the hierarchy from the root CompositePodGroup down to the leaf PodGroup objects. At each level, a parent CompositePodGroup is considered schedulable only when its child groups satisfy its scheduling policy (for example, placing at least minGroupCount of child groups when using the gang policy), while each leaf PodGroup must satisfy its own Pod-level policy (for example, placing at least minCount of member Pods when using the gang policy).
• All-or-nothing scheduling: Once a valid combination of child groups is found that satisfies the requirements of the root CompositePodGroup, the Pods across the entire hierarchy are scheduled and bound atomically. If the root group cannot satisfy its policy constraints, the entire hierarchy remains unschedulable and no Pods are bound, preventing partial deployments and deadlocks.

Workload-aware preemption for the CompositePodGroup API

Kubernetes v1.37 extends workload-aware preemption to support CompositePodGroup hierarchies as well. Specifically, if a CompositePodGroup cannot be scheduled due to insufficient capacity in the cluster, the scheduler can invoke preemption to evict lower-priority workloads in order to fit the Pods belonging to that CompositePodGroup.

A CompositePodGroup can be selected for preemption as well. To specify the desired behavior during preemption, workload owners can specify an appropriate disruptionMode in the CompositePodGroup spec:

• single: Allows individual child groups within the CompositePodGroup to be preempted and disrupted independently. This is the behavior when disruptionMode is not set.
• all: Enforces "all-or-nothing" disruption semantics across the entire CompositePodGroup hierarchy. If any Pod within the descendant subtree must be preempted, the scheduler evicts all Pods across the entire hierarchy together.

Topology-aware scheduling

In Kubernetes v1.37, topology-aware scheduling expands to support complex, multi-level workload hierarchies and delivers performance improvements for existing single-level deployments.

Multi-level topology-aware scheduling

In Kubernetes v1.36, we introduced foundational topology-aware scheduling, allowing you to define co-location constraints directly on a PodGroup. While effective for single-level groupings, complex distributed workloads-such as large-scale AI/ML training, JobSet deployments, or disaggregated inference via LeaderWorkerSet (LWS)-often require co-location across multiple levels of cluster infrastructure simultaneously.

For example, an entire workload may need to run within a single availability zone, while different parts of that workload (such as specific worker groups or driver processes) require strict co-location within specific server racks.

In Kubernetes v1.37, alongside the new CompositePodGroup API (scheduling.k8s.io/v1alpha3), topology-aware scheduling expands to support multi-level topology-aware scheduling. You can now express complex co-location requirements by specifying topology constraints at different levels of a group hierarchy.

Top-down topology constraint resolution

During hierarchical scheduling, the kube-scheduler resolves multi-level topology constraints in a top-down manner. Specifically, topology domains that are considered during the scheduling of a child group are confined within a topology domain that corresponds to the placement assumed by the parent group.

Configuration and runtime execution

Using the updated Workload API (scheduling.k8s.io/v1beta1), you can configure multi-level topology constraints directly within compositePodGroupTemplates. In the example below, the parent template constrains the overall workload to a single availability zone (topology.kubernetes.io/zone), while child templates for workers and driver constrain their respective Pods to server racks (topology.example.com/rack) within that selected zone:

apiVersion: scheduling.k8s.io/v1beta1
kind: Workload
metadata:
 name: multi-level-tas-workload
 namespace: job-ns
 annotations:
 kubernetes.io/description: "Workload defining zone-level co-location for the root group and rack-level co-location for child groups."
spec:
 compositePodGroupTemplates:
 - name: root
 schedulingPolicy:
 gang:
 minGroupCount: 2
 schedulingConstraints:
 topology:
 - key: topology.kubernetes.io/zone
 podGroupTemplates:
 - name: workers
 schedulingPolicy:
 gang:
 minCount: 8
 schedulingConstraints:
 topology:
 - key: topology.example.com/rack
 - name: driver
 schedulingPolicy:
 gang:
 minCount: 1
 schedulingConstraints:
 topology:
 - key: topology.example.com/rack

When a controller creates an instance of this workload at runtime, it spawns the corresponding runtime objects from these templates:

1. The root CompositePodGroup referencing the root template, carrying the availability zone topology constraint and the hierarchical gang scheduling policy.
2. The two child PodGroup objects (tas-workload-workers and tas-workload-driver), each referencing the root CompositePodGroup as their parent group via the parentCompositePodGroupName spec field:

apiVersion: scheduling.k8s.io/v1alpha3
kind: CompositePodGroup
metadata:
 name: tas-workload-root
 namespace: job-ns
 annotations:
 kubernetes.io/description: "Root group constraining the entire workload to a single availability zone."
spec:
 workloadRef:
 workloadName: multi-level-tas-workload
 templateName: root
 schedulingPolicy:
 gang:
 minGroupCount: 2
 schedulingConstraints:
 topology:
 - key: topology.kubernetes.io/zone
---
apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
 name: tas-workload-workers
 namespace: job-ns
 annotations:
 kubernetes.io/description: "Worker group requiring 8 Pods co-located within a single rack in the selected zone."
spec:
 parentCompositePodGroupName: tas-workload-root
 workloadRef:
 workloadName: multi-level-tas-workload
 templateName: workers
 schedulingPolicy:
 gang:
 minCount: 8
 schedulingConstraints:
 topology:
 - key: topology.example.com/rack
---
apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
 name: tas-workload-driver
 namespace: job-ns
 annotations:
 kubernetes.io/description: "Driver group requiring 1 Pod placed in a rack within the selected zone."
spec:
 parentCompositePodGroupName: tas-workload-root
 workloadRef:
 workloadName: multi-level-tas-workload
 templateName: driver
 schedulingPolicy:
 gang:
 minCount: 1
 schedulingConstraints:
 topology:
 - key: topology.example.com/rack

During scheduling, the scheduler evaluates multiple candidate availability zones across the cluster for tas-workload-root. For each candidate zone, it subdivides the nodes by rack topology to explore feasible rack placements for tas-workload-workers and tas-workload-driver strictly within that zone, systematically evaluating multiple combinations across available zones and racks before making a scheduling decision.

By allowing topology constraints to be modeled hierarchically, Kubernetes v1.37 provides a structured way to express multi-level co-location requirements across complex cluster infrastructures.

Performance improvements for single-level TAS

Alongside the Alpha introduction of multi-level hierarchies, Kubernetes v1.37 reduces the cost of placement evaluation for existing single-level topology-aware scheduling. We are continuously working to optimize the efficiency of placement evaluation algorithms in kube-scheduler and plan to deliver further performance improvements in future releases.

Controller Integration APIs

Kubernetes v1.37 introduces new standard building blocks so that every controller can expose the same scheduling primitives in their own APIs, and share the same logic for translating them into scheduling objects. These primitives express specific scheduling behaviors - such as policies or disruption logic - while leaving the field naming flexible for each controller. A prime example of this is the native Job controller, which we detail in the next section.

Types prefixed with WorkloadPodGroup describe a leaf group of Pods; types prefixed with WorkloadCompositePodGroup describe a group of groups. A controller embeds them verbatim into its own API, under whatever field name fits its domain:

• WorkloadPodGroupSchedulingPolicy - either basic, meaning standard Pod-by-Pod scheduling, or gang with a minCount. The composite variant takes a minGroupCount instead.
• WorkloadPodGroupSchedulingConstraints - the topology constraints (topology[].key) the group's Pods must be co-located within.
• WorkloadPodGroupDisruptionMode - single or all, with the preemption semantics described earlier in this post.
• WorkloadPodGroupResourceClaim - the ResourceClaims shared across the group.

Only the shapes are shared, so controllers retain full autonomy over how they name and nest these fields in their own APIs.

The workloadbuilder library turns that intent into the scheduling objects. A controller describes its workload as a tree of WorkloadItem nodes - a node with children compiles to a CompositePodGroupTemplate, a node without children to a PodGroupTemplate - and attaches its own defaults plus the user-supplied building blocks to each node. From there, Validate() reports problems back at the exact field path within the controller's own API, BuildWorkload() compiles the tree into a Workload, and NewPodGroup() and NewCompositePodGroup() stamp out the runtime group objects.

Validation is deny-by-default: a controller declares the policies and disruption modes it actually supports through AllowedPolicies and AllowedDisruptionModes, and anything outside those lists is rejected. Building blocks added in future releases therefore stay unavailable until a controller explicitly opts into them.

For hierarchical workloads where a parent controller owns the Workload and delegates group creation to its children, NewBuilderFromExistingWorkload lets a child materialize only its own PodGroup from the parent's Workload.

Neither the building blocks nor the library have a feature gate of their own; they become user-visible through whichever controller adopts them. The native Job controller is the first to do so, and we detail it in the next section.

Integration with the Job controller

Building upon the new controller integration APIs, the Job API now features an explicit .spec.scheduling field, so you can declare how a Job should be scheduled instead of relying on the Job controller to infer it from the Job's shape. This expands support well beyond static, indexed, and fully-parallel Jobs.

.spec.scheduling is composed of the building blocks described above:

• schedulingPolicy - basic for standard Pod-by-Pod scheduling, or gang for all-or-nothing scheduling.
• schedulingConstraints - the topology domain the Job's Pods must be co-located within.
• disruptionMode - whether the Job's Pods can be preempted individually (single) or only as a whole (all).
• resourceClaims - the ResourceClaims shared by all of the Job's Pods.

For example:

apiVersion: batch/v1
kind: Job
metadata:
 name: distributed-training-job
 annotations:
 kubernetes.io/description: "Distributed Job using explicit WAS scheduling with gang policy and zone topology constraints."
spec:
 parallelism: 8
 completions: 8
 scheduling:
 schedulingPolicy:
 gang: {} # minCount omitted → defaults to parallelism (8)
 schedulingConstraints:
 topology:
 - key: topology.kubernetes.io/zone
 disruptionMode:
 all: {}
 template:
 spec:
 containers:
 ...

Omitting .spec.scheduling, or omitting schedulingPolicy within it, selects the basic policy, which behaves exactly like standard Job scheduling today.

For every Job it manages, the controller compiles this configuration into a Workload and a PodGroup owned by the Job, and sets .spec.schedulingGroup.podGroupName on each Pod it creates so the scheduler treats them as one group. Once created, .spec.scheduling is immutable, with one exception: schedulingPolicy.gang.minCount can be updated, which lets you resize a running gang.

DRA ResourceClaim support for workloads

As the core WAS APIs mature, so do their integrations with Dynamic Resource Allocation (DRA). Kubernetes v1.36 introduced the DRAWorkloadResourceClaims feature gate. The associated feature allows ResourceClaims to be replicated and reserved for entire PodGroups and shared by all their member Pods:

apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
 name: training-job-workers-pg
spec:
 ...
 resourceClaims:
 - name: pg-claim
 resourceClaimTemplateName: my-claim-template
---
apiVersion: v1
kind: Pod
metadata:
 name: topology-aware-workers-pg-pod
spec:
 ...
 schedulingGroup:
 podGroupName: training-job-workers-pg
 resourceClaims:
 - name: pg-claim
 resourceClaimTemplateName: my-claim-template

In Kubernetes v1.37, the DRAWorkloadResourceClaims feature gate graduated to Beta.

While the API and core functionality of the feature remain unchanged, one change eliminates some potentially surprising behavior when disabling the feature. Previously when one of a Pod's spec.resourceClaims referenced a ResourceClaimTemplate and matched one of its PodGroup's spec.resourceClaims and the DRAWorkloadResourceClaims feature gate was disabled, a ResourceClaim was created for the Pod instead of the PodGroup. In that scenario in v1.37, no ResourceClaim is created at all. This change prevents Kubernetes from creating a flood of ResourceClaims from a ResourceClaimTemplate and potentially exhausting DRA resources when a claim intended to be shared by a whole PodGroup is replicated for each and every Pod in the group.

For more information, see the feature documentation.

What's next?

The Workload-Aware Scheduling Working Group (WG WAS) is currently finalizing its plans for the Kubernetes v1.38 release cycle. While the roadmap is still taking shape (stay tuned!), the following key initiatives are already planned:

• Graduation of Workload and PodGroup APIs to GA: Solidifying the core foundation of workload-aware scheduling as a stable Kubernetes API.
• Graduation of Topology-Aware Scheduling (TAS) and CompositePodGroup (CPG) to Beta: Bringing these advanced placement and hierarchical scheduling features to Beta stability.
• Graduation of controller integration building blocks to Beta: Further refining the integration APIs to ensure a robust developer experience.
• Increased adoption and integration: Expanding the ecosystem by integrating workload-aware scheduling with other controllers, with a particular focus on hierarchical orchestrators such as JobSet.
• Kueue Integration: Fostering closer alignment between WAS and Kueue. In the near term, we aim to ensure Kueue is fully aware of WAS features for seamless interoperability. In the long term, we envision Kueue leveraging WAS as its underlying engine for capabilities like gang-scheduling and topology-aware placement.

Getting started

Many of the workload-aware scheduling improvements are now available as Beta features in v1.37, while new advanced capabilities are introduced in Alpha. Both Beta and Alpha features here are disabled by default and require manual enablement.

Beta features:

• Workload API, gang scheduling, and preemption: The GenericWorkload feature gate (which now integrates gang scheduling and workload-aware preemption) is Beta and disabled by default on the kube-apiserver, kube-controller-manager and kube-scheduler. Ensure your manifests are updated to use the scheduling.k8s.io/v1beta1 API group.
• DRA ResourceClaim support for workloads: Enable the DRAWorkloadResourceClaims feature gate on the kube-apiserver, kube-controller-manager, kube-scheduler and kubelet.

Alpha features:

• Topology-aware scheduling: Enable the TopologyAwareWorkloadScheduling feature gate on the kube-apiserver and kube-scheduler.

• CompositePodGroup API: Enable the CompositePodGroup feature gate on the kube-apiserver, kube-controller-manager and kube-scheduler, and ensure the scheduling.k8s.io/v1alpha3 API version is enabled. Note that enabling CompositePodGroup on the kube-controller-manager also requires the TopologyAwareWorkloadScheduling feature gate to be enabled.

• Workload API integration with the Job controller: Enable the WorkloadWithJob feature gate on the kube-apiserver and kube-controller-manager.

• PodGroup preemptionPolicy: Enable the PodGroupPreemptionPolicy feature gate on the kube-apiserver and kube-scheduler.

Controller integration APIs:

The new workloadbuilder library is available to developers building both out-of-tree and in-tree controllers who want to integrate with WAS. It does not require a feature gate. You can explore the library and find usage examples directly in the kubernetes/component-helpers repository.

We encourage you to try out workload-aware scheduling in your test clusters and share your experiences to help shape the future of Kubernetes scheduling. You can send your feedback by:

• Reaching out via Slack (#wg-workload-aware-scheduling).
• Joining the WG Workload-Aware Scheduling or SIG Scheduling meetings.
• Filing a new issue in the Kubernetes repository.

Learn more

To dive deeper into the architecture and design of these features, read the KEPs:

• KEP-4671: Gang Scheduling Support in Kubernetes
• KEP-5710: Workload-aware preemption
• KEP-5732: Topology-aware workload scheduling
• KEP-6012: CompositePodGroup API
• KEP-6089: WAS: Controller Integration APIs
• KEP-5547: WAS: Integrate Workload APIs with Job controller
• KEP-5729: DRA: ResourceClaim Support for Workloads

08 Sep 2026 6:30pm GMT

Kubernetes v1.37 promotes the KubeletInUserNamespace feature gate to beta. With this feature enabled, all of the node components (kubelet, CRI and OCI runtimes, CNI plugins, and kube-proxy) can run as a non-root user on the host, using a Linux user namespace. This technique is also known as rootless mode. The work started as an experiment in 2018, and was merged into Kubernetes v1.22 (2021) as an alpha feature (Kubernetes Enhancement Proposal KEP-2033).

This feature should not be confused with user namespaces for pods (hostUsers: false with the UserNamespacesSupport feature gate, GA since v1.36), which puts pods in user namespaces but still runs the node components as root. These two features do not conflict. Moreover, they can be combined to nest Kubernetes inside Kubernetes without resorting to the full privileged: true.

Why run the node components in a user namespace?

Because the node components have historically had container-breakout vulnerabilities that could compromise full root privileges on the host.

Examples of such vulnerabilities include:

• CVE-2022-0811 ("cr8escape"): CRI-O could be tricked into setting arbitrary sysctls, such as kernel.core_pattern, resulting in arbitrary code execution as root on the host
• CVE-2023-27561: runc could be tricked into bypassing the masked paths of a container via a volume mount race, exposing the host's procfs files (a regression of CVE-2019-19921)
• CVE-2024-10220: the kubelet could be made to execute arbitrary commands as root via gitRepo volumes (gitRepo volumes had a similar vulnerability, CVE-2018-11235, back in 2018 too)
• CVE-2025-31133: runc could be tricked into bind-mounting attacker-controlled paths and writing to the host's procfs files, such as /proc/sysrq-trigger and /proc/sys/kernel/core_pattern
• CVE-2026-53488: containerd could be tricked into executing arbitrary commands on the host, via crafted labels in a container image

By running the node components in a user namespace, the potential damage is confined to the non-root user's account. Notably, an attacker cannot conceal their intrusion by modifying the kernel, the boot loader, or the firmware.

It should still be noted that user namespaces are not effective for mitigating vulnerabilities in the kernel itself. User namespaces should be used in conjunction with traditional hardening measures such as seccomp to prevent containers from invoking unnecessary system calls.

Use cases

• Production clusters: mitigate potential container-breakout vulnerabilities.
• Shared machines (e.g., HPC): users can deploy Kubernetes without asking the machine administrator for root privileges, and without the risk of accidentally breaking other users' environments.
• Laptops: prevent a local cluster from accidentally breaking the host system configuration, e.g., the host iptables rules used for VPNs.
• AI sandbox: a Kubernetes application developer may create a dedicated local user account for running an AI coding agent and a test Kubernetes cluster. This setup is useful for preventing the AI agent from breaking the host when it is deceived by malicious information on the Internet.
• Kubernetes-in-Kubernetes: a nested cluster can run inside a parent cluster as a user-namespaced pod (hostUsers: false), isolating workloads more strictly than Kubernetes API namespaces do.
• Bootstrapping: a temporary unprivileged cluster can be used to bootstrap an actual cluster, e.g., with Cluster API.

How does it work?

A Linux kernel user namespace maps a host level non-root user (e.g., UID 1000) to a fake root user inside the namespace. The UID 0 privileges are limited to the inside of the namespace. The fake root is enough for most of the node components' tasks: mounting volumes, creating cgroups, and configuring the network namespaces of pods. It still comes with some caveats that may break compatibility with specific CNI and CSI drivers, though.

The user namespace has to be created outside of Kubernetes. For example, Rootless Docker can be used to prepare the user namespace in which Kubernetes runs.

The KubeletInUserNamespace feature gate itself is quite "boring": basically it just lets the kubelet ignore permission errors that occur when setting some sysctl values (e.g., vm.overcommit_memory and kernel.panic) and when watching kernel messages via /dev/kmsg.

See Running Kubernetes Node Components as a Non-root User for further information.

What changed from Alpha to Beta?

• The KubeletInUserNamespace feature gate is now enabled by default. Enabling the gate does not put the kubelet into a user namespace automatically, so nothing changes for existing "rootful" clusters.
• kubectl get nodes -o yaml now reports whether nodes are running in a user namespace via the runningInUserNamespace property. A cluster administrator can use this property to set node labels or taints, to avoid scheduling workloads that need real root privileges (e.g., some CNI plugin installers) onto rootless nodes.
• For Kubernetes' own CI/CD testing, the node conformance end to end tests now run on a rootless cluster (ci-kubernetes-e2e-kind-rootless).

Several related improvements have also happened outside the promotion of the feature gate itself:

• Linux kernel v6.3 (2023): added support for idmapped tmpfs.
• Kubernetes v1.33 (2025): enabled the UserNamespacesSupport feature gate by default, allowing user-namespaced pods (hostUsers: false) to be created without extra configuration.
• containerd v2.1 (2025): added support for writable cgroups.

With these improvements, a Kubernetes cluster with KubeletInUserNamespace can now also be nested inside Kubernetes pods with hostUsers: false (UserNamespacesSupport).

How to use it

kind

The easiest way is to use kind (a Kubernetes SIG Testing project) to run a Kubernetes cluster in rootless Docker, rootless nerdctl, or rootless Podman:

# Example using Docker
dockerd-rootless-setuptool.sh install
kind create cluster

Depending on the host configuration, you may need additional configuration for systemd, kernel modules, sysctl, etc.

See the Docker documentation and the kind documentation for further information.

minikube

minikube (a Kubernetes SIG Cluster Lifecycle project) also supports running a Kubernetes cluster in rootless Docker or rootless Podman:

dockerd-rootless-setuptool.sh install
minikube start --driver=docker

See the minikube documentation for further information.

Usernetes

Usernetes (a third-party project) is a distribution of rootless Kubernetes, maintained by the author of this article. The project began in 2018, and it is where the KubeletInUserNamespace feature gate originally came from.

Unlike kind and minikube, Usernetes supports creating a cluster with multiple rootless Docker / Podman / nerdctl nodes, connected using VXLAN via the Flannel CNI plugin.

Usernetes also experimentally supports a Kubernetes-in-Kubernetes mode.

k3s

k3s (a CNCF Sandbox project) also supports rootless mode. Unlike kind, minikube, and the current generation of Usernetes, rootless k3s does not rely on an external runtime such as rootless Docker.

What's next?

Depending on feedback and adoption, the Kubernetes project plans to graduate this feature to General Availability (GA) in a future release. If you have feedback on this feature, please open an issue in the kubernetes/kubernetes repository.

The project is also discussing several Kubernetes Enhancement Proposals that may contribute to simplifying Kubernetes-in-Kubernetes with this feature:

• KEP-5474: Enable Writable cgroups for unprivileged containers
• KEP-5714: Allow specifying whether to unshare cgroup namespaces

Getting involved

We always welcome new contributors. If you would like to get involved, you can join the Node Special Interest Group (SIG Node).

If you would like to share feedback, you can do so on our public Slack channel (visit https://slack.k8s.io/ for an invitation if you need one).

Special thanks to everyone who helped design and implement this feature, including but not limited to (in alphabetical order):

• Bing Hongtao (HirazawaUi)
• Jordan Liggitt (liggitt)
• Sergey Kanzhelev (SergeyKanzhelev)
• Tim Hockin (thockin)

04 Sep 2026 6:30pm GMT

Kubernetes 1.37 is here and Dynamic Resource Allocation (DRA) keeps pushing past where it started! This release brings DRA Extended Resource support to GA, a milestone the team has been building toward for three straight releases. Several more features graduate to Beta or GA. A fresh batch of alpha features rounds out the release.

I'll dive into what's new for DRA in Kubernetes 1.37!

What's stable in 1.37

DRA Extended Resource support has graduated to GA. This is the mechanism that lets DRA drivers satisfy requests made through the traditional extended resource API, think example.com/gpu in a Pod spec, without requiring a separate device plugin alongside the DRA driver. An extended resource name can be set directly on a DeviceClass, and Pods requesting it get matched to a device through DRA with no ResourceClaim needed on the workload's part.

It's been on a steady path since KEP acceptance in 1.34. Alpha landed in 1.35, Beta in 1.36, and now it's Stable. For cluster operators, this is what makes DRA adoption gradual. Existing workloads written against extended resources keep working unmodified while the backend allocation logic moves over to DRA.

ResourceClaims status with possible standardized network interface data adds a devices field to ResourceClaim .status, letting DRA drivers report per-device status, including, for network devices, the interface name, MAC address, and IP addresses. This gives users and controllers visibility into device state that was previously invisible once a device was configured in a Pod, and makes it possible to build things like network services that rely on a device's reported IPs.

DRA: device taints and tolerations is now Stable; DRA drivers can mark devices as tainted so they're skipped for new Pod scheduling, and cluster admins can apply the same taints cluster-wide via a DeviceTaintRule, without reconfiguring drivers. Pods already using a tainted device can be evicted automatically, unless their ResourceClaim explicitly tolerates the taint. This mirrors node taints and tolerations, letting operators take a single device offline for maintenance or mark it degraded, without disrupting the rest of the cluster.

Standard numaNode device attribute standardizes resource.kubernetes.io/numaNode as a shared attribute name, so devices from different drivers can be compared on the same NUMA node instead of each driver inventing its own name for it. It landed directly as stable in 1.37, since it's a naming/registration KEP with no feature gate or in-tree behavior change.

Feature promoted to Beta

ResourceClaim support for workloads graduates to Beta behind the DRAWorkloadResourceClaims feature gate, which stays disabled by default. In a cluster that has the feature enabled, Workloads and PodGroups can reference ResourceClaims directly, so a single claim can be shared across an entire group of Pods. This is instead of claims being capped at 256 Pods through the old per-Pod reservation limit.

The DRA Device Attributes Downward API is aimed at supporting device injection into KubeVirt VMs. Drivers populate a Metadata field when preparing a claim, and the framework writes it to a JSON file mounted into the container via CDI, letting workloads read a device's PCI bus address, MAC address, and other attributes directly instead of requiring custom controllers to watch and translate ResourceClaims and ResourceSlices.

Alpha features

List types for attributes moved into a second Alpha in 1.37, letting a device attribute hold more than one value instead of a single scalar, such as a CPU that's adjacent to more than one PCIe root. This makes it possible to match or distinguish devices based on overlapping or non-overlapping sets of values, while single-value attributes keep working as they do today.

Node allocatable resource requests moved into Alpha 2. It lets the scheduler and kubelet treat DRA-managed CPU, memory, and similar node resources the same way they treat ordinary resource requests, so a node doesn't get oversubscribed and users no longer have to duplicate the same request in both a ResourceClaim and the pod spec.

Resource availability visibility moved to a second Alpha in Kubernetes 1.37. Users create a ResourcePoolStatusRequest to get a point-in-time availability snapshot. To refresh it, delete and recreate the request; it is not a continuous monitoring API.

DRA: Optional Node Operations lets a driver skip kubelet's prepare and unprepare calls for allocations that don't need any setup on the node. This makes it possible to avoid an unnecessary dependency on the driver for allocations where there's genuinely nothing for it to do locally.

Derived Attributes is a new feature that lets you use CEL expressions to match up devices based on your own custom rules. Before this, pairing devices from different vendors (like a GPU/TPU and a NIC on the same NUMA node) only worked if both drivers used the exact same attribute name. If one used numa and the other used numaNode, the scheduler couldn't pair them together. Now, you can easily bridge these differences yourself inside your manifest, meaning you don't have to wait for hardware vendors to agree on standardized attribute names. Beyond just fixing naming differences, you can also use CEL to handle more complex scenarios like slicing a specific ID out of a long, monolithic topology string, or grouping devices into custom performance tiers based on their available capacity.

DRA Device Compatibility Groups lets drivers tag partitions of a device, like MIG vs vGPU profiles on the same GPU, with compatibility groups, so the scheduler rejects incompatible combinations up front instead of the driver failing at node preparation time. It's controlled by the DRADeviceCompatibilityGroups feature gate, disabled by default.

PreQueueingHint extension point is new as Alpha in 1.37. DRA ResourceClaim events used to trigger a full scan of every unschedulable pod, an O(N²) cost during large scale-ups. The DRA plugin now uses a pod informer index to narrow that to just the pods actually affected, cutting the requeue path to O(1) and roughly doubling scheduling throughput in early benchmarks. Controlled by the SchedulerPreQueueingHints feature gate.

DRA Consumable Capacity now supports fractional values in CapacityRequestPolicyRange, enabling more precise capacity requests and allocation for devices with fractional resources. This improves flexibility for workloads that require fine-grained resource allocation. The enhancement is gated by the DRAFractionalCapacityRange feature gate, which is in Beta in 1.37.

What's next

DRA continues to mature with every release. Several features currently in Alpha and Beta are on track to progress in the coming releases, and the community keeps working on DRA's performance, scalability, and reliability. Expect another ambitious set of DRA features in Kubernetes 1.38.

Getting involved

A good starting point is joining the WG Device Management Slack channel and meetings which happens at US/EU and EU/APAC friendly time slots.

Not all enhancement ideas are tracked as issues yet, so come talk to us if you want to help or have some ideas yourself! We have work to do at all levels, from difficult core changes to usability enhancements in kubectl which could be picked up by newcomers.

Acknowledgments

The following KEP owners added or promoted a feature in the 1.37 release (in alphabetic order):

• Alay Patel (alaypatel07)
• Byonggon Chun(bg-chun)
• Gaurav Ghildiyal (gauravkghildiyal)
• Jiefeng Xu (jiefeng-xu)
• John A. Hull (johnahull)
• Jon Huhn (nojnhuh)
• Lionel Jouin (LionelJouin)
• Patrick Ohly (pohly)
• Praveen Krishna (pravk03)
• Shingo Omura (everpeace)
• Troy Chiu (troychiu)

This would not have been possible without the help of the reviewers and approvers. So a huge thanks to everyone else who helped shape this release, in ways big and small. Given enough eyeballs, all bugs are shallow and this release had plenty of them, watching closely and caring enough to make things better. DRA got better this cycle because of all of you.

03 Sep 2026 6:30pm GMT

Kubernetes v1.37 includes API support for horizontal autoscaling of workloads down to zero replicas. This feature is now Beta and enabled by default. A HorizontalPodAutoscaler (HPA) that uses a suitable object metric or external metric can now scale a workload to zero replicas, then bring it back when the metric changes.

Before v1.37, you needed an add-on or external component, or you had to enable the Alpha feature gate, to scale from zero. It is now part of core Kubernetes.

Scaling to zero removes the last idle Pod from workloads such as queue consumers and batch processors. The savings are largest when each Pod reserves expensive resources, including dedicated CPUs or GPUs.

The trade-off is cold-start time: the HPA must observe the metric, schedule a Pod, and start the application. This works well when work can wait in a durable queue.

Kubernetes Services do not buffer requests while no Pods are ready, so HTTP and other request-driven workloads need a separate buffering layer.

Why scaling from zero needs a different metric

The HPA commonly scales on CPU or memory usage. Both metrics come from running Pods. Once the replica count reaches zero, there are no Pods left to measure and no signal that can tell the HPA to scale back up.

Object and external metrics do not have that limitation. A queue length, for example, exists independently of the workers that consume it. The HPA can continue reading the queue length while no workers are running.

The following example scales a queue consumer to and from zero using an external metric.

Configure an external metric

The following example uses a Prometheus metric named queue_consumer_lag. It assumes that Prometheus already collects a series similar to this one:

queue_consumer_lag{namespace="default",name="worker_tasks"}

Kubernetes needs a metrics adapter to make that value available through the External Metrics API. One implementation is the Prometheus Adapter, which can expose the series using an externalRules entry:

externalRules:
- seriesQuery: '{__name__="queue_consumer_lag",name!=""}'
 metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (name)
 resources:
 overrides:
 namespace:
 resource: namespace

The exact adapter installation and discovery rules depend on your monitoring setup. See the Prometheus Adapter guide to external metrics for the full configuration options.

Before creating the HPA, you can verify that Kubernetes can read the metric:

kubectl get --raw \
 '/apis/external.metrics.k8s.io/v1beta1/namespaces/default/queue_consumer_lag?labelSelector=name%3Dworker_tasks'

The request should return the current value for worker_tasks. If it does not, fix the metrics pipeline before configuring the HPA. An HPA cannot scale from zero when its metric is unavailable.

Configure the HPA

The following HPA targets a Deployment named queue-worker. It allows between zero and ten replicas, with one replica requested for each 30 queued tasks:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
 name: queue-worker
 annotations:
 kubernetes.io/description: "Scales queue-worker based on the number of queued tasks"
spec:
 scaleTargetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: queue-worker
 minReplicas: 0
 maxReplicas: 10
 metrics:
 - type: External
 external:
 metric:
 name: queue_consumer_lag
 selector:
 matchLabels:
 name: worker_tasks
 target:
 type: Value
 value: "30"

When the queue is empty, the HPA can reduce the Deployment to zero replicas. When tasks arrive, the external metric remains available and the HPA calculates a new replica count, capped at ten by maxReplicas.

Start the Deployment with at least one replica. Manually setting a Deployment to zero has always paused autoscaling. The HPA preserves that behavior and will not wake a workload that it did not scale down itself.

Normal HPA behavior still applies. In particular, the default downscale stabilization window is five minutes. The window prevents a short drop in queue length from immediately removing all workers. You can configure the window through spec.behavior.scaleDown if your workload needs different behavior.

How the HPA distinguishes zero from paused

Scaling from zero creates an ambiguity. A replica count of zero can mean that the HPA scaled the workload down, or that an operator manually paused it.

The controller resolves this with a ScaledToZero status condition. When the HPA scales a workload from one or more replicas to zero, it records ScaledToZero=True. The condition tells later reconciliation loops that the controller owns the zero state and should continue evaluating object or external metrics.

After scaling the workload back up, the controller changes the condition to ScaledToZero=False with the reason NotScaledToZero. A workload at zero without the ScaledToZero=True condition remains paused.

You can inspect the conditions with:

kubectl describe hpa queue-worker

If the adapter cannot return the configured metric, the HPA reports ScalingActive=False with a reason such as FailedGetExternalMetric. Restore the metric or manually scale the workload to recover capacity.

Before upgrading or rolling back

In Kubernetes v1.37, the HPAScaleToZero feature gate is enabled by default on both the kube-apiserver and kube-controller-manager. The API server accepts minReplicas: 0; the controller manager performs the condition-based scaling.

During a version-skewed control plane upgrade, wait until both components support the feature and have it enabled before creating HPAs with minReplicas: 0. A controller manager with the feature disabled treats replicas: 0 as a manual pause and may leave a workload at zero.

Before disabling the feature gate or downgrading to a version without the condition-based implementation:

• Change affected HPAs to minReplicas: 1 or higher.
• Scale any workload currently at zero to at least one replica.

minReplicas: 0 also requires at least one object or external metric. The API server rejects an HPA that only contains resource metrics such as CPU or memory.

From Alpha to Beta

The first Alpha implementation shipped in Kubernetes v1.16. Kubernetes v1.36 added the ScaledToZero condition and the controller behavior needed to distinguish an automatic scale-down from a manual pause.

Kubernetes v1.37 enables the feature by default after adding integration and end-to-end coverage for scaling down to zero and back up from an external metric. The next step is to gather operational feedback before considering graduation to GA.

How can I learn more?

• Read the documentation for scaling to and from zero.
• Read KEP-2021: HPA supports scaling to and from zero pods for object and external metrics.
• Learn how to configure the Prometheus Adapter for external metrics.

How to get involved

This feature is owned by SIG Autoscaling. Join Kubernetes Slack and the #sig-autoscaling channel to share feedback from Beta usage.

Acknowledgements

Thanks to the SIG Autoscaling contributors who took this feature from the original v1.16 implementation to the condition-based redesign and Beta graduation. Thanks also to Guy Templeton and Adrian Moisey for reviewing the KEP, and to the release, documentation, and production-readiness reviewers who helped prepare it for Kubernetes v1.37.

02 Sep 2026 6:30pm GMT