fidzu : Kubernetes

As Kubernetes adoption has grown, the conversation has shifted beyond running containers to managing increasingly complex application lifecycles. Modern platforms support stateless web services, stateful databases, batch processing, AI workloads, and platform services. At the same time, they must remain reliable during upgrades, scaling events, and infrastructure failures.

Every Kubernetes user relies on SIG Apps, whether they realize it or not. Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs form the foundation of how applications are deployed, updated, scaled, and operated across the Kubernetes ecosystem.

SIG Apps is focused on improving workload resilience, refining application lifecycle management, and addressing the operational challenges that emerge when applications encounter node failures, rollout disruptions, and increasingly complex infrastructure environments.

In this spotlight, we sit down with two of the three SIG Apps chairs Janet Kuo and Maciej Szulik to discuss the evolution of Kubernetes workload management, the challenges of balancing application reliability with operational simplicity, and the future of application lifecycle management within one of Kubernetes' most influential Special Interest Groups.

Introducing SIG Apps

Natalie Fisher: Can you introduce yourself, your role, and how you got involved in SIG Apps?

Janet Kuo: I'm a Senior Staff Software Engineer at Google and have been a Kubernetes maintainer since 2015, joining the community just as we were racing toward the 1.0 launch. In those early days, my focus was on building the core Workloads API, specifically developing controllers like Deployment, ReplicaSet, StatefulSet, and DaemonSet, defining their rollout behaviors, and bringing them from initial designs to GA. That hands-on work was my entry point into SIG Apps.

Since then, I've stayed deeply involved in both the technical and community sides of Kubernetes. I have led SIG Apps as Co-Chair and Tech Lead since 2019. Currently, in addition to maintaining the workloads API, I am driving new subprojects like the Agent Sandbox to ensure Kubernetes is ready for next-generation agentic and AI workloads.

Maciej Szulik: I started contributing to Kubernetes all the way back in 2014. Since then, I've worked across various areas of the project: controllers, kubectl, and apimachinery, which eventually led me to become one of the Chairs and Tech Leads for SIG Apps. My current focus is reliability of the workload controllers under the SIG Apps umbrella and stability and ease of use of kubectl as part of my SIG CLI Tech Lead role. I also care about overall community health and growth as part of my Steering Committee role. Outside of Kubernetes, I work as a Staff Platform Engineer at Defense Unicorns, where I'm helping make Kubernetes more airgap-native with a project called zarf.

The problem and the solution

SIG Apps is responsible for the core workload APIs that power how applications run on Kubernetes. From Deployments and StatefulSets to Jobs and CronJobs, these controllers determine how workloads are created, updated, scaled, and recovered when things go wrong.

As Kubernetes expands to support increasingly diverse workloads - including AI, batch processing, and large-scale distributed applications - SIG Apps continues to evolve these APIs while balancing reliability, backward compatibility, and operational simplicity.

NF: For readers who may not be familiar, what is SIG Apps, and what role does it play within the broader Kubernetes ecosystem?

MS: SIG Apps is the Kubernetes Special Interest Group responsible for the workloads APIs. CronJob and Job help running batch workloads, whereas DaemonSet, Deployment, ReplicaSet, and StatefulSet serve the majority of other applications. More broadly, SIG Apps owns the layer most developers actually touch day-to-day: the controllers that turn a workload specification into running, self-healing pods. It's the group deciding how Deployments roll out, how Jobs retry, how DaemonSets place a pod per node.

JK: Adding to what Maciej described, as the industry shifts, we are seeing a massive demand to run complex, non-traditional workloads like distributed AI training, batch computing, and dynamic agent environments. Our role is expanding: we aren't just maintaining the classic workloads API, but we are actively evolving it and establishing new patterns (like the Agent Sandbox) to make sure Kubernetes remains the best platform for the next generation of workloads, such as AI.

NF: Looking at the workload APIs owned by SIG Apps (Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs), which areas are receiving the most attention from maintainers and contributors?

MS: After a long stretch focused on making batch workloads run smoothly on Kubernetes, we've shifted attention to make sure serving workloads (DaemonSets, StatefulSets, etc) aren't left behind. This means performance and high-scale improvements to rollout and scaling behavior, plus working through our backlog of user-reported issues, prioritizing the ones with the strongest support from the user base.

Current focus areas

As Kubernetes workloads grow in scale and complexity, the challenges facing workload controllers evolve as well. We asked the SIG Apps chairs where contributors are focusing their efforts today and which resilience problems they believe are the highest priorities.

NF: From your perspective, what are the most important workload resilience problems SIG Apps is trying to solve today?

MS: Node lifecycle challenges have come up repeatedly across SIG Apps, SIG Node, and SIG Autoscaling discussions. DaemonSets and Jobs are just where the pain is most visible, since they're the workloads most directly bound to node state. Rather than solve it piecemeal within one SIG, we've settled on spinning up a dedicated Node Lifecycle Working Group to focus on this properly and hopefully land long-term solutions instead of one-off patches.

JK: From an AI perspective, resilience is critical. When you are running a massive distributed LLM training job that spans hundreds of GPUs, a single node failure can halt the entire pipeline. Similarly, if a DaemonSet that runs your logging or GPU monitoring agent gets stuck on a bad node, it impacts the entire cluster's health.

In addition to the work in the Node Lifecycle WG to handle infrastructure-level degradation, SIG Apps is addressing this at the orchestration layer through subprojects like JobSet (for distributed training) and LeaderWorkerSet (LWS) (for sharded LLM inference). These APIs introduce patterns like "all-or-nothing" failure handling, where a single pod or job failure triggers a coordinated group-level restart to resume from the last clean checkpoint, rather than letting stuck workloads hang in an inconsistent state.

Real-world impact

The work happening within SIG Apps extends far beyond controller implementations and API design. We wanted to understand what these improvements mean in practice for platform teams operating Kubernetes clusters in production.

NF: For platform teams operating Kubernetes in production, what practical improvements would they notice if the node lifecycle and workload resilience work currently under discussion is successfully delivered?

MS: I'm mostly looking from the sidelines, the folks actually in the Node Lifecycle Working Group would give you a sharper answer. But from where I sit, I'm hoping their work translates into fewer 3am pages that turn out to be "a DaemonSet rollout got stuck because node X was flaky, and someone had to manually cordon/delete/restart to unstick it."

JK: +1 to what Maciej said, and beyond reducing manual intervention, platform teams will also see much better resource predictability and cost efficiency. For example, in AI workloads where GPU idle time is extremely expensive, having Kubernetes automatically detect a degraded node and reschedule the training coordinator or agent before the job crashes means less wasted compute and more stable job execution.

Challenges and trade-offs

Evolving APIs that millions of workloads rely on requires careful engineering and even more careful decision-making. We asked the SIG Apps chairs about the technical and operational trade-offs they weigh when introducing changes to Kubernetes' core workload controllers.

NF: What are some of the hardest technical or operational trade-offs SIG Apps encounters when evolving core workload controllers?

MS: Honestly, a few tensions keep coming up: how aggressively a controller should give up on stuck pods, and what signals it actually needs to make that call correctly. At the same time, we always have to think about backward compatibility. Deployment, DaemonSet, and Job behavior has been depended on for a decade [by Kubernetes users, tooling, automation, and higher-level controllers], so even a change that's clearly "more correct" can break automation people built around the old behavior without meaning to.

JK: One of our hardest trade-offs is resisting the urge to make "elegant" design changes that break backward compatibility. Instead, we have to design opt-in features that let users adopt new behaviors without forcing them on legacy workloads. When we need to support completely new paradigms, we prefer introducing them as CRDs first rather than bloating the core APIs, like we are doing with Agent Sandbox, JobSet, and LWS.

Looking ahead

While much of SIG Apps' work focuses on maintaining the stability of existing workload APIs, the group is also shaping the future of Kubernetes through new enhancements and proposals. We concluded by asking about one proposal that recently returned to active development and what it represents for the future of workload management.

NF: The SIG recently discussed reviving KEP-4443 with a target release of Kubernetes 1.38. What opportunities or challenges does this proposal aim to address, and why is now the right time to revisit it?

KEP-4443 addresses a small but real gap in the Job API: a PodFailurePolicy can be configured to add a condition reason to the JobFailed condition, but different pod failure policy rules targeting different container exit codes all produce that same generic reason. The proposal is simple: an optional Name field on each PodFailurePolicyRule, which gets appended to the JobFailed condition reason, so higher-level tools like JobSet can finally react differently depending on which rule triggered the failure.

As for timing, the answer is as simple as it always is in open source: we lost the original contributor who was driving this. Now we've got someone new interested in picking it up, that's why we're targeting the next release.

Getting Involved

NF: For someone interested in contributing to SIG Apps, where would you recommend they start, especially if they are not yet a Kubernetes maintainer?

MS: The best place to start is the #sig-apps slack channel and our regular SIG Apps meetings. We've all started there, and if it feels intimidating, or nobody replies right away, that's completely normal. Everyone's busy. It's not personal.

JK: In addition to what Maciej answered, I'd suggest looking at our newer subprojects and initiatives. Contributing to stable APIs like Deployment or StatefulSet can be daunting because the barrier for making changes is very high due to backward compatibility, and there is much less low-hanging fruit.

If you are new to the community, projects like the Agent Sandbox are fantastic entry points. They are actively evolving, have a friendly group of maintainers, and offer plenty of greenfield development opportunities where you can make a significant impact quickly.

Summary

SIG Apps has shaped how Kubernetes applications are deployed and operated since the project's earliest days. While users often interact with Deployments, StatefulSets, Jobs, and DaemonSets without thinking about the controllers behind them, the work within SIG Apps continues to shape the reliability and scalability of workloads across the Kubernetes ecosystem.

From improving workload resilience and node lifecycle behavior to enabling new patterns for AI and distributed computing, the SIG is evolving Kubernetes while remaining committed to one of the project's core principles: preserving the stability and backward compatibility that users depend on. Whether you're interested in core workload APIs, emerging projects like Agent Sandbox, or helping improve the operational experience of Kubernetes users everywhere, SIG Apps offers many opportunities to get involved.

22 Sep 2026 6:00pm GMT

Kubernetes v1.37 promotes the PersistentVolumeClaimUnusedSinceTime feature gate to Beta (enabled by default). With this feature, the PersistentVolumeClaim (PVC) protection controller adds an Unused condition to each PVC, telling you whether any running pod currently references it - no custom tooling or cross-referencing required.

For the API definition of PVC conditions, see the PersistentVolumeClaim API reference. Read on to learn how the Unused condition works and how to use it.

Why track PVC usage?

In large-scale Kubernetes clusters, it is common for users to create PVCs and then delete the associated pods without cleaning up the storage, because Kubernetes does not automatically delete PVCs when their pods are removed (to protect against accidental data loss). Over time, these orphaned PVCs may accumulate, silently consuming storage capacity and driving up cloud costs.

Before Kubernetes v1.37, it was easy to identify an unused PersistentVolume, but much harder to determine whether a PVC was still being used. Doing so required cross-referencing pods, PersistentVolumes, and PVCs over a potentially large window of time. Administrators often resorted to custom monitoring pipelines or scripts to answer a seemingly simple question: "Is anything actually using this volume?"

The PersistentVolumeClaimUnusedSinceTime feature solves this by making the answer available natively in the PVC status. Once the feature is enabled, every PVC gets an Unused condition managed by the PVC protection controller.

User stories

• Storage administrator: "I want to know which PVCs in my cluster are not being used by any pod so I can safely identify orphaned volumes and schedule them for deletion."
• DevOps engineer: "I want to list PVCs that have the Unused condition set to True so I can automate cleanup in development environments."

How does it work?

The PVC protection controller - which already watches pods to enforce the storage object in use protection - now also manages a new Unused condition on PVCs.

The condition works as follows:

A few details worth noting:

• Terminated pods don't count: A pod that has completed (phase Succeeded or Failed) does not keep the PVC marked as in use. This means batch jobs with restartPolicy: Never won't prevent the PVC from becoming Unused=True after they finish.
• Pending pods do count: Even an unschedulable pod (for example, one with an impossible node selector) still counts as using the PVC. The intent to use the volume is enough.
• Multiple pods: If several pods reference the same PVC, the condition transitions to Unused=True only after the *last" non-terminated pod is removed or terminates.

Using lastTransitionTime to find when a PVC became idle

Like every Kubernetes condition, the Unused condition carries a standard lastTransitionTime field. This means you get a useful bonus for free: when the condition transitions from False to True, the lastTransitionTime records exactly when the PVC became idle. You can use this timestamp to answer questions like "how long has this PVC been sitting unused?" - for example, to find PVCs that have been idle for more than 30 days (see the example query below).

What changed from Alpha to Beta?

Kubernetes v1.36 introduced this feature as Alpha, where you had to enable the PersistentVolumeClaimUnusedSinceTime feature gate explicitly. For Beta in v1.37, the feature gate is enabled by default, and the feature has full end-to-end test coverage.

How to use it

Since the feature is Beta and enabled by default in Kubernetes v1.37, the Unused condition will appear on PVCs automatically. Here is a walkthrough to see it in action:

1. Create a PVC:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
    name: my-data
   spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
    storage: 1Gi
   

2. After a short time, inspect the PVC conditions:

   kubectl get pvc my-data -o jsonpath='{.status.conditions[*]}' | jq .
   

   You should see an Unused condition with status True and reason NoPodsUsingPVC:

   {
    "lastProbeTime": null,
    "lastTransitionTime": "2026-09-14T12:03:11Z",
    "message": "No pods are currently referencing this PVC",
    "reason": "NoPodsUsingPVC",
    "status": "True",
    "type": "Unused"
   }
   

3. Create a pod that uses the PVC:

   apiVersion: v1
   kind: Pod
   metadata:
    name: my-app
   spec:
    containers:
    - name: app
    image: busybox
    command: ["sleep", "3600"]
    volumeMounts:
    - name: data
    mountPath: /data
    volumes:
    - name: data
    persistentVolumeClaim:
    claimName: my-data
   

4. Check the condition again - it should now show Unused=False:

   kubectl get pvc my-data -o jsonpath='{.status.conditions[?(@.type=="Unused")].status}'
   

   Output:

   False
   

5. Delete the pod and wait for the condition to transition back to Unused=True:

   kubectl delete pod my-app
   kubectl get pvc my-data -o jsonpath='{.status.conditions[?(@.type=="Unused")]}'
   

   The condition should show Unused=True with reason NoPodsUsingPVC again.

Finding unused PVCs across the cluster

To list all PVCs that have been unused for more than 30 days, you can use a command like:

kubectl get pvc -A -o json | jq -r '
 .items[]
 | select(.status.conditions[]? | select(.type=="Unused" and .status=="True"))
 | select(
 (.status.conditions[] | select(.type=="Unused") | .lastTransitionTime) as $t
 | (now - ($t | fromdateiso8601)) > (30 * 86400)
 )
 | "\(.metadata.namespace)/\(.metadata.name) unused since \(.status.conditions[] | select(.type=="Unused") | .lastTransitionTime)"
'

What's next?

Depending on feedback and adoption, the Kubernetes project intends to graduate this feature to General Availability (GA) in a future release. If you have feedback on this feature, please open an issue in the kubernetes/kubernetes repository.

To learn more about this enhancement, refer to KEP-5541: PersistentVolumeClaim last used time.

Getting involved

The Kubernetes project always welcomes new contributors. If you would like to get involved, you can join us at SIG Storage.

If you would like to share feedback, you can do so on our public Slack channel (visit https://slack.k8s.io/ for an invitation if you need one).

Special thanks to the contributors who helped design and implement this feature (alphabetical order):

• Arvind Parekh (ArvindParekh)
• Hemant Kumar (gnufied)
• Jan Šafránek (jsafrane)
• Kevin Hannon (kannon92)
• Roman Bednář (RomanBednar)

21 Sep 2026 6:30pm GMT

Kubernetes v1.37 brings important storage security features: emptyDir permission modes and bind mount options. They help application programmers and security professionals implement rigorous security policies, for example, prohibiting deletion of files across containers or execution of arbitrary binaries from writable volumes, directly in Kubernetes without any complicated circumvention.

Linux storage and permission fundamentals

Before diving into the new Kubernetes features, let us briefly review the low-level Linux security mechanisms that make them possible.

Bind mount flags

When Linux mounts or remounts a directory, Virtual File System (VFS) flags control what actions are permitted on that filesystem:

• noexec: Do not permit direct execution of any binaries on the mounted filesystem.
• nosuid: Do not allow set-user-identifier or set-group-identifier bits to take effect.
• nodev: Do not interpret character or block special devices on the file system.

Directory permissions and the sticky bit

Standard Unix permissions regulate access across three scopes: Owner, Group, and Others (e.g., 0755 or 0777).

Beyond standard read, write, and execute bits, Linux supports the sticky bit (as in mode 01777). When applied to a directory, the sticky bit ensures that a file inside that directory can only be deleted or renamed by the file's owner or root. This is essential for shared writable directories like /tmp.

Motivation for the improvements

Why does Kubernetes need bind mount options and emptyDir permissions?

The primary goal of these features is to increase the security of Kubernetes workloads by allowing security-related bind mount options on volume mounts. By default, volumes are bind-mounted into containers by the container runtime and kubelet without noexec, nosuid, or nodev flags. This default can undermine security. For example, with noexec missing, a compromised process can use any writable volume (emptyDir, PersistentVolume, etc.) to download, chmod +x, and execute arbitrary binaries even when the container has a read-only root filesystem (readOnlyRootFilesystem: true). Supporting noexec, nodev, and nosuid gives users a native way to harden volume mounts to match security benchmarks and policy.

The gap is most visible with emptyDir volumes, which are the most common writable volume type and have been the subject of multiple security findings:

• Issue #48912: Recognized security gap - the inability to set mount options on emptyDir was flagged in an audit but remained unresolved until now.
• Issue #119627: Kubernetes 1.24 Security Audit (Finding NCC-E003660-7HM) - external auditors specifically noted that the inability to mount emptyDir with noexec represents a security failure.

However, the same gap applies to all volume types. PersistentVolumes have a mountOptions field, but those options are filesystem-level flags applied by the CSI driver at the node, so they do not reliably translate into bind mount flags inside the container. Previously, there was no mechanism to set noexec, nosuid, or nodev on the bind mount that the container runtime creates for any volume type.

Additionally, the emptyDir volume type defaults to creating directories with a hardcoded mode of 0777. This previously meant that any process that can discover the volume could read, write, and delete anything in the volume, regardless of who created it.

You could - and still can - use an initial container to set a different access mode, but this is more complex, and hard to verify for compliance.

This causes real problems:

• Multi-container pods sharing an emptyDir could not prevent one container from deleting another's files. The sticky bit (01777) solves this, but there was no native way to set it.
• Some applications and security frameworks expect /tmp directories to have the sticky bit set (mode 01777). Without native support for setting the emptyDir mode, users had to use init containers or alternative volume types to meet this requirement.
• Platform engineers who want tighter permissions (e.g., 0750 for owner and group only) have to use init containers running chmod, which adds unnecessary complexity.

The emptyDir volume type was a notable gap. As one of the most common writable volume types in Kubernetes, it had no way to control its creation permissions.

Real-world use cases

Application developers, working closely with security engineers, are responsible for maintaining the security posture of their applications and ensuring workloads do not pose risks to the wider infrastructure. These features allow development teams to confidently address critical security scenarios:

Preventing Privilege Escalation on Writable Mounts: An application developer configuring temporary workspace volumes (like emptyDir or /tmp mounts) can ensure they are mounted with nosuid and noexec. This guarantees that even if the application is compromised and a malicious payload is downloaded, the workload cannot execute the payload or use it to escalate privileges on the node.

Securing Shared Scratch Space in Multi-Container Pods: A developer configuring CI/CD pipeline pods often needs multiple containers (e.g., a builder container and a sidecar logger) to share a workspace. By setting mode: 01777 on an emptyDir, the developer ensures the shared workspace behaves like a traditional Unix /tmp directory. Each container can write files independently, but a compromised process in one container cannot delete the build artifacts produced by another.

Enforcing Principle of Least Privilege for Application Data: An application developer deploying a database pod can lock down access to the database's temporary storage. By setting mode: 0750 on the emptyDir, the developer ensures that only the specific database user and group can read or write to the volume, explicitly denying access to any other processes or sidecars in the same pod.

Note: Both features are behind Alpha feature gates in Kubernetes v1.37. To use them, enable VolumeBindMountOptions and EmptyDirVolumeMode on the API server and kubelet.

Example 1: Enforcing bind mount options

This full Pod manifest mounts an emptyDir volume at /tmp with bindMountOptions: [noexec, nosuid].

apiVersion: v1
kind: Pod
metadata:
 name: hardened-bindmount-pod
 namespace: default
spec:
 os:
 name: linux
 containers:
 - name: hardened-app
 image: alpine:latest
 command: ["sleep", "3600"]
 securityContext:
 readOnlyRootFilesystem: true
 volumeMounts:
 - name: temp-storage
 mountPath: /tmp
 bindMountOptions:
 - noexec
 - nosuid
 volumes:
 - name: temp-storage
 emptyDir: {}

Example 2: emptyDir volume permission mode with sticky bit

This full Pod manifest creates an emptyDir volume using mode: 01777 to enforce standard Unix /tmp sticky bit protections across containers.

apiVersion: v1
kind: Pod
metadata:
 name: hardened-emptydir-pod
 namespace: default
spec:
 os:
 name: linux
 containers:
 - name: app-container
 image: alpine:latest
 command: ["sleep", "3600"]
 volumeMounts:
 - name: shared-tmp
 mountPath: /tmp
 volumes:
 - name: shared-tmp
 emptyDir:
 mode: 01777

Verifying the features in Linux

To verify that these features are actively enforcing restrictions, you can run kubectl exec into the container. The following examples simulate attempts to perform actions that are successfully blocked by these features.

Verifying noexec

Attempt to write and run a script on a volume mounted with noexec:

# 1. Exec into the pod
kubectl exec -it hardened-bindmount-pod -- sh

# 2. Create an executable script on the mounted volume
cd /tmp
echo '#!/bin/sh' > test.sh
echo 'echo "Executing untrusted code..."' >> test.sh
chmod +x test.sh

# 3. Attempt to run the script
./test.sh

Expected result:

sh: ./test.sh: Permission denied

Even if an executable file is created, the Linux kernel refuses execution because MS_NOEXEC is enforced at the bind mount level.

Verifying the sticky bit

Attempt to delete another user's file in an emptyDir with 01777 permission mode:

# 1. Exec into the pod
kubectl exec -it hardened-emptydir-pod -- sh

# 2. Verify directory permissions on /tmp
ls -ld /tmp
# Output: drwxrwxrwt 2 root root ... /tmp (Notice the 't' indicating sticky bit)

# 3. Create a file as the guest user
su -s /bin/sh -c "touch /tmp/guest_file" guest

# 4. Attempt to delete that file as nobody
su -s /bin/sh -c "rm /tmp/guest_file" nobody

Expected result:

rm: can't remove '/tmp/guest_file': Operation not permitted

The kernel blocks deletion because the sticky bit (01777) restricts file removal strictly to the owner of the file.

Things to know

Keep these key details in mind as you begin using these features. Full details are available in the official documentation for bind mount options, emptyDir volume mode, and emptyDir volumes.

• Default unchanged: If you omit bindMountOptions or do not set an emptyDir mode, you get standard default behaviors (like 0777 permissions) exactly as before.
• Broad volume support: bindMountOptions works with emptyDir, PersistentVolumes, CSI volumes, projected volumes, ConfigMaps, Secrets, and more. The only exception is image volumes, which are explicitly unsupported. The mode field works with all emptyDir medium types: default (disk-backed), Memory (tmpfs), and HugePages.
• Runtime capabilities matter (for bindMountOptions): The container runtime must support the CRI mount_options field and advertise it via runtimeFeatures. The scheduler uses node declared features to avoid placing pods on incompatible nodes. If a pod reaches such a node anyway, the kubelet rejects it. There is no silent degradation. However, using mode for an emptyDir does not require runtime support.
• Not the same as PV mountOptions: PersistentVolume mountOptions apply at the storage layer via the CSI driver. The new bindMountOptions controls bind mount flags applied inside the container by the runtime. They operate at different layers and do not conflict.
• fsGroup interaction: If fsGroup is set in the pod's security context, the group permissions applied by fsGroup will override the mode specified for the emptyDir volume. This is the same behavior that exists for defaultMode on Secret and ConfigMap volumes.
• Linux only: Flags like noexec, nosuid, nodev, and Unix permission modes are Linux concepts. bindMountOptions has no effect on Windows nodes. On Windows, the mode field is also skipped for emptyDir volumes, since Windows does not support Unix-style file permissions.
• Version skew safety: Both features are additive. For emptyDir mode: if the API server has the gate enabled but the kubelet does not, the field is accepted but ignored - the kubelet falls back to 0777. For bindMountOptions: the scheduler uses Node Declared Features to prevent placing pods on nodes without runtime support; if a pod reaches such a node, the kubelet rejects it rather than silently ignoring the options.
• Feature Gates: Both capabilities are available as Alpha features in Kubernetes v1.37:
  • VolumeBindMountOptions: Controls bind mount flags on volume mounts.
  • EmptyDirVolumeMode: Controls creation permission modes on emptyDir volumes.

How do I get involved?

These new features are driven by SIG Node and SIG Storage. You can find more details in the KEPs for these enhancements: KEP-5855 (bind mount options) and KEP-5502 (emptyDir permission mode).

Reach out to SIG Node:

• Slack: #sig-node
• Mailing list

Reach out to SIG Storage:

• Slack: #sig-storage
• Mailing list

16 Sep 2026 6:30pm GMT

With the release of Kubernetes v1.37, the Pod-Level Resource Managers feature has graduated to Beta status (disabled by default)!

First introduced as an Alpha feature in Kubernetes v1.36, this enhancement builds on Pod-Level Resources by equipping Kubelet's Topology Manager, CPU Manager, and Memory Manager to use Pod-level resource declarations (.spec.resources) directly when making hardware placement decisions.

Bringing pod-level resources to node managers

Before this feature, obtaining exclusive NUMA-aligned CPU cores or memory for latency-critical applications forced cluster operators into an all-or-nothing choice: assign integer resource requests to every container in the Pod, or forfeit exclusive NUMA alignment entirely. For modern workloads running lightweight sidecars (such as logging agents or telemetry exporters), allocating dedicated physical cores to auxiliary containers was wasteful.

Pod-Level Resource Managers solves this challenge by enabling hybrid allocation models. The Kubelet can reserve exclusive NUMA-aligned resources for primary application containers while placing non-Guaranteed sidecars into a pod-isolated shared pool. This ensures primary workloads get unthrottled, NUMA-local performance while sidecars benefit from running in a pod-isolated shared pool, enjoying local NUMA alignment and protection from external node interference without consuming dedicated physical cores.

What's new in Beta

Graduating to Beta brings key operational and API enhancements:

• Graduation to Beta: Controlled by the PodLevelResourceManagers feature gate, available to opt in (disabled by default) in Kubernetes v1.37.
• PodResources API Reporting: The v1 PodResources gRPC service (PodResourcesLister) introduces top-level cpu_ids and memory fields on PodResources responses. Monitoring tools and device plugins can query pod-level exclusive assignments directly without double-counting container allocations.

Getting started and providing feedback

For a deep dive into the technical details and configuration of this feature, check out the official documentation:

• Pod-level resource managers
• Pod-level resource managers reference

To follow a step-by-step tutorial on configuring and deploying workloads:

• Use pod-level resources with Kubelet resource managers

To learn more about how to assign resources to pods:

• Assign Pod-level CPU and memory resources

As this feature moves through Beta toward GA, your feedback is invaluable. Please report any issues or share your experiences via the standard Kubernetes communication channels:

• Slack: #sig-node
• Mailing list
• Open Community Issues/PRs

15 Sep 2026 6:30pm GMT

Memory QoS has graduated to Beta in Kubernetes v1.37 and is now enabled by default. On Linux nodes running cgroup v2, the feature uses the memory controller to give the kernel better guidance on how to treat container memory. It was first introduced as Alpha in v1.22, and expanded in v1.36 with tiered memory reservation.

This post covers what changed in v1.37, what the Beta promotion means for cluster operators, and how to configure the feature.

What changed in v1.37

Memory QoS is Beta and enabled by default

The MemoryQoS feature gate is now Beta in v1.37. This means every v1.37 kubelet has the feature gate turned on without any configuration change. Turning on the feature by default is safe because the default kubelet configuration does not enable memory throttling or memory reservation. No memory.high, memory.min, or memory.low values are written to cgroups unless you explicitly configure them.

You can opt into specific behaviors through kubelet configuration fields:

1. Set memoryThrottlingFactor (for example, 0.9) to enable memory.high throttling on Burstable and BestEffort containers. The default is null, which means no throttling.
2. Set memoryReservationPolicy to TieredReservation to enable tiered memory protection via memory.min and memory.low. The default is None, which means no memory reservation.

Default memoryThrottlingFactor changed to null

In earlier Alpha releases, memoryThrottlingFactor defaulted to 0.9, which meant enabling the feature gate caused the kubelet to set memory.high on containers. In v1.37, the default is null, so the kubelet does not set memory.high unless you configure a value.

This change was made because, with the feature gate now on by default, an automatic memory.high could throttle workloads that were previously running without throttling. Making it null ensures that upgrading to v1.37 does not change runtime behavior for existing clusters.

If your kubelet configuration file already contains an explicit memoryThrottlingFactor value, that value is preserved during the upgrade and throttling continues to work as before. If your configuration file does not include memoryThrottlingFactor, the kubelet uses the new null default and stops setting memory.high. To keep throttling in that case, add memoryThrottlingFactor explicitly:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9

How to configure MemoryQoS in v1.37

For full details on configuring Memory QoS, see Memory QoS with cgroup v2, Configuring memory reservation, and System requirements

Enable memory throttling only

Set memoryThrottlingFactor to a value between 0 and 1. The kubelet uses this factor to calculate memory.high for Burstable and BestEffort containers. See Memory throttling for how memory.high is calculated for each QoS class.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9

Enable memory throttling and tiered reservation

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9
memoryReservationPolicy: TieredReservation

Enable tiered reservation without throttling

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryReservationPolicy: TieredReservation

Disable Memory QoS entirely

To disable the feature after upgrading, set the feature gate to false and ensure a compatible kubelet configuration. The kubelet rejects the configuration if memoryThrottlingFactor is set to anything other than the former default of 0.9, or if memoryReservationPolicy is TieredReservation, so remove or adjust those fields if you set them.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
 MemoryQoS: false

When the feature gate is off, or memoryReservationPolicy is not TieredReservation, the kubelet resets stale protection at startup on cgroup v2 nodes: memory.min=0 and memory.low=0 on the root kubepods cgroup, and memory.low=0 on the Burstable QoS cgroup. For containers, stale memory.high values are reset to max on reconciliation paths such as restart or resize.

Known limitation: memory reservation is node-wide

memoryReservationPolicy applies to every pod on the node. With TieredReservation, every Guaranteed pod gets memory.min and every Burstable pod gets memory.low; there is no way to opt individual pods in or out. A node that mixes workloads needing hard reservation with workloads that should stay reclaimable has to choose one policy for all of them.

Hard reservation also covers everything charged to the container's cgroup, including page cache, so a pod that reads large files can hold memory the kernel would otherwise reclaim to serve its neighbors.

SIG Node is tracking both in kubernetes/kubernetes#140246. If this affects you, that issue is the best place to describe your workload.

What to expect next

The next milestone for Memory QoS is graduation to GA. Feedback from Beta users will shape any remaining adjustments before that step. If you run into issues, please file bugs at kubernetes/kubernetes.

How can I learn more?

• KEP-2570: Memory QoS
• Pod Quality of Service Classes
• Memory QoS with cgroup v2
• Managing Resources for Containers
• Kubernetes cgroups v2 support
• Linux kernel cgroups v2 documentation

Getting involved

This feature is driven by SIG Node. If you are interested in contributing or have feedback, you can reach out through:

• Slack: #sig-node
• Mailing list
• SIG Node meetings

14 Sep 2026 6:30pm GMT

Changed Block Tracking (CBT) support for CSI drivers shipped as Alpha in September 2025. With the March 2026 v1.0.0 release of the external-snapshot-metadata project, the feature moved to Beta.

If you aren't yet familiar with changed block tracking for storage in Kubernetes, the Alpha announcement covers the motivation, the three primary components (the CSI SnapshotMetadata gRPC service, the SnapshotMetadataService CRD, and the external-snapshot-metadata sidecar), and a walkthrough of how to use the API. CBT currently applies to block volumes; file-volume and network file-share changed-list tracking is not covered by this feature. This post focuses on what is different in Beta.

What's new in Beta

The main change in that release was the promotion of the SnapshotMetadataService CRD from v1alpha1 to v1beta1. The CRD used to advertise a driver's metadata service now serves cbt.storage.k8s.io/v1beta1. The schema itself is unchanged, but this release removed v1alpha1 (rather than serving it alongside the new version). If you are upgrading from Alpha, you need to:

• Re-apply the CRD definition shipped with v1.0.0.
• Update SnapshotMetadataService manifests to use apiVersion: cbt.storage.k8s.io/v1beta1.
• Update any client or controller code that talks to the CRD.

This is a one-time change. There is no automatic conversion between the two versions.

Compatibility

• Minimum Kubernetes version: 1.33
• CSI spec: 1.10 or newer
• Container image: registry.k8s.io/sig-storage/csi-snapshot-metadata:v1.0.0

Trying it out

The Getting Started section in the Alpha blog still applies. In short:

1. Make sure your CSI driver supports volume snapshots and ships the external-snapshot-metadata sidecar.
2. Install the SnapshotMetadataService CRD (the v1beta1 definition from the v1.0.0 release).
3. Create a SnapshotMetadataService resource for your driver.
4. Use a client - snapshot-metadata-lister, or your own implementation - to call GetMetadataAllocated and GetMetadataDelta.

If you want to see the full flow end-to-end, the hostpath driver example is a good starting point.

What's next?

The focus for the rest of the Beta cycle is wider CSI driver adoption and operational feedback before the feature moves towards GA. If you maintain a CSI driver, this is a good time to evaluate adding support. If you are building a backup application on top of the API, feedback on the streaming clients and the iterator package is very welcome.

Where can I learn more?

• The CSI developer documentation for snapshot metadata.
• KEP-3314.
• The external-snapshot-metadata repository.
• The gRPC schema.
• The snapshot-metadata-lister example client.

How do I get involved?

This work is the result of contributions from many people across SIG Storage. A big thank you to everyone who helped review, code, and test the feature through Alpha and into Beta:

• Ben Swartzlander (bswartz)
• Carl Braganza (carlbraganza)
• Daniil Fedotov (hairyhum)
• Ivan Sim (ihcsim)
• Nikhil Ladha (Nikhil-Ladha)
• Praveen M (iPraveenParihar)
• Rakshith R (Rakshith-R)
• Xing Yang (xing-yang)

If you would like to get involved with CSI or storage in Kubernetes, SIG Storage is the place to start. The Data Protection Working Group also holds regular meetings, and new attendees are always welcome.

14 Sep 2026 6:30pm GMT

I'm excited to announce that native histogram support for Kubernetes metrics is graduating to Beta and is enabled by default in Kubernetes v1.37!

Native histograms (previously introduced as Alpha in Kubernetes v1.36 under KEP-5808) bring high-resolution, low-cardinality observability to Kubernetes metrics. By adopting Prometheus Native Histograms, Kubernetes components now expose latency and duration metrics with far greater accuracy while significantly reducing telemetry storage and scraping overhead.

Why move beyond classic histograms?

Since the early days of Kubernetes observability, duration and latency metrics (such as API server request latencies or scheduling durations) have relied on classic Prometheus histograms.

Classic histograms require metric authors to define a static list of cumulative bucket boundaries (le labels), such as 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10. While familiar, this approach introduces three major challenges:

1. The Bucket Guessing Game: If a workload's latency profile changes, for example, shifting into microsecond ranges or experiencing long-tail tail latencies beyond the highest bucket, the histogram loses visibility. Specifying bucket boundaries upfront requires knowing the distribution before observing it
2. High Cardinality & Storage Cost: With classic histograms, each bucket boundary is exported as a separate time series (_bucket{le="..."}). A histogram with 10 buckets across multiple labels multiplies the number of time series by 10, increasing memory consumption in Prometheus and inflating time series database (TSDB) storage costs
3. Interpolation Error in Quantiles: Calculating percentiles using histogram_quantile() relies on linear interpolation between static bucket boundaries. When bucket spans are coarse, quantile calculations can suffer from significant estimation error

What are Prometheus native histograms?

Prometheus Native Histograms replace static user-defined buckets with dynamic, exponential buckets.

Instead of emitting a separate time series for every single bucket boundary, a native histogram is stored as a single time series containing a rich schema of positive and negative spans, zero thresholds, and exponential scaling factors.

• High Resolution Automatically: Exponential buckets dynamically adjust to any value range - from nanoseconds to hours - without requiring pre-configured bucket boundaries
• Up to 90% Fewer Time Series: By consolidating buckets into structured spans within a single time series, scraping and storage overhead are dramatically reduced
• Accurate Quantile Calculation: Quantiles can be calculated with mathematical bounds on error (≃5% worst-case relative error under default settings) across the entire spectrum of observations

How native histograms work in Kubernetes

In Kubernetes, native histogram support is implemented directly inside the shared metrics subsystem (k8s.io/component-base/metrics).

Figure 1 illustrates how native histogram metrics are processed and exposed across Kubernetes components.

1. Dual exposition for zero breaking changes

A primary design requirement for KEP-5808 was zero disruption for existing observability stacks. When the NativeHistograms feature gate is enabled, Kubernetes components use dual exposition:

• Classic buckets (h.Bucket) are still emitted alongside native spans. Existing Prometheus servers, dashboards, and alerting rules that rely on traditional text scraping or classic bucket labels continue to work unmodified
• Native spans (h.Schema, h.PositiveSpan) are included in the same Protobuf payload for collectors that understand native histograms

2. Tuned default exponential configuration

When NativeHistograms is enabled, the k8s.io/component-base/metrics package automatically applies standardized exponential options to all histogram metrics:

• BucketFactor: 1.1: Configures exponential buckets where each bucket is at most 10% wider than the preceding one. This guarantees a mathematically bounded worst-case relative error of at most ~5% for quantile calculations regardless of whether an operation takes 1 millisecond or 10 seconds.
• MaxBucketNumber: 160: Caps the maximum number of buckets per histogram to 160. Following OpenTelemetry SDK recommendations for base-2 exponential histogram aggregation, this limit protects component memory usage even under extreme outlier distributions.

3. Broad component support

Because native histograms are integrated into component-base/metrics, all major Kubernetes control plane and node components inherit support automatically, including:

• kube-apiserver (e.g., apiserver_request_duration_seconds, authentication/authorization metrics, validation latencies)
• kube-scheduler (e.g., scheduler_plugin_execution_duration_seconds, scheduler_scheduling_algorithm_duration_seconds)
• kubelet (node-level container runtime and pod lifecycle metrics)
• kube-controller-manager and kube-proxy

How to scrape native histograms

The simple answer: upgrade to Kubernetes v1.37, and it works.

Because Kubernetes v1.37 enables NativeHistograms by default, your cluster is already emitting dual-exposition metrics. How you configure Prometheus to scrape native histograms depends on your Prometheus version:

1. Prometheus scrape configuration by version

• Prometheus 3.0+ (Recommended): Use explicit per-job configuration in your scrape_configs rather than global flags (the global --enable-feature=native-histograms flag is deprecated in Prometheus 3.9+):

  scrape_configs:
   - job_name: 'kubernetes-apiservers'
   scrape_native_histograms: true
   always_scrape_classic_histograms: true # Recommended during transition
  

  You must read the caution in Migrating dashboards and alerts in the Native Histograms documentation. In summary: always set always_scrape_classic_histograms: true during your transition period. Without this setting, Prometheus will only ingest the native format and stop ingesting classic _bucket, _count, and _sum series. Setting always_scrape_classic_histograms: true ensures existing dashboards (histogram_quantile(..._bucket...)) and alerts continue to work while you migrate them to native histograms.

• Prometheus 2.40 - 2.x: Enable Native Histograms globally by starting Prometheus with the feature flag:

  prometheus --enable-feature=native-histograms
  

  Note that in Prometheus 2.x, this is an all-or-nothing setting for all scrape targets.

2. Verify Protobuf dual exposition

Standard Prometheus text scraping (application/openmetrics-text or plain text format) only transfers classic buckets. When scrape_native_histograms is enabled, Prometheus automatically negotiates Protobuf format with Kubernetes endpoints.

You can verify that a Kubernetes component is exporting native histograms using curl with an Accept header specifying Protobuf. For example:

## THIS IS NOT SECURE. ONLY DO THIS IN A TEST CONTEXT.
curl --insecure \
 -H "Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited" \
 --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
 https://localhost:6443/metrics

When decoded, the returned MetricFamily for histogram metrics (like apiserver_request_duration_seconds) will contain both traditional bucket entries and populated schema / positive_span fields.

Querying native histograms in PromQL

Once native histograms are ingested into Prometheus, you can query them using standard PromQL histogram functions without needing static le bucket labels or _bucket suffixes:

# 1. Calculating P99 latency for a single target:
# Classic histogram (requires _bucket suffix):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m]))

# Native histogram (operates directly on the metric name):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds[5m]))

# 2. Aggregating across multiple instances (e.g., all API servers):
# Classic histogram (requires sum by (le) to preserve bucket boundaries):
histogram_quantile(0.99, sum by (le) (rate(apiserver_request_duration_seconds_bucket[5m])))

# Native histogram (no grouping by le required!):
histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds[5m])))

With native histograms, functions like histogram_quantile() operate directly on the dynamic exponential spans inside the time series, producing highly accurate quantiles without static bucket interpolation error.

For official documentation on querying Native Histograms in PromQL, see:

• PromQL histogram_quantile documentation
• PromQL histogram_fraction documentation
• PromQL histogram_sum documentation
• PromQL histogram_count documentation
• PromQL histogram_count documentation
• PromQL histogram_stddev and histogram_stdvar documentation

Dashboard migration & rollback strategy

Recommended migration workflow

To safely transition your monitoring infrastructure to Native Histograms without breaking existing alerts or dashboards, I recommend a four-step migration workflow:

1. Enable Both Formats: In your Prometheus 3.x scrape config, set scrape_native_histograms: true AND always_scrape_classic_histograms: true so both formats are collected safely during transition
2. Migrate Queries: Update your Grafana dashboards and Prometheus alerting rules from classic quantile queries (histogram_quantile(..._bucket...)) to native histogram queries (histogram_quantile(...)), and replace references to classic _count and _sum series with histogram_count(...) and histogram_sum(...)
3. Verify in Staging/Production: Validate that all dashboards and SLO alerts fire and graph correctly using the new native histogram queries
4. Unlock ~10x Storage Savings: Once migration is complete, set always_scrape_classic_histograms: false. Prometheus will stop ingesting the static _bucket, _count, and _sum time series, reducing your histogram time series count by up to 90%!

Opt-out and rollback flexibility

Because native histograms are dual-exposed, using them is entirely opt-in from a collector perspective:

• Instant Collector Rollback: If you need to stop ingesting native histograms, simply set scrape_native_histograms: false in your Prometheus job configuration. No Kubernetes restart is required, and Prometheus will immediately resume scraping only the classic format without data loss
• Component Feature Gate Rollback: Administrators can also disable the feature gate on Kubernetes components using --feature-gates=NativeHistograms=false (requires component restart)

What's next & how to get involved

As native histograms progress toward General Availability (GA) in future Kubernetes releases, SIG Instrumentation will continue evaluating ecosystem readiness, performance characteristics, and long-term plans for eventually deprecating static classic buckets once native histogram adoption becomes ubiquitous across the monitoring community.

• Read the KEP-5808 page or the KEP GitHub issue to learn more.
• Read the Prometheus Native Histograms specification and PromQL querying functions documentation
• Get involved with SIG Instrumentation on Slack in #sig-instrumentation or join the weekly SIG meetings

Acknowledgements

A huge thank you to contributors across SIG Instrumentation and component owners who collaborated on the design, implementation, testing, and review of native histograms in Kubernetes!

11 Sep 2026 6:30pm GMT

In Kubernetes, resource allocation has historically been a static decision made during a Pod's initial scheduling and placement. With the graduation of the core in-Place Pod resize feature to General Availability in v1.35, application developers and cluster operators gained the powerful ability to dynamically adjust CPU and memory allocations of running containers without incurring disruptive restarts or application downtime.

However, in-place resizing introduced a unique resource scheduling gap: if a running Pod requested a resource scale-up that exceeded the host node's allocatable headroom, the Kubelet was forced to mark the request as Deferred. The Pod would remain parked in this state indefinitely, waiting for resources on the node to naturally free up.

To bridge this scheduling gap, Kubernetes v1.37 introduces scheduler preemption for in-place Pod resize (Alpha), behind the InPlacePodVerticalScalingSchedulerPreemption feature gate. This feature allows the Kubernetes scheduler to actively free up capacity on a fully-utilized node by preempting lower-priority workloads, enabling the pending in-place resizes of critical, higher-priority applications to succeed.

The "deferred" resize challenge

To understand why this preemption mechanism is needed, it is helpful to look at how Kubernetes handles running Pod resizing. When a user or controller (such as the Vertical Pod Autoscaler) updates the resource requests of an active container, the Kubelet evaluates whether the underlying node has enough spare allocatable capacity to fulfill the increase.

If the node's resources are fully utilized and cannot satisfy the new limits, the Kubelet sets the container's resizeStatus (reported in the Pod's status.containerStatuses[]) to Deferred. Unlike an Infeasible resize request (which is immediately rejected because it exceeds physical machine boundaries, namespace limit ranges, or admission quotas) a Deferred status indicates that the request is valid but is temporarily unable to be actuated, waiting until node capacity becomes available.

Before the introduction of this preemption mechanism, a Pod's in-place resize scale-up request could become permanently blocked if the node was heavily utilized. Even when a critical application (such as an in-memory database or a real-time web server) required more memory to prevent an imminent out-of-memory (OOM) crash, and the node lacked free capacity, the resize remained Deferred.

In this scenario, cluster administrators had limited choices:

1. Manually evict lower-priority Pods from the node to clear resource headroom.
2. Rely on the cluster autoscaler to eventually spin up a larger node and reschedule the Pod. However, this is an operation that is highly disruptive and violates the core "no restart" value proposition of in-place scaling.
3. Rely on a custom autoscaling solution, for example a cluster autoscaler that can trigger dynamic node resizing operations itself.

Because the kube-scheduler was unaware of deferred resizes on running Pods, it could not leverage standard priority-based preemption to evict lower-priority workloads and make room for the higher-priority running Pod's resource growth.

Why this matters

In production Kubernetes environments, cluster administrators strive to maximize resource utilization and efficiency. A common strategy is to bin-pack unused capacity on not-yet-full nodes with lower-priority workloads, such as batch jobs, background data processing, or best-effort tasks.

Without scheduler preemption for in-place resizing, this created a major operational dilemma. If lower-priority workloads consumed the remaining headroom on a node, higher-priority applications running on that same node would become blocked (Deferred) when they needed to scale up to handle sudden traffic surges or memory spikes. Operators were forced to choose between running low-utilization clusters with idle buffer capacity or risking that critical workloads could not resize when needed.

With scheduler preemption for in-place Pod resize, you can confidently bin-pack unused space across your clusters with lower-priority workloads without worrying about them degrading higher-priority Pods or blocking their scale-up requests. If a high-priority workload requires an in-place resize that exceeds available node capacity, the scheduler automatically preempts the lower-priority Pods to clear headroom. You achieve high cluster utilization and cost efficiency while preserving the responsiveness and reliability of critical services.

Architectural mechanics: How it works

Scheduler preemption for in-place Pod resize integrates directly into the core scheduling cycle to coordinate resources dynamically and safely.

Centralized scheduler tracking

The kube-scheduler monitors the cluster for running Pods with a Deferred resize status condition. Normally, Pods with spec.nodeName populated are considered successfully placed and bypass the active scheduling queue. Under this feature gate, the scheduler intercepts Pods carrying the Deferred condition, permitting them to remain in active scheduling evaluations specifically to trigger preemption. The scheduler maintains continuous tracking of these Pods until the Kubelet successfully completes the resize actuation.

Single-node preemption boundary

Unlike placement preemption, which evaluates all nodes in a cluster to find the best scheduling fit, preemption for in-place resizing is strictly localized to the Pod's currently assigned node. The scheduler identifies eligible lower-priority "victim" Pods on the same host and initiates their graceful eviction, freeing up local capacity. Preemption is strictly scoped to the same node where the deferred Pod is running; if a node cannot accommodate the resize even after evicting all eligible lower-priority workloads, the resize remains in the Deferred state.

Resource reservation safety

To prevent scheduling races and double-allocation, the scheduler treats resources requested for a resize as already consumed. This enables the Kubelet to actuate the resize once the preemption takes effect.

Separation of concerns & critical admission

When a node is under resource pressure, the Kubelet includes a local mechanism known as the critical Pod admission handler. During initial Pod admission, if a critical system Pod arrives on a node that lacks spare capacity, this local handler can directly evict lower-priority Pods on that node to guarantee admission for the critical workload.

A significant architectural benefit of this new feature is the strict separation of concerns between the Kubelet and the scheduler. Under the InPlacePodVerticalScalingSchedulerPreemption feature gate, the Kubelet's critical Pod admission handler does not perform local preemption checks or trigger local evictions for in-place resizing operations. Instead, the Kubelet defers the request and delegates the preemption decision entirely to the scheduler. This guarantees that a single, centralized orchestrator manages all resize-related preemption logic, respecting global priorities, Pod disruption budgets (PDBs), and graceful termination policies.

Managing competing updates & races

If a competing, higher-priority resize request is submitted for another running Pod on the same node during an active preemption cycle, the Kubelet prioritizes the higher-priority request. The scheduler is designed to observe these updates and will dynamically trigger a new round of preemption if more capacity is required to fulfill the new state.

Node-level preemption configuration

Administrators and automated controllers (such as a cluster autoscaler) can disable preemption specifically for in-place resizes on particular nodes. This is configured using the new spec.podPreemptionPolicy field in the Node Spec:

apiVersion: v1
kind: Node
metadata:
 name: batch-workload-node
spec:
 podPreemptionPolicy:
 disableResizePreemption:
 - "cluster-autoscaler.kubernetes.io/disable-preemption"
 - "operator.example.com/policy-override"

An example use case for this policy is when a controller would prefer to size down other pods or dynamically adjust the node capacity itself when possible, only enabling scheduler preemption as a last resort.

Try it out!

To utilize scheduler preemption for in-place Pod resize:

• Your cluster must be running Kubernetes v1.37 or later across both the control plane and all worker nodes.
• The InPlacePodVerticalScalingSchedulerPreemption feature gate must be enabled across all control plane components (kube-apiserver, kube-scheduler) and the kubelet.

Mini-tutorial: Observe resize preemption in action

To see this feature in action locally, you can test scheduler preemption on a single-node kind cluster with constrained CPU headroom.

1. Create a kind cluster with scheduler resize preemption enabled

Create a kind cluster configuration file named kind-config.yaml with the InPlacePodVerticalScalingSchedulerPreemption feature gate enabled:

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
 InPlacePodVerticalScalingSchedulerPreemption: true

Create the cluster using this configuration, passing the --image flag to ensure the cluster is running Kubernetes v1.37 (or later):

kind create cluster --config kind-config.yaml --image kindest/node:v1.37.0

Once your cluster is ready, inspect the node to check how many allocatable CPU cores it has:

kubectl get nodes -o custom-columns=NAME:.metadata.name,ALLOCATABLE_CPU:.status.allocatable.cpu

In a standard local kind environment, the output shows 8 allocatable CPU cores:

NAME ALLOCATABLE_CPU
kind-control-plane 8

2. Create PriorityClasses and deploy Pods

Create two PriorityClasses and deploy a low-priority Pod (requesting 3 CPU) alongside a high-priority Pod (requesting 4 CPU). Together, these workloads consume 7 of the 8 available CPU cores, leaving 1 CPU of free allocatable headroom on the node.

# preemption-demo.yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
 name: high-priority
value: 1000000
globalDefault: false
description: "High priority workload"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
 name: low-priority
value: 1000
globalDefault: false
description: "Low priority workload"
---
apiVersion: v1
kind: Pod
metadata:
 name: low-priority-pod
spec:
 priorityClassName: low-priority
 containers:
 - name: worker
 image: nginx
 resources:
 requests:
 cpu: "3"
 memory: "500Mi"
 limits:
 cpu: "3"
 memory: "500Mi"
---
apiVersion: v1
kind: Pod
metadata:
 name: high-priority-pod
spec:
 priorityClassName: high-priority
 containers:
 - name: app
 image: nginx
 resources:
 requests:
 cpu: "4"
 memory: "1Gi"
 limits:
 cpu: "4"
 memory: "1Gi"

Save this manifest to preemption-demo.yaml and apply it:

kubectl apply -f preemption-demo.yaml

Wait until both Pods are running on the node:

kubectl get pods

Output:

NAME READY STATUS RESTARTS AGE
high-priority-pod 1/1 Running 0 9s
low-priority-pod 1/1 Running 0 9s

3. Request an in-place scale-up

Patch the high-priority Pod to increase its CPU request from 4 to 6 (+2 CPU delta). Because only 1 CPU of headroom is free on the node, this resize request exceeds remaining allocatable capacity:

kubectl patch pod high-priority-pod --subresource resize --patch \
 '{"spec":{"containers":[{"name":"app", "resources":{"requests":{"cpu":"6"}, "limits":{"cpu":"6"}}}]}}'

4. Inspect the preemption event on the low-priority Pod

With InPlacePodVerticalScalingSchedulerPreemption enabled, the scheduler intercepts the Deferred resize condition on high-priority-pod and targets low-priority-pod for preemption.

To verify that the scheduler actively preempted the low-priority Pod, inspect its events:

kubectl get events --field-selector involvedObject.name=low-priority-pod

In the event stream (or via kubectl describe pod low-priority-pod), you will see a Preempted event emitted by the scheduler:

LAST SEEN TYPE REASON OBJECT MESSAGE
5s Normal Preempted pod/low-priority-pod Preempted by pod 97dba925-6b5f-4e2f-99f9-d51c30016586 on node kind-control-plane
5s Normal Killing pod/low-priority-pod Stopping container worker

5. Trace the resize event lifecycle on the high-priority Pod

Next, inspect the event history on high-priority-pod to observe how the resize progressed from being deferred to successfully completed:

kubectl get events --field-selector involvedObject.name=high-priority-pod

You will observe a sequence of events as the Kubelet coordinates with the scheduler:

LAST SEEN TYPE REASON OBJECT MESSAGE
33s Warning ResizeDeferred pod/high-priority-pod Pod resize OutOfcpu: {"containers":[{"name":"app","resources":{"limits":{"cpu":"6","memory":"1Gi"},"requests":{"cpu":"6","memory":"1Gi"}}}],"generation":2,"error":"Node didn't have enough resource: cpu, requested: 6000, used: 3950, capacity: 8000"}
32s Normal ResizeStarted pod/high-priority-pod Pod resize started: {"containers":[{"name":"app","resources":{"limits":{"cpu":"6","memory":"1Gi"},"requests":{"cpu":"6","memory":"1Gi"}}}],"generation":2}
32s Normal ResizeCompleted pod/high-priority-pod Pod resize completed: {"containers":[{"name":"app","resources":{"limits":{"cpu":"6","memory":"1Gi"},"requests":{"cpu":"6","memory":"1Gi"}}}],"generation":2}

1. ResizeDeferred: The Kubelet initially marks the resize request as deferred (Warning) due to insufficient CPU headroom on the node (OutOfcpu).
2. ResizeStarted: Once the scheduler preempts low-priority-pod and capacity is released, the Kubelet accepts the new allocation and begins actuating the resize.
3. ResizeCompleted: The Kubelet successfully updates container cgroup limits via the container runtime without restarting the Pod.

Finally, verify that the allocated CPU on the container reflects the new request (appending {"\n"} to the JSONPath query ensures a trailing newline in your terminal):

kubectl get pod high-priority-pod -o jsonpath='{.status.containerStatuses[0].allocatedResources.cpu}{"\n"}'

Output:

6

This confirms that the in-place resize succeeded.

Getting involved

This feature represents a major step forward for resource scheduling, bringing enterprise-grade density control and workload prioritization to dynamic resource scaling. We invite cluster operators, platform architects, and developers to enable the InPlacePodVerticalScalingSchedulerPreemption feature gate in their testing environments and share feedback.

If you want to share your experience with this feature, please get in touch with the community via SIG Scheduling or SIG Node channels!

10 Sep 2026 6:30pm GMT

Kubernetes has many ways to describe what is happening on a Node. Readiness, taints, Pod state, labels, annotations, and provider-specific APIs each expose part of the picture. What has been missing is a shared, Kubernetes-owned way to say that a Node is draining, undergoing maintenance, or undergoing Graceful Node Shutdown.

Kubernetes v1.37 introduces five well-known Node conditions that provide that description:

• DrainInProgress
• Drained
• MaintenancePlanned
• MaintenanceInProgress
• GracefulNodeShutdownInProgress

The new Node lifecycle conditions

Maintenance can include hardware or software rollout, remediation, decommissioning, or debugging. Whether maintenance requires a drain depends on its impact. A Kubernetes upgrade usually should follow a drain, while a kernel live patch might not need one.

Like other Node conditions, each lifecycle condition uses status to report whether the observation is active:

• True: the lifecycle state is currently observed.
• False: the lifecycle state is not currently observed.
• Unknown: Kubernetes cannot determine whether the lifecycle state is active.

The reason provides a stable, machine-readable cause for the current status, and message can provide additional human-readable detail.

For example, an authorized maintenance controller could publish:

# Node .status excerpt
status:
 conditions:
 - type: MaintenancePlanned
 status: "True"
 reason: MaintenanceWindow
 lastTransitionTime: "2026-12-09T12:00:00Z"
 message: "Hardware maintenance is scheduled for this Node"

What changes in Kubernetes v1.37

The v1.37 release reserves these names as well-known NodeConditionType constants and introduces the Alpha NodeLifecycleConditions feature gate, which is disabled by default. In v1.37 the gate is effectively a no-op: it does not restrict who can set these conditions, and no core component reads them. It exists so that the built-in behavior planned for future releases - controllers that consume these conditions - can be opted into when it arrives. You do not need to enable it to start publishing the conditions today.

For this release, an administrator or an administrator-authorized controller is responsible for setting and clearing the lifecycle conditions.

In this first release, no core workload controller changes its behavior based on these conditions, but an administrator can publish them to communicate maintenance and drains to cluster users.

How to use lifecycle conditions today

The immediate value is operational clarity. Administrators and lifecycle automation can use these conditions as a common status channel for Node lifecycle work that already happens today.

For example, maintenance automation can set MaintenancePlanned when a future maintenance window is scheduled, then set MaintenanceInProgress when work starts. Drain automation can set DrainInProgress when it begins evicting Pods and Drained when the administrator's selected drain criteria have been met. The GracefulNodeShutdownInProgress condition can report that Graceful Node Shutdown is in progress on the Node.

The recommended pattern is to use lifecycle conditions to report status, while lifecycle operations are managed through other mechanisms. Continue to use existing Kubernetes mechanisms such as kubectl cordon, kubectl drain, taints, and workload-specific controls to change scheduling or eviction behavior. Use lifecycle conditions to make the state of that work visible to people, dashboards, alerts, and automation that choose to consume the signal.

When setting a condition, use True while the lifecycle state is active. Set the condition to False, or remove it, when the state is no longer active. Use a stable reason value and a clear message so that both people and automation can understand why the condition changed. Cluster administrators should also decide which component owns each lifecycle condition to avoid conflicting writes.

Why a shared signal matters

Node lifecycle affects components across the cluster. The kubelet, node lifecycle controller, workload controllers, scheduler, autoscalers, storage operators, and external maintenance systems all need some understanding of what is happening to a Node.

Today, each component has to reconstruct that understanding from indirect signals. One controller might look at Node readiness, another at taints, and another at Pods that are terminating or missing. Infrastructure providers and operators often add their own labels or annotations.

Those signals remain useful for their intended purposes, but they do not answer the same question. A taint can influence scheduling or eviction, for example, but it does not attest that a drain is in progress or that an administrator's drain criteria have been met. A NotReady Node does not explain whether the cause is an unexpected failure, a graceful shutdown, or planned maintenance.

Without shared lifecycle context, independently correct components can make conflicting decisions. A DaemonSet controller can replace a Pod that the kubelet intentionally terminated during graceful shutdown. A Job controller can wait indefinitely for a terminal Pod phase on a Node that an administrator is removing. A storage operator might learn about maintenance only after drain has already started.

The new conditions provide a stable place on the Node for that missing context, as part of the larger effort to enhance Node Lifecycle management.

The foundation for lifecycle-aware Kubernetes

The value of a shared signal comes from what can consume it - core controllers, administrators, or the ecosystem of lifecycle projects. Follow-up enhancements can build on the conditions without every component inventing a different way to infer Node lifecycle state.

Consider a long-standing DaemonSet rollout edge case. A Node that is broken or undergoing maintenance can remain unavailable for reasons unrelated to the new DaemonSet revision. That Node still consumes the rollout's availability budget, which can slow or block the controller from progressing the rollout on healthy Nodes.

The DaemonSet controller knows that a Pod is unavailable, but it cannot tell whether the new revision failed or an administrator intentionally took the Node out of service. Readiness, taints, and Pod state expose pieces of the situation, but none provides authoritative maintenance context.

The MaintenanceInProgress condition creates a Kubernetes-owned place to publish that context. Future work can define how the DaemonSet controller uses it for rollout ordering, availability accounting, and status reporting. Those behaviors still require careful design, but the goal is for administrators to no longer have to manually adjust the rollout.

Future expansions and getting involved

Node lifecycle is a cross-cutting problem. Solving it starts with components sharing enough context to make compatible decisions. The next stage is to build on Node Lifecycle Conditions to improve scenarios such as Graceful Node Shutdown, drain, and maintenance. Longer-term lifecycle coordination may require explicit ownership, locking, and potentially a dedicated API.

The Kubernetes ecosystem already includes many solutions for Node maintenance, remediation, drain, autoscaling, and fleet management. The experience behind those projects is essential to building a foundation that works across different environments and operational models. The Node Lifecycle Working Group, SIG Node, and SIG Apps invite maintainers and users to share their use cases and ideas to shape the future work.

Follow the work through KEP-5683: Node Lifecycle Conditions. To participate in our discussions, join one of our groups:

• Node Lifecycle Working Group
• SIG Node
• SIG Apps

09 Sep 2026 6:30pm GMT

AI/ML and complex batch workloads continue to push the boundaries of Kubernetes scheduling. Following the foundational workload-centric enhancements introduced in previous releases, Kubernetes v1.37 delivers the next major milestone in the Workload-Aware Scheduling (WAS) journey. In this release, the core Workload and PodGroup APIs-enabling gang scheduling-along with Workload-Aware Preemption (WAP) and shared DRA ResourceClaims for PodGroups, all graduate to Beta, solidifying their role in the Kubernetes ecosystem.

To address the hierarchical scheduling requirements of modern high-performance distributed workloads, v1.37 introduces the new CompositePodGroup API. This new API allows expressing multi-level topology constraints, gang scheduling, and preemption policies for complex, heterogeneous groups of Pods. Crucially, this architectural expansion unlocks native scheduling support for advanced workload structures commonly managed by higher-order extension APIs such as JobSet and LeaderWorkerSet (LWS).

Alongside these API additions, v1.37 focuses on streamlining adoption by introducing a new set of controller integration APIs and the workloadbuilder Go library. These provide standardized building blocks that significantly simplify how out-of-tree controllers can integrate with WAS capabilities. Utilizing these new tools, the native Job controller integration has been upgraded to fully consume the expanded WAS APIs-enabling advanced scheduling policies, flexible disruption modes, and topology-aware scheduling for standard batch workloads.

Gang scheduling and Workload / PodGroup APIs

Kubernetes v1.37 delivers a major milestone: Workload / PodGroup APIs and gang scheduling are officially graduating to Beta. This graduation signals that native, "all-or-nothing" scheduling for workloads is solidifying for wider adoption.

Key updates to the API and gang scheduling algorithm in this release include:

Beta graduation and API versioning changes

The core Workload and PodGroup APIs have been promoted to v1beta1, meaning they are now one step away from General Availability (GA). For early adopters who have been testing these features, take note of the alpha versioning transition: v1alpha2 has been entirely replaced by v1alpha3. This transition introduces breaking changes designed to clean up the API structure around disruptionMode.

Native PodGroup queueing

A significant under-the-hood improvement in v1.37 makes the PodGroup a first-class citizen in the scheduling queue. Previously, even if belonging to a PodGroup, all member Pods were queued individually. Now, only the top-level PodGroup object is queued. This ensures all Pods share the same queueing behavior and lays the groundwork for more advanced PodGroup queueing strategies in the future.

Dynamic elasticity with minCount mutability

In earlier iterations, the minCount field, which dictates the minimum number of Pods required to successfully schedule a PodGroup, was strictly immutable. In v1.37, minCount is now mutable. This API change unlocks flexibility for elastic workloads. Controllers can now dynamically adjust the minimum required size of a gang on the fly, allowing workloads to gracefully degrade or expand without interrupting already-scheduled Pods.

Workload-aware preemption

In Kubernetes v1.37 the separate WorkloadAwarePreemption feature gate for workload-aware preemption was merged into the GenericWorkload feature gate, becoming a core part of the gang scheduling effort.

While the core concepts of workload-aware preemption stay the same, there are some differences between the v1.36 and v1.37 releases:

Performance and optimality

To check whether a preemptor can fit in the cluster thanks to preemption, the scheduler simulates the removal of all potential victims and re-runs the scheduling algorithm. After that it tries to reprieve as many victims as possible. In the v1.36 release, the scheduling algorithm was run for each victim reprieval, verifying whether with the victim reprieved, the algorithm can still find a valid placement for the preemptor. In v1.37, the scheduling algorithm is run only once and the preemptor Pods are assumed based on its output. Later, the reprieval checks whether a victim can still run in its place with the preemptor assumed.

PodGroup as a victim

One of the limitations of v1.36 was the fact that the default preemption for single Pods was not aware of PodGroups and was not respecting their disruptionMode fields, allowing for disruption of single Pods even when the PodGroup had disruptionMode: {all: {}} set. Kubernetes v1.37 removes this limitation; the default preemption now respects the PodGroup disruptionMode field.

Rename of the disruptionMode fields

During the promotion of the API to Beta, the disruptionMode field was changed to decouple its naming from the PodGroup object, allowing consistent naming across PodGroups and CompositePodGroups. The modes changed as follows: PodGroup became all, and Pod became single.

Support for preemptionPolicy

In v1.36, the PodGroup does not have a preemptionPolicy field. The PodGroup can perform preemption as long as none of the Pods forming it has preemptionPolicy: Never set. In v1.37, when the PodGroupPreemptionPolicy feature gate is enabled, a PodGroup also has a preemptionPolicy field. It serves as an authoritative field for whether a PodGroup can perform preemption.

CompositePodGroup API

In Kubernetes v1.36, workload-aware scheduling established a clean separation between static workload templates (Workload) and runtime group state (PodGroup), but the supported scheduling policies were limited to a single, flat group. The CompositePodGroup API, introduced in Kubernetes v1.37, extends this model to support hierarchical scheduling requirements.

This API allows its consumers to express multi-level scheduling requirements by organizing a workload in a tree-shaped hierarchy consisting of CompositePodGroup and PodGroup objects. Each CompositePodGroup carries policies and constraints that apply to other groups (CompositePodGroups and/or PodGroups), similar to how PodGroups govern scheduling behavior for a flat group of Pods. The scheduler treats such a hierarchy as a single scheduling unit and aims to satisfy the requirements specified by every group within that hierarchy.

Defining a workload hierarchy

To express multi-level scheduling requirements, you define a hierarchy of templates in a Workload object. Controllers then create the corresponding CompositePodGroup and PodGroup objects from that hierarchy.

To support this, the Workload API is extended with the spec.compositePodGroupTemplates field. Each CompositePodGroupTemplate defines a template for a parent CompositePodGroup and directly nests the templates (podGroupTemplates and/or compositePodGroupTemplates) from which its child groups derive.

Below is a sample Workload object that defines a two-level template hierarchy:

apiVersion: scheduling.k8s.io/v1beta1
kind: Workload
metadata:
 name: example-workload
 annotations:
 kubernetes.io/description: "Two-level workload hierarchy requiring 4 worker Pods and 1 driver Pod to schedule together."
spec:
 compositePodGroupTemplates:
 - name: workload-root
 schedulingPolicy:
 gang:
 minGroupCount: 2
 podGroupTemplates:
 - name: workers
 schedulingPolicy:
 gang:
 minCount: 4
 - name: driver
 schedulingPolicy:
 gang:
 minCount: 1

After creating example-workload, a controller can stamp out the corresponding runtime group objects from these templates:

1. A root CompositePodGroup that references the workload-root template in example-workload and carries its group-level scheduling policy (gang scheduling with minGroupCount: 2):

   apiVersion: scheduling.k8s.io/v1alpha3
   kind: CompositePodGroup
   metadata:
    name: example-root-group
    annotations:
    kubernetes.io/description: "Root group coordinating gang scheduling across child worker and driver PodGroups."
   spec:
    workloadRef:
    workloadName: example-workload
    templateName: workload-root
    schedulingPolicy:
    gang:
    minGroupCount: 2
   

2. Two child PodGroup objects (example-workload-workers and example-workload-driver) that reference their respective leaf templates in example-workload and link to the root group via parentCompositePodGroupName:

   apiVersion: scheduling.k8s.io/v1beta1
   kind: PodGroup
   metadata:
    name: example-workload-workers
    annotations:
    kubernetes.io/description: "Worker group requiring at least 4 Pods to be scheduled together."
   spec:
    parentCompositePodGroupName: example-root-group
    workloadRef:
    workloadName: example-workload
    templateName: workers
    schedulingPolicy:
    gang:
    minCount: 4
   ---
   apiVersion: scheduling.k8s.io/v1beta1
   kind: PodGroup
   metadata:
    name: example-workload-driver
    annotations:
    kubernetes.io/description: "Driver group requiring 1 Pod to schedule alongside the workers."
   spec:
    parentCompositePodGroupName: example-root-group
    workloadRef:
    workloadName: example-workload
    templateName: driver
    schedulingPolicy:
    gang:
    minCount: 1
   

How multi-level gang scheduling works

To schedule a hierarchical workload, kube-scheduler evaluates the entire group tree as a unified scheduling unit:

• Recursive evaluation: The scheduler traverses the hierarchy from the root CompositePodGroup down to the leaf PodGroup objects. At each level, a parent CompositePodGroup is considered schedulable only when its child groups satisfy its scheduling policy (for example, placing at least minGroupCount of child groups when using the gang policy), while each leaf PodGroup must satisfy its own Pod-level policy (for example, placing at least minCount of member Pods when using the gang policy).
• All-or-nothing scheduling: Once a valid combination of child groups is found that satisfies the requirements of the root CompositePodGroup, the Pods across the entire hierarchy are scheduled and bound atomically. If the root group cannot satisfy its policy constraints, the entire hierarchy remains unschedulable and no Pods are bound, preventing partial deployments and deadlocks.

Workload-aware preemption for the CompositePodGroup API

Kubernetes v1.37 extends workload-aware preemption to support CompositePodGroup hierarchies as well. Specifically, if a CompositePodGroup cannot be scheduled due to insufficient capacity in the cluster, the scheduler can invoke preemption to evict lower-priority workloads in order to fit the Pods belonging to that CompositePodGroup.

A CompositePodGroup can be selected for preemption as well. To specify the desired behavior during preemption, workload owners can specify an appropriate disruptionMode in the CompositePodGroup spec:

• single: Allows individual child groups within the CompositePodGroup to be preempted and disrupted independently. This is the behavior when disruptionMode is not set.
• all: Enforces "all-or-nothing" disruption semantics across the entire CompositePodGroup hierarchy. If any Pod within the descendant subtree must be preempted, the scheduler evicts all Pods across the entire hierarchy together.

Topology-aware scheduling

In Kubernetes v1.37, topology-aware scheduling expands to support complex, multi-level workload hierarchies and delivers performance improvements for existing single-level deployments.

Multi-level topology-aware scheduling

In Kubernetes v1.36, we introduced foundational topology-aware scheduling, allowing you to define co-location constraints directly on a PodGroup. While effective for single-level groupings, complex distributed workloads-such as large-scale AI/ML training, JobSet deployments, or disaggregated inference via LeaderWorkerSet (LWS)-often require co-location across multiple levels of cluster infrastructure simultaneously.

For example, an entire workload may need to run within a single availability zone, while different parts of that workload (such as specific worker groups or driver processes) require strict co-location within specific server racks.

In Kubernetes v1.37, alongside the new CompositePodGroup API (scheduling.k8s.io/v1alpha3), topology-aware scheduling expands to support multi-level topology-aware scheduling. You can now express complex co-location requirements by specifying topology constraints at different levels of a group hierarchy.

Top-down topology constraint resolution

During hierarchical scheduling, the kube-scheduler resolves multi-level topology constraints in a top-down manner. Specifically, topology domains that are considered during the scheduling of a child group are confined within a topology domain that corresponds to the placement assumed by the parent group.

Configuration and runtime execution

Using the updated Workload API (scheduling.k8s.io/v1beta1), you can configure multi-level topology constraints directly within compositePodGroupTemplates. In the example below, the parent template constrains the overall workload to a single availability zone (topology.kubernetes.io/zone), while child templates for workers and driver constrain their respective Pods to server racks (topology.example.com/rack) within that selected zone:

apiVersion: scheduling.k8s.io/v1beta1
kind: Workload
metadata:
 name: multi-level-tas-workload
 namespace: job-ns
 annotations:
 kubernetes.io/description: "Workload defining zone-level co-location for the root group and rack-level co-location for child groups."
spec:
 compositePodGroupTemplates:
 - name: root
 schedulingPolicy:
 gang:
 minGroupCount: 2
 schedulingConstraints:
 topology:
 - key: topology.kubernetes.io/zone
 podGroupTemplates:
 - name: workers
 schedulingPolicy:
 gang:
 minCount: 8
 schedulingConstraints:
 topology:
 - key: topology.example.com/rack
 - name: driver
 schedulingPolicy:
 gang:
 minCount: 1
 schedulingConstraints:
 topology:
 - key: topology.example.com/rack

When a controller creates an instance of this workload at runtime, it spawns the corresponding runtime objects from these templates:

1. The root CompositePodGroup referencing the root template, carrying the availability zone topology constraint and the hierarchical gang scheduling policy.
2. The two child PodGroup objects (tas-workload-workers and tas-workload-driver), each referencing the root CompositePodGroup as their parent group via the parentCompositePodGroupName spec field:

apiVersion: scheduling.k8s.io/v1alpha3
kind: CompositePodGroup
metadata:
 name: tas-workload-root
 namespace: job-ns
 annotations:
 kubernetes.io/description: "Root group constraining the entire workload to a single availability zone."
spec:
 workloadRef:
 workloadName: multi-level-tas-workload
 templateName: root
 schedulingPolicy:
 gang:
 minGroupCount: 2
 schedulingConstraints:
 topology:
 - key: topology.kubernetes.io/zone
---
apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
 name: tas-workload-workers
 namespace: job-ns
 annotations:
 kubernetes.io/description: "Worker group requiring 8 Pods co-located within a single rack in the selected zone."
spec:
 parentCompositePodGroupName: tas-workload-root
 workloadRef:
 workloadName: multi-level-tas-workload
 templateName: workers
 schedulingPolicy:
 gang:
 minCount: 8
 schedulingConstraints:
 topology:
 - key: topology.example.com/rack
---
apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
 name: tas-workload-driver
 namespace: job-ns
 annotations:
 kubernetes.io/description: "Driver group requiring 1 Pod placed in a rack within the selected zone."
spec:
 parentCompositePodGroupName: tas-workload-root
 workloadRef:
 workloadName: multi-level-tas-workload
 templateName: driver
 schedulingPolicy:
 gang:
 minCount: 1
 schedulingConstraints:
 topology:
 - key: topology.example.com/rack

During scheduling, the scheduler evaluates multiple candidate availability zones across the cluster for tas-workload-root. For each candidate zone, it subdivides the nodes by rack topology to explore feasible rack placements for tas-workload-workers and tas-workload-driver strictly within that zone, systematically evaluating multiple combinations across available zones and racks before making a scheduling decision.

By allowing topology constraints to be modeled hierarchically, Kubernetes v1.37 provides a structured way to express multi-level co-location requirements across complex cluster infrastructures.

Performance improvements for single-level TAS

Alongside the Alpha introduction of multi-level hierarchies, Kubernetes v1.37 reduces the cost of placement evaluation for existing single-level topology-aware scheduling. We are continuously working to optimize the efficiency of placement evaluation algorithms in kube-scheduler and plan to deliver further performance improvements in future releases.

Controller Integration APIs

Kubernetes v1.37 introduces new standard building blocks so that every controller can expose the same scheduling primitives in their own APIs, and share the same logic for translating them into scheduling objects. These primitives express specific scheduling behaviors - such as policies or disruption logic - while leaving the field naming flexible for each controller. A prime example of this is the native Job controller, which we detail in the next section.

Types prefixed with WorkloadPodGroup describe a leaf group of Pods; types prefixed with WorkloadCompositePodGroup describe a group of groups. A controller embeds them verbatim into its own API, under whatever field name fits its domain:

• WorkloadPodGroupSchedulingPolicy - either basic, meaning standard Pod-by-Pod scheduling, or gang with a minCount. The composite variant takes a minGroupCount instead.
• WorkloadPodGroupSchedulingConstraints - the topology constraints (topology[].key) the group's Pods must be co-located within.
• WorkloadPodGroupDisruptionMode - single or all, with the preemption semantics described earlier in this post.
• WorkloadPodGroupResourceClaim - the ResourceClaims shared across the group.

Only the shapes are shared, so controllers retain full autonomy over how they name and nest these fields in their own APIs.

The workloadbuilder library turns that intent into the scheduling objects. A controller describes its workload as a tree of WorkloadItem nodes - a node with children compiles to a CompositePodGroupTemplate, a node without children to a PodGroupTemplate - and attaches its own defaults plus the user-supplied building blocks to each node. From there, Validate() reports problems back at the exact field path within the controller's own API, BuildWorkload() compiles the tree into a Workload, and NewPodGroup() and NewCompositePodGroup() stamp out the runtime group objects.

Validation is deny-by-default: a controller declares the policies and disruption modes it actually supports through AllowedPolicies and AllowedDisruptionModes, and anything outside those lists is rejected. Building blocks added in future releases therefore stay unavailable until a controller explicitly opts into them.

For hierarchical workloads where a parent controller owns the Workload and delegates group creation to its children, NewBuilderFromExistingWorkload lets a child materialize only its own PodGroup from the parent's Workload.

Neither the building blocks nor the library have a feature gate of their own; they become user-visible through whichever controller adopts them. The native Job controller is the first to do so, and we detail it in the next section.

Integration with the Job controller

Building upon the new controller integration APIs, the Job API now features an explicit .spec.scheduling field, so you can declare how a Job should be scheduled instead of relying on the Job controller to infer it from the Job's shape. This expands support well beyond static, indexed, and fully-parallel Jobs.

.spec.scheduling is composed of the building blocks described above:

• schedulingPolicy - basic for standard Pod-by-Pod scheduling, or gang for all-or-nothing scheduling.
• schedulingConstraints - the topology domain the Job's Pods must be co-located within.
• disruptionMode - whether the Job's Pods can be preempted individually (single) or only as a whole (all).
• resourceClaims - the ResourceClaims shared by all of the Job's Pods.

For example:

apiVersion: batch/v1
kind: Job
metadata:
 name: distributed-training-job
 annotations:
 kubernetes.io/description: "Distributed Job using explicit WAS scheduling with gang policy and zone topology constraints."
spec:
 parallelism: 8
 completions: 8
 scheduling:
 schedulingPolicy:
 gang: {} # minCount omitted → defaults to parallelism (8)
 schedulingConstraints:
 topology:
 - key: topology.kubernetes.io/zone
 disruptionMode:
 all: {}
 template:
 spec:
 containers:
 ...

Omitting .spec.scheduling, or omitting schedulingPolicy within it, selects the basic policy, which behaves exactly like standard Job scheduling today.

For every Job it manages, the controller compiles this configuration into a Workload and a PodGroup owned by the Job, and sets .spec.schedulingGroup.podGroupName on each Pod it creates so the scheduler treats them as one group. Once created, .spec.scheduling is immutable, with one exception: schedulingPolicy.gang.minCount can be updated, which lets you resize a running gang.

DRA ResourceClaim support for workloads

As the core WAS APIs mature, so do their integrations with Dynamic Resource Allocation (DRA). Kubernetes v1.36 introduced the DRAWorkloadResourceClaims feature gate. The associated feature allows ResourceClaims to be replicated and reserved for entire PodGroups and shared by all their member Pods:

apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
 name: training-job-workers-pg
spec:
 ...
 resourceClaims:
 - name: pg-claim
 resourceClaimTemplateName: my-claim-template
---
apiVersion: v1
kind: Pod
metadata:
 name: topology-aware-workers-pg-pod
spec:
 ...
 schedulingGroup:
 podGroupName: training-job-workers-pg
 resourceClaims:
 - name: pg-claim
 resourceClaimTemplateName: my-claim-template

In Kubernetes v1.37, the DRAWorkloadResourceClaims feature gate graduated to Beta.

While the API and core functionality of the feature remain unchanged, one change eliminates some potentially surprising behavior when disabling the feature. Previously when one of a Pod's spec.resourceClaims referenced a ResourceClaimTemplate and matched one of its PodGroup's spec.resourceClaims and the DRAWorkloadResourceClaims feature gate was disabled, a ResourceClaim was created for the Pod instead of the PodGroup. In that scenario in v1.37, no ResourceClaim is created at all. This change prevents Kubernetes from creating a flood of ResourceClaims from a ResourceClaimTemplate and potentially exhausting DRA resources when a claim intended to be shared by a whole PodGroup is replicated for each and every Pod in the group.

For more information, see the feature documentation.

What's next?

The Workload-Aware Scheduling Working Group (WG WAS) is currently finalizing its plans for the Kubernetes v1.38 release cycle. While the roadmap is still taking shape (stay tuned!), the following key initiatives are already planned:

• Graduation of Workload and PodGroup APIs to GA: Solidifying the core foundation of workload-aware scheduling as a stable Kubernetes API.
• Graduation of Topology-Aware Scheduling (TAS) and CompositePodGroup (CPG) to Beta: Bringing these advanced placement and hierarchical scheduling features to Beta stability.
• Graduation of controller integration building blocks to Beta: Further refining the integration APIs to ensure a robust developer experience.
• Increased adoption and integration: Expanding the ecosystem by integrating workload-aware scheduling with other controllers, with a particular focus on hierarchical orchestrators such as JobSet.
• Kueue Integration: Fostering closer alignment between WAS and Kueue. In the near term, we aim to ensure Kueue is fully aware of WAS features for seamless interoperability. In the long term, we envision Kueue leveraging WAS as its underlying engine for capabilities like gang-scheduling and topology-aware placement.

Getting started

Many of the workload-aware scheduling improvements are now available as Beta features in v1.37, while new advanced capabilities are introduced in Alpha. Both Beta and Alpha features here are disabled by default and require manual enablement.

Beta features:

• Workload API, gang scheduling, and preemption: The GenericWorkload feature gate (which now integrates gang scheduling and workload-aware preemption) is Beta and disabled by default on the kube-apiserver, kube-controller-manager and kube-scheduler. Ensure your manifests are updated to use the scheduling.k8s.io/v1beta1 API group.
• DRA ResourceClaim support for workloads: Enable the DRAWorkloadResourceClaims feature gate on the kube-apiserver, kube-controller-manager, kube-scheduler and kubelet.

Alpha features:

• Topology-aware scheduling: Enable the TopologyAwareWorkloadScheduling feature gate on the kube-apiserver and kube-scheduler.

• CompositePodGroup API: Enable the CompositePodGroup feature gate on the kube-apiserver, kube-controller-manager and kube-scheduler, and ensure the scheduling.k8s.io/v1alpha3 API version is enabled. Note that enabling CompositePodGroup on the kube-controller-manager also requires the TopologyAwareWorkloadScheduling feature gate to be enabled.

• Workload API integration with the Job controller: Enable the WorkloadWithJob feature gate on the kube-apiserver and kube-controller-manager.

• PodGroup preemptionPolicy: Enable the PodGroupPreemptionPolicy feature gate on the kube-apiserver and kube-scheduler.

Controller integration APIs:

The new workloadbuilder library is available to developers building both out-of-tree and in-tree controllers who want to integrate with WAS. It does not require a feature gate. You can explore the library and find usage examples directly in the kubernetes/component-helpers repository.

We encourage you to try out workload-aware scheduling in your test clusters and share your experiences to help shape the future of Kubernetes scheduling. You can send your feedback by:

• Reaching out via Slack (#wg-workload-aware-scheduling).
• Joining the WG Workload-Aware Scheduling or SIG Scheduling meetings.
• Filing a new issue in the Kubernetes repository.

Learn more

To dive deeper into the architecture and design of these features, read the KEPs:

• KEP-4671: Gang Scheduling Support in Kubernetes
• KEP-5710: Workload-aware preemption
• KEP-5732: Topology-aware workload scheduling
• KEP-6012: CompositePodGroup API
• KEP-6089: WAS: Controller Integration APIs
• KEP-5547: WAS: Integrate Workload APIs with Job controller
• KEP-5729: DRA: ResourceClaim Support for Workloads

08 Sep 2026 6:30pm GMT

Kubernetes v1.37 promotes the KubeletInUserNamespace feature gate to beta. With this feature enabled, all of the node components (kubelet, CRI and OCI runtimes, CNI plugins, and kube-proxy) can run as a non-root user on the host, using a Linux user namespace. This technique is also known as rootless mode. The work started as an experiment in 2018, and was merged into Kubernetes v1.22 (2021) as an alpha feature (Kubernetes Enhancement Proposal KEP-2033).

This feature should not be confused with user namespaces for pods (hostUsers: false with the UserNamespacesSupport feature gate, GA since v1.36), which puts pods in user namespaces but still runs the node components as root. These two features do not conflict. Moreover, they can be combined to nest Kubernetes inside Kubernetes without resorting to the full privileged: true.

Why run the node components in a user namespace?

Because the node components have historically had container-breakout vulnerabilities that could compromise full root privileges on the host.

Examples of such vulnerabilities include:

• CVE-2022-0811 ("cr8escape"): CRI-O could be tricked into setting arbitrary sysctls, such as kernel.core_pattern, resulting in arbitrary code execution as root on the host
• CVE-2023-27561: runc could be tricked into bypassing the masked paths of a container via a volume mount race, exposing the host's procfs files (a regression of CVE-2019-19921)
• CVE-2024-10220: the kubelet could be made to execute arbitrary commands as root via gitRepo volumes (gitRepo volumes had a similar vulnerability, CVE-2018-11235, back in 2018 too)
• CVE-2025-31133: runc could be tricked into bind-mounting attacker-controlled paths and writing to the host's procfs files, such as /proc/sysrq-trigger and /proc/sys/kernel/core_pattern
• CVE-2026-53488: containerd could be tricked into executing arbitrary commands on the host, via crafted labels in a container image

By running the node components in a user namespace, the potential damage is confined to the non-root user's account. Notably, an attacker cannot conceal their intrusion by modifying the kernel, the boot loader, or the firmware.

It should still be noted that user namespaces are not effective for mitigating vulnerabilities in the kernel itself. User namespaces should be used in conjunction with traditional hardening measures such as seccomp to prevent containers from invoking unnecessary system calls.

Use cases

• Production clusters: mitigate potential container-breakout vulnerabilities.
• Shared machines (e.g., HPC): users can deploy Kubernetes without asking the machine administrator for root privileges, and without the risk of accidentally breaking other users' environments.
• Laptops: prevent a local cluster from accidentally breaking the host system configuration, e.g., the host iptables rules used for VPNs.
• AI sandbox: a Kubernetes application developer may create a dedicated local user account for running an AI coding agent and a test Kubernetes cluster. This setup is useful for preventing the AI agent from breaking the host when it is deceived by malicious information on the Internet.
• Kubernetes-in-Kubernetes: a nested cluster can run inside a parent cluster as a user-namespaced pod (hostUsers: false), isolating workloads more strictly than Kubernetes API namespaces do.
• Bootstrapping: a temporary unprivileged cluster can be used to bootstrap an actual cluster, e.g., with Cluster API.

How does it work?

A Linux kernel user namespace maps a host level non-root user (e.g., UID 1000) to a fake root user inside the namespace. The UID 0 privileges are limited to the inside of the namespace. The fake root is enough for most of the node components' tasks: mounting volumes, creating cgroups, and configuring the network namespaces of pods. It still comes with some caveats that may break compatibility with specific CNI and CSI drivers, though.

The user namespace has to be created outside of Kubernetes. For example, Rootless Docker can be used to prepare the user namespace in which Kubernetes runs.

The KubeletInUserNamespace feature gate itself is quite "boring": basically it just lets the kubelet ignore permission errors that occur when setting some sysctl values (e.g., vm.overcommit_memory and kernel.panic) and when watching kernel messages via /dev/kmsg.

See Running Kubernetes Node Components as a Non-root User for further information.

What changed from Alpha to Beta?

• The KubeletInUserNamespace feature gate is now enabled by default. Enabling the gate does not put the kubelet into a user namespace automatically, so nothing changes for existing "rootful" clusters.
• kubectl get nodes -o yaml now reports whether nodes are running in a user namespace via the runningInUserNamespace property. A cluster administrator can use this property to set node labels or taints, to avoid scheduling workloads that need real root privileges (e.g., some CNI plugin installers) onto rootless nodes.
• For Kubernetes' own CI/CD testing, the node conformance end to end tests now run on a rootless cluster (ci-kubernetes-e2e-kind-rootless).

Several related improvements have also happened outside the promotion of the feature gate itself:

• Linux kernel v6.3 (2023): added support for idmapped tmpfs.
• Kubernetes v1.33 (2025): enabled the UserNamespacesSupport feature gate by default, allowing user-namespaced pods (hostUsers: false) to be created without extra configuration.
• containerd v2.1 (2025): added support for writable cgroups.

With these improvements, a Kubernetes cluster with KubeletInUserNamespace can now also be nested inside Kubernetes pods with hostUsers: false (UserNamespacesSupport).

How to use it

kind

The easiest way is to use kind (a Kubernetes SIG Testing project) to run a Kubernetes cluster in rootless Docker, rootless nerdctl, or rootless Podman:

# Example using Docker
dockerd-rootless-setuptool.sh install
kind create cluster

Depending on the host configuration, you may need additional configuration for systemd, kernel modules, sysctl, etc.

See the Docker documentation and the kind documentation for further information.

minikube

minikube (a Kubernetes SIG Cluster Lifecycle project) also supports running a Kubernetes cluster in rootless Docker or rootless Podman:

dockerd-rootless-setuptool.sh install
minikube start --driver=docker

See the minikube documentation for further information.

Usernetes

Usernetes (a third-party project) is a distribution of rootless Kubernetes, maintained by the author of this article. The project began in 2018, and it is where the KubeletInUserNamespace feature gate originally came from.

Unlike kind and minikube, Usernetes supports creating a cluster with multiple rootless Docker / Podman / nerdctl nodes, connected using VXLAN via the Flannel CNI plugin.

Usernetes also experimentally supports a Kubernetes-in-Kubernetes mode.

k3s

k3s (a CNCF Sandbox project) also supports rootless mode. Unlike kind, minikube, and the current generation of Usernetes, rootless k3s does not rely on an external runtime such as rootless Docker.

What's next?

Depending on feedback and adoption, the Kubernetes project plans to graduate this feature to General Availability (GA) in a future release. If you have feedback on this feature, please open an issue in the kubernetes/kubernetes repository.

The project is also discussing several Kubernetes Enhancement Proposals that may contribute to simplifying Kubernetes-in-Kubernetes with this feature:

• KEP-5474: Enable Writable cgroups for unprivileged containers
• KEP-5714: Allow specifying whether to unshare cgroup namespaces

Getting involved

We always welcome new contributors. If you would like to get involved, you can join the Node Special Interest Group (SIG Node).

If you would like to share feedback, you can do so on our public Slack channel (visit https://slack.k8s.io/ for an invitation if you need one).

Special thanks to everyone who helped design and implement this feature, including but not limited to (in alphabetical order):

• Bing Hongtao (HirazawaUi)
• Jordan Liggitt (liggitt)
• Sergey Kanzhelev (SergeyKanzhelev)
• Tim Hockin (thockin)

04 Sep 2026 6:30pm GMT

Kubernetes 1.37 is here and Dynamic Resource Allocation (DRA) keeps pushing past where it started! This release brings DRA Extended Resource support to GA, a milestone the team has been building toward for three straight releases. Several more features graduate to Beta or GA. A fresh batch of alpha features rounds out the release.

I'll dive into what's new for DRA in Kubernetes 1.37!

What's stable in 1.37

DRA Extended Resource support has graduated to GA. This is the mechanism that lets DRA drivers satisfy requests made through the traditional extended resource API, think example.com/gpu in a Pod spec, without requiring a separate device plugin alongside the DRA driver. An extended resource name can be set directly on a DeviceClass, and Pods requesting it get matched to a device through DRA with no ResourceClaim needed on the workload's part.

It's been on a steady path since KEP acceptance in 1.34. Alpha landed in 1.35, Beta in 1.36, and now it's Stable. For cluster operators, this is what makes DRA adoption gradual. Existing workloads written against extended resources keep working unmodified while the backend allocation logic moves over to DRA.

ResourceClaims status with possible standardized network interface data adds a devices field to ResourceClaim .status, letting DRA drivers report per-device status, including, for network devices, the interface name, MAC address, and IP addresses. This gives users and controllers visibility into device state that was previously invisible once a device was configured in a Pod, and makes it possible to build things like network services that rely on a device's reported IPs.

DRA: device taints and tolerations is now Stable; DRA drivers can mark devices as tainted so they're skipped for new Pod scheduling, and cluster admins can apply the same taints cluster-wide via a DeviceTaintRule, without reconfiguring drivers. Pods already using a tainted device can be evicted automatically, unless their ResourceClaim explicitly tolerates the taint. This mirrors node taints and tolerations, letting operators take a single device offline for maintenance or mark it degraded, without disrupting the rest of the cluster.

Standard numaNode device attribute standardizes resource.kubernetes.io/numaNode as a shared attribute name, so devices from different drivers can be compared on the same NUMA node instead of each driver inventing its own name for it. It landed directly as stable in 1.37, since it's a naming/registration KEP with no feature gate or in-tree behavior change.

Feature promoted to Beta

ResourceClaim support for workloads graduates to Beta behind the DRAWorkloadResourceClaims feature gate, which stays disabled by default. In a cluster that has the feature enabled, Workloads and PodGroups can reference ResourceClaims directly, so a single claim can be shared across an entire group of Pods. This is instead of claims being capped at 256 Pods through the old per-Pod reservation limit.

The DRA Device Attributes Downward API is aimed at supporting device injection into KubeVirt VMs. Drivers populate a Metadata field when preparing a claim, and the framework writes it to a JSON file mounted into the container via CDI, letting workloads read a device's PCI bus address, MAC address, and other attributes directly instead of requiring custom controllers to watch and translate ResourceClaims and ResourceSlices.

Alpha features

List types for attributes moved into a second Alpha in 1.37, letting a device attribute hold more than one value instead of a single scalar, such as a CPU that's adjacent to more than one PCIe root. This makes it possible to match or distinguish devices based on overlapping or non-overlapping sets of values, while single-value attributes keep working as they do today.

Node allocatable resource requests moved into Alpha 2. It lets the scheduler and kubelet treat DRA-managed CPU, memory, and similar node resources the same way they treat ordinary resource requests, so a node doesn't get oversubscribed and users no longer have to duplicate the same request in both a ResourceClaim and the pod spec.

Resource availability visibility moved to a second Alpha in Kubernetes 1.37. Users create a ResourcePoolStatusRequest to get a point-in-time availability snapshot. To refresh it, delete and recreate the request; it is not a continuous monitoring API.

DRA: Optional Node Operations lets a driver skip kubelet's prepare and unprepare calls for allocations that don't need any setup on the node. This makes it possible to avoid an unnecessary dependency on the driver for allocations where there's genuinely nothing for it to do locally.

Derived Attributes is a new feature that lets you use CEL expressions to match up devices based on your own custom rules. Before this, pairing devices from different vendors (like a GPU/TPU and a NIC on the same NUMA node) only worked if both drivers used the exact same attribute name. If one used numa and the other used numaNode, the scheduler couldn't pair them together. Now, you can easily bridge these differences yourself inside your manifest, meaning you don't have to wait for hardware vendors to agree on standardized attribute names. Beyond just fixing naming differences, you can also use CEL to handle more complex scenarios like slicing a specific ID out of a long, monolithic topology string, or grouping devices into custom performance tiers based on their available capacity.

DRA Device Compatibility Groups lets drivers tag partitions of a device, like MIG vs vGPU profiles on the same GPU, with compatibility groups, so the scheduler rejects incompatible combinations up front instead of the driver failing at node preparation time. It's controlled by the DRADeviceCompatibilityGroups feature gate, disabled by default.

PreQueueingHint extension point is new as Alpha in 1.37. DRA ResourceClaim events used to trigger a full scan of every unschedulable pod, an O(N²) cost during large scale-ups. The DRA plugin now uses a pod informer index to narrow that to just the pods actually affected, cutting the requeue path to O(1) and roughly doubling scheduling throughput in early benchmarks. Controlled by the SchedulerPreQueueingHints feature gate.

DRA Consumable Capacity now supports fractional values in CapacityRequestPolicyRange, enabling more precise capacity requests and allocation for devices with fractional resources. This improves flexibility for workloads that require fine-grained resource allocation. The enhancement is gated by the DRAFractionalCapacityRange feature gate, which is in Beta in 1.37.

What's next

DRA continues to mature with every release. Several features currently in Alpha and Beta are on track to progress in the coming releases, and the community keeps working on DRA's performance, scalability, and reliability. Expect another ambitious set of DRA features in Kubernetes 1.38.

Getting involved

A good starting point is joining the WG Device Management Slack channel and meetings which happens at US/EU and EU/APAC friendly time slots.

Not all enhancement ideas are tracked as issues yet, so come talk to us if you want to help or have some ideas yourself! We have work to do at all levels, from difficult core changes to usability enhancements in kubectl which could be picked up by newcomers.

Acknowledgments

The following KEP owners added or promoted a feature in the 1.37 release (in alphabetic order):

• Alay Patel (alaypatel07)
• Byonggon Chun(bg-chun)
• Gaurav Ghildiyal (gauravkghildiyal)
• Jiefeng Xu (jiefeng-xu)
• John A. Hull (johnahull)
• Jon Huhn (nojnhuh)
• Lionel Jouin (LionelJouin)
• Patrick Ohly (pohly)
• Praveen Krishna (pravk03)
• Shingo Omura (everpeace)
• Troy Chiu (troychiu)

This would not have been possible without the help of the reviewers and approvers. So a huge thanks to everyone else who helped shape this release, in ways big and small. Given enough eyeballs, all bugs are shallow and this release had plenty of them, watching closely and caring enough to make things better. DRA got better this cycle because of all of you.

03 Sep 2026 6:30pm GMT

Kubernetes v1.37 includes API support for horizontal autoscaling of workloads down to zero replicas. This feature is now Beta and enabled by default. A HorizontalPodAutoscaler (HPA) that uses a suitable object metric or external metric can now scale a workload to zero replicas, then bring it back when the metric changes.

Before v1.37, you needed an add-on or external component, or you had to enable the Alpha feature gate, to scale from zero. It is now part of core Kubernetes.

Scaling to zero removes the last idle Pod from workloads such as queue consumers and batch processors. The savings are largest when each Pod reserves expensive resources, including dedicated CPUs or GPUs.

The trade-off is cold-start time: the HPA must observe the metric, schedule a Pod, and start the application. This works well when work can wait in a durable queue.

Kubernetes Services do not buffer requests while no Pods are ready, so HTTP and other request-driven workloads need a separate buffering layer.

Why scaling from zero needs a different metric

The HPA commonly scales on CPU or memory usage. Both metrics come from running Pods. Once the replica count reaches zero, there are no Pods left to measure and no signal that can tell the HPA to scale back up.

Object and external metrics do not have that limitation. A queue length, for example, exists independently of the workers that consume it. The HPA can continue reading the queue length while no workers are running.

The following example scales a queue consumer to and from zero using an external metric.

Configure an external metric

The following example uses a Prometheus metric named queue_consumer_lag. It assumes that Prometheus already collects a series similar to this one:

queue_consumer_lag{namespace="default",name="worker_tasks"}

Kubernetes needs a metrics adapter to make that value available through the External Metrics API. One implementation is the Prometheus Adapter, which can expose the series using an externalRules entry:

externalRules:
- seriesQuery: '{__name__="queue_consumer_lag",name!=""}'
 metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (name)
 resources:
 overrides:
 namespace:
 resource: namespace

The exact adapter installation and discovery rules depend on your monitoring setup. See the Prometheus Adapter guide to external metrics for the full configuration options.

Before creating the HPA, you can verify that Kubernetes can read the metric:

kubectl get --raw \
 '/apis/external.metrics.k8s.io/v1beta1/namespaces/default/queue_consumer_lag?labelSelector=name%3Dworker_tasks'

The request should return the current value for worker_tasks. If it does not, fix the metrics pipeline before configuring the HPA. An HPA cannot scale from zero when its metric is unavailable.

Configure the HPA

The following HPA targets a Deployment named queue-worker. It allows between zero and ten replicas, with one replica requested for each 30 queued tasks:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
 name: queue-worker
 annotations:
 kubernetes.io/description: "Scales queue-worker based on the number of queued tasks"
spec:
 scaleTargetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: queue-worker
 minReplicas: 0
 maxReplicas: 10
 metrics:
 - type: External
 external:
 metric:
 name: queue_consumer_lag
 selector:
 matchLabels:
 name: worker_tasks
 target:
 type: Value
 value: "30"

When the queue is empty, the HPA can reduce the Deployment to zero replicas. When tasks arrive, the external metric remains available and the HPA calculates a new replica count, capped at ten by maxReplicas.

Start the Deployment with at least one replica. Manually setting a Deployment to zero has always paused autoscaling. The HPA preserves that behavior and will not wake a workload that it did not scale down itself.

Normal HPA behavior still applies. In particular, the default downscale stabilization window is five minutes. The window prevents a short drop in queue length from immediately removing all workers. You can configure the window through spec.behavior.scaleDown if your workload needs different behavior.

How the HPA distinguishes zero from paused

Scaling from zero creates an ambiguity. A replica count of zero can mean that the HPA scaled the workload down, or that an operator manually paused it.

The controller resolves this with a ScaledToZero status condition. When the HPA scales a workload from one or more replicas to zero, it records ScaledToZero=True. The condition tells later reconciliation loops that the controller owns the zero state and should continue evaluating object or external metrics.

After scaling the workload back up, the controller changes the condition to ScaledToZero=False with the reason NotScaledToZero. A workload at zero without the ScaledToZero=True condition remains paused.

You can inspect the conditions with:

kubectl describe hpa queue-worker

If the adapter cannot return the configured metric, the HPA reports ScalingActive=False with a reason such as FailedGetExternalMetric. Restore the metric or manually scale the workload to recover capacity.

Before upgrading or rolling back

In Kubernetes v1.37, the HPAScaleToZero feature gate is enabled by default on both the kube-apiserver and kube-controller-manager. The API server accepts minReplicas: 0; the controller manager performs the condition-based scaling.

During a version-skewed control plane upgrade, wait until both components support the feature and have it enabled before creating HPAs with minReplicas: 0. A controller manager with the feature disabled treats replicas: 0 as a manual pause and may leave a workload at zero.

Before disabling the feature gate or downgrading to a version without the condition-based implementation:

• Change affected HPAs to minReplicas: 1 or higher.
• Scale any workload currently at zero to at least one replica.

minReplicas: 0 also requires at least one object or external metric. The API server rejects an HPA that only contains resource metrics such as CPU or memory.

From Alpha to Beta

The first Alpha implementation shipped in Kubernetes v1.16. Kubernetes v1.36 added the ScaledToZero condition and the controller behavior needed to distinguish an automatic scale-down from a manual pause.

Kubernetes v1.37 enables the feature by default after adding integration and end-to-end coverage for scaling down to zero and back up from an external metric. The next step is to gather operational feedback before considering graduation to GA.

How can I learn more?

• Read the documentation for scaling to and from zero.
• Read KEP-2021: HPA supports scaling to and from zero pods for object and external metrics.
• Learn how to configure the Prometheus Adapter for external metrics.

How to get involved

This feature is owned by SIG Autoscaling. Join Kubernetes Slack and the #sig-autoscaling channel to share feedback from Beta usage.

Acknowledgements

Thanks to the SIG Autoscaling contributors who took this feature from the original v1.16 implementation to the condition-based redesign and Beta graduation. Thanks also to Guy Templeton and Adrian Moisey for reviewing the KEP, and to the release, documentation, and production-readiness reviewers who helped prepare it for Kubernetes v1.37.

02 Sep 2026 6:30pm GMT

I am excited to announce that etcd RangeStream is graduating to beta in Kubernetes v1.37. Paired with etcd v3.7, it reduces the memory the API server and etcd need to read a large collection, and makes peak usage more predictable.

The cost of large reads

The API server serves most list and watch requests from its in-memory watch cache. Populating that cache requires reading a resource's full state from etcd, at startup and on every re-initialization. For a resource with many objects, or large ones, such as Pods, that read is expensive.

The API server already paginated these reads, asking etcd for a fixed number of keys at a time rather than the whole collection at once. But a page bounded by key count has no awareness of object size, so a page of large objects can still be very large. That makes memory usage hard to predict, and a bad combination of object size and concurrent reads can be enough to trigger an OOM. etcd's unary Range assembles each page in full before sending it, and the API server holds it while decoding, so the same payload sits in memory on both sides at once. Most of that cost lands on etcd, which is also where streaming helps most.

Streaming reads with RangeStream

etcd v3.7 adds a streaming version of that read, the RangeStream RPC. It takes the same RangeRequest as Range and returns the same result set, but instead of building the whole response up front, etcd splits it into chunks and streams them. Chunk size is tuned adaptively to the values being returned, so a collection of large objects is bounded by bytes rather than by a key count, and memory is freed as the stream progresses instead of being held until a whole page is assembled.

When the feature is enabled, the API server uses RangeStream wherever it reads a whole collection out of etcd. This includes watch cache initialization, and the fallback paths where a list request cannot be served from the cache and reads etcd directly. In either case the API server decodes each chunk as it arrives and releases it before pulling the next one, so neither side ever holds the whole collection.

Requirements

• Kubernetes v1.37 or later
• etcd v3.7 or later

RangeStream is used when the EtcdRangeStream feature gate is enabled on the kube-apiserver, which is beta and on by default in v1.37, and etcd is v3.7 or later. The API server resolves etcd's support at startup and also falls back at runtime if a call returns Unimplemented, so an API server paired with an older etcd keeps using the paginated Range path on its own. To turn it off, disable the gate:

--feature-gates=EtcdRangeStream=false

Confirming RangeStream is in use

The API server records streamed reads under their own operation label on its etcd metrics. A non-zero count here means RangeStream is in use:

etcd_request_duration_seconds_count{operation="listStream"}

If it stays at zero, the API server is still using the paginated Range path, most likely because etcd is older than v3.7.

Learn more

• KEP-5966: etcd RangeStream
• RangeStream in the etcd API guide
• Announcing etcd v3.7
• SIG etcd

If you have questions or feedback, join the #sig-etcd channel on Kubernetes Slack.

01 Sep 2026 6:30pm GMT

I am excited that storage version migration (SVM) has graduated to General Availability (GA) in Kubernetes v1.37!

After a number of releases of work and testing, the built-in StorageVersionMigration API (storagemigration.k8s.io/v1) and control plane controller are now fully stable and enabled by default across all v1.37 Kubernetes clusters.

The problem with stale storage versions

In Kubernetes, stored API resources are written using a specific storage version (schema representation). The way Kubernetes interacts with object storage fundamentally requires mutation of a resource in order to ensure that the latest storage version is used for all resources. This creates problems when you want to change the storage version of a resource.

One example of a scenario where you may want to change the storage version of a resource is when you are promoting a CRD to drop an older API version (such as v1alpha1) to a newer version (leaving just v1beta1 and v1). It's a problem to drop the older API version whilst there are still resources stored with the old alpha version.

To avoid problems, you designate v1 as the new storage version; but, on it's own, that's not enough. While new writes are stored as v1, any existing resource could remain stored as v1alpha1 or v1beta1 in storage. You cannot safely remove v1alpha1 from the CRD's .status.storedVersions or drop serving support until every single resource in storage has been re-written to not be serialized and stored with the alpha version.

Another relevant example is encryption at rest and, related, key rotation. When you configure encryption at rest or rotate encryption keys, existing resources in storage remain unencrypted (or encrypted under old keys) until they are actively re-written through the Kubernetes API server.

Historically, cluster administrators and CRD authors had to rely on manual kubectl get / kubectl replace scripts, or to deploy the out-of-tree kube-storage-version-migrator component to force re-writes. These approaches were often tedious, error-prone, and difficult to monitor.

How storage version migration works

Initiating a storage version migration is as simple as creating a declarative StorageVersionMigration object. The built-in StorageVersionMigrator controller in the Kubernetes control plane watches for these objects, and automatically migrates existing resources to the default storage version for that API.

Example: Migrating a custom resource API

Suppose you have updated a CustomResourceDefinition (crontabs.example.com) to use v1 as its storage version. To migrate all existing stored resources off older versions, create a StorageVersionMigration:

apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
 name: crontabs-migration
spec:
 resource:
 group: example.com
 resource: crontabs

Apply the manifest using kubectl:

kubectl apply -f crontabs-migration.yaml

Monitoring and verifying migrations

The StorageVersionMigrator controller updates the status of the StorageVersionMigration object as migration progresses. You can inspect the migration status using kubectl:

kubectl get storageversionmigration.storagemigration.k8s.io/crontabs-migration -o yaml

A successful migration will report a Succeeded condition set to True:

status:
 conditions:
 - type: Running
 status: "False"
 lastUpdateTime: "2026-08-02T10:05:00Z"
 reason: StorageVersionMigrationInProgress
 - type: Succeeded
 status: "True"
 lastUpdateTime: "2026-08-02T10:05:00Z"
 reason: StorageVersionMigrationSucceeded

Once the migration has succeeded, you can be confident that all instances of the resource in storage are stored in the current storage version. For CRDs, the stored version should be updated in the CRD's .status.storedVersions to only contain the preferred version. If the .status.storedVersions is not updated following a successful migration then that means that the CRD was updated during the migration. In that case, the migration should be retried in order to safely deprecate an older storage version.

Including migrations in your CRD manifests

Because StorageVersionMigration is a standard declarative Kubernetes API, CRD authors can bundle or trigger migrations directly alongside CRD upgrades. For example, you can include the migration in the same manifest as your updated CustomResourceDefinition:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
 name: crontabs.example.com
spec:
 group: example.com
 # Updated versions list where v1 has storage: true
 ...
---
apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
 name: crontabs-migration
spec:
 resource:
 group: example.com
 resource: crontabs

What's next?

• Learn more about the concepts behind Storage Versions.
• Read the step-by-step task guide: Migrate Kubernetes Objects Using Storage Version Migration.

SIG API Machinery would love to hear your feedback as you adopt built-in Storage Version Migration in your clusters. Reach out to us on the #sig-api-machinery Slack channel or participate in our community discussions!

31 Aug 2026 6:30pm GMT

Kubernetes brings a wealth of features that make it easy to run your production workloads securely and reliably. While aspects like scheduling, health checks and resource limits are probably at the front of your mind, one other important feature of Kubernetes is production identity - how your workload can authenticate to other systems in order to do its job.

Up until now, the primary production identity mechanism built into Kubernetes has been service account JWTs (JSON Web Tokens). These are cryptographically-signed tokens, issued by the control plane of your cluster, that let anyone in the world understand who is calling when your workload uses them.

In Kubernetes 1.37, the foundations of a new built-in production identity technology have gone GA. Pod Certificates (and the closely-associated Cluster Trust Bundles) build X.509 certificate issuance for TLS and mTLS directly into core Kubernetes.

Why?

Service account JWTs have a lot going for them:

• They are built directly into Kubelet, and work pretty magically. They are written to your workload container's filesystem before your workload starts up, and automatically kept up to date.
• The issuance system follows least-privilege principles; the node restriction admission plugin ensures that tokens can only be requested by the Kubelet that is actually currently running your pod.
• They can be federated, allowing you to use them to authenticate to other systems outside of Kubernetes. Service account JWTs underpin the pod-to-cloud authentication store for all of the largest cloud providers, and have widespread support across many additional services and software packages. If it can understand JWTs, you can authenticate to it with a service account token.

However, service account JWTs have one big downside - they are bearer tokens. With bearer tokens, if you have the token, then you are the identity asserted by the token. And since you necessarily have to hand copies of the JWT to all your peers in order to authenticate to them, they can be you, too.

There are partial mitigations for this, and service account tokens make use of them (time-, object-, and audience-binding), but none are complete defences.

A solution to this problem lies in proof-of-possession credentials, where you don't send your entire credential to your peer, but only a proof that you possess the credential. In practice, these schemes are always built on asymmetric cryptographic signatures (RSA, ECDSA, and friends).

There are few different standard approaches, such as request signing (AWS SigV4, JWT DPoP, RFC 9421), but the most widely-deployed and understood solution is X.509 certificates, as used in TLS. In TLS, your credential is split into two pieces

• A private key, which for maximum security should be generated within your workload (or within a hardware security module), and never leave.
• A certificate, which is a description of your identity and public key, signed by a Certificate Authority.

The goal of Pod Certificates is to make using X.509 certificates from your Kubernetes workload just as easy as using service account JWTs, while maintaining Kubernetes' high security bar. I think we've hit this target.

As I'll cover in the architecture and example sections below, there are many similarities between the design of service account JWT issuance and Pod Certificates. One significant place they diverge, however, is that Pod Certificates is a much more flexible mechanism. Kubernetes only offers one flavor of service account JWTs, with standardized claims.

The X.509 ecosystem is significantly more varied than the JWT ecosystem, and X.509 certificates used for different purposes contain different extensions and information. For this reason, Pod Certificates has common machinery built into Kubelet, but offers a pluggable interface so that many different types of certificates can be issued within a single cluster, at the same time.

In the fullness of time, I expect Kubernetes to offer at least two built-in certificate providers:

• One that issues server TLS certificates for the DNS names used by Kubernetes services.
• One that offers SPIFFE client certificates, filling the same role that service account JWTs fill today.

In the remainder of this article, I'll take you through the overall architecture of a Kubernetes workload using Pod Certificates, as well as give you an example of installing and using a real (toy) Pod Certificates signer controller.

Architecture

When you use Pod Certificates and Cluster Trust Bundles, there are the following major components:

• Your application, which requests certificates in its pod spec, and reads the keys, certificates and trust bundles from the container filesystem to use for (m)TLS.
• Kubelet, which issues PodCertificateRequest objects and reads ClusterTrustBundle objects on behalf of your workload.
• The signer controller, which answers PodCertificateRequests and publishes ClusterTrustBundles.

The best way to get a sense of what these components each do is to follow the issuance process chronologically:

1. Once your application pod is scheduled to a node, Kubelet identifies all of the podCertificate and clusterTrustBundle projected volumes sources in its spec.
2. For each podCertificate source:
   1. Kubelet generates a new private key according to the keyType field.
   2. Kubelet creates a PodCertificateRequest addressed to the signer named in the source.
   3. The signer controller sees the PodCertificateRequest and decides whether or not to issue the certificate.
   4. The signer controller issues the certificate by filling out the status.certificateChain field.
   5. The signer controller also fills out the status.beginRefreshAt field to instruct Kubelet when it should begin trying to refresh the certificate.
      certificate to the container filesystem.
3. For each clusterTrustBundle source:
   6) Kubelet retrieves the issued certificate, and writes the private key and
   1. Kubelet collects all the ClusterTrustBundles that match the signer name
   2. Kubelet unifies all of the certificates from all matching ClusterTrustBundles, and (stably) reorders them (to prevent applications from accidentally depending on a particular ordering).
      and label selectors in the source.
   3. Kubelet writes the certificates to the file path named in the source.
      and trust anchors from the filesystem.
4. Your application pod starts up, and the application reads keys, certificates,
5. Kubelet periodically updates the files from clusterTrustBundle sources as the contents of the selected ClusterTrustBundles changes. The application must pick up the changes using inotify or polling.
6. As each certificate's beginRefreshAt time passes, Kubelet repeats the process in step 2 to refresh the certificates, and write the update private keys and certificate chains to the filesystem. As in step 5, the application must pick up changes using inotify or polling.

Some key takeaways:

• Automatic rotation is built in. Applications must properly handle it. Any signers eventually shipped in core Kubernetes will issue certificates with a max lifetime of 24 hours. The maximum lifetime allowed for other signers is 91 days.
• To make automatic rotation support as simple as possible, Kubelet supports writing the private key and certificate chain to a single file (a credential bundle) This allows the application to simply subscribe to inotify events for (or poll) the single file, read the contents, and use them. Kubelet does support writing the private key and certificate chain to separate files, but then the application needs to carefully manage the potential race conditions of reading the files mid-rotation.
• Wherever possible, security checks are built into kube-apiserver, rather than burdening signer or application developers. As an example, the built-in node restriction admission plugin enforces node isolation, ensuring that one compromised node cannot spread access by requesting certificates for pods that aren't scheduled to it.

Try it out

Because the Kubernetes project does not yet ship any Pod Certificate signers in core, in order to try these features out, you will need to install a third-party signer into your cluster. To make this easier, I have written Tinycert, which you can install into your cluster (or a Kind cluster).

Tinycert is not a full production solution, but it's a good starting point for experimenting with Pod Certificates, as well as a base for creating your own signers.

Tinycert provides:

• The ahmedtd.github.io/tinycert-service signer, which issues certificates with DNS SANs for all of the Kubernetes Services your Pod is part of.
• The ahmedtd.github.io/tinycert-spiffe signer, which issues SPIFFE-compatible certificates that identify the namespace and service account of your Pod. These can be used as both client and (with effort) server certificates.
• A Go library, github.com/ahmedtd/tinycert/lib/spiffefsd to help your applications load SPIFFE certificates and trust bundles from a SPIFFE Filesystem Delivery (Draft Standard) folder, as well as configure the Go TLS library for proper client and server authentication.
• An example of a SPIFFE client and server application communicating using mutual TLS and SPIFFE certificates.

What next?

• Take a look at the documentation for Pod Certificates and Cluster Trust Bundles.
• Review and offer feedback on the SPIFFE Filesystem Delivery draft standard, which aims to make it as easy as possible to use SPIFFE certificates directly on native Kubernetes.
• Participate in Kubernetes SIG Auth to help shape the future of signers that are built directly in to core Kubernetes.
• Try building your own signer based on Tinycert.

Happy hacking!

28 Aug 2026 6:30pm GMT