22 Sep 2026

feedKubernetes Blog

Spotlight on SIG Apps

As Kubernetes adoption has grown, the conversation has shifted beyond running containers to managing increasingly complex application lifecycles. Modern platforms support stateless web services, stateful databases, batch processing, AI workloads, and platform services. At the same time, they must remain reliable during upgrades, scaling events, and infrastructure failures.

Every Kubernetes user relies on SIG Apps, whether they realize it or not. Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs form the foundation of how applications are deployed, updated, scaled, and operated across the Kubernetes ecosystem.

SIG Apps is focused on improving workload resilience, refining application lifecycle management, and addressing the operational challenges that emerge when applications encounter node failures, rollout disruptions, and increasingly complex infrastructure environments.

In this spotlight, we sit down with two of the three SIG Apps chairs Janet Kuo and Maciej Szulik to discuss the evolution of Kubernetes workload management, the challenges of balancing application reliability with operational simplicity, and the future of application lifecycle management within one of Kubernetes' most influential Special Interest Groups.

Introducing SIG Apps

Natalie Fisher: Can you introduce yourself, your role, and how you got involved in SIG Apps?

Janet Kuo: I'm a Senior Staff Software Engineer at Google and have been a Kubernetes maintainer since 2015, joining the community just as we were racing toward the 1.0 launch. In those early days, my focus was on building the core Workloads API, specifically developing controllers like Deployment, ReplicaSet, StatefulSet, and DaemonSet, defining their rollout behaviors, and bringing them from initial designs to GA. That hands-on work was my entry point into SIG Apps.

Since then, I've stayed deeply involved in both the technical and community sides of Kubernetes. I have led SIG Apps as Co-Chair and Tech Lead since 2019. Currently, in addition to maintaining the workloads API, I am driving new subprojects like the Agent Sandbox to ensure Kubernetes is ready for next-generation agentic and AI workloads.

Maciej Szulik: I started contributing to Kubernetes all the way back in 2014. Since then, I've worked across various areas of the project: controllers, kubectl, and apimachinery, which eventually led me to become one of the Chairs and Tech Leads for SIG Apps. My current focus is reliability of the workload controllers under the SIG Apps umbrella and stability and ease of use of kubectl as part of my SIG CLI Tech Lead role. I also care about overall community health and growth as part of my Steering Committee role. Outside of Kubernetes, I work as a Staff Platform Engineer at Defense Unicorns, where I'm helping make Kubernetes more airgap-native with a project called zarf.

The problem and the solution

SIG Apps is responsible for the core workload APIs that power how applications run on Kubernetes. From Deployments and StatefulSets to Jobs and CronJobs, these controllers determine how workloads are created, updated, scaled, and recovered when things go wrong.

As Kubernetes expands to support increasingly diverse workloads - including AI, batch processing, and large-scale distributed applications - SIG Apps continues to evolve these APIs while balancing reliability, backward compatibility, and operational simplicity.

NF: For readers who may not be familiar, what is SIG Apps, and what role does it play within the broader Kubernetes ecosystem?

MS: SIG Apps is the Kubernetes Special Interest Group responsible for the workloads APIs. CronJob and Job help running batch workloads, whereas DaemonSet, Deployment, ReplicaSet, and StatefulSet serve the majority of other applications. More broadly, SIG Apps owns the layer most developers actually touch day-to-day: the controllers that turn a workload specification into running, self-healing pods. It's the group deciding how Deployments roll out, how Jobs retry, how DaemonSets place a pod per node.

JK: Adding to what Maciej described, as the industry shifts, we are seeing a massive demand to run complex, non-traditional workloads like distributed AI training, batch computing, and dynamic agent environments. Our role is expanding: we aren't just maintaining the classic workloads API, but we are actively evolving it and establishing new patterns (like the Agent Sandbox) to make sure Kubernetes remains the best platform for the next generation of workloads, such as AI.

NF: Looking at the workload APIs owned by SIG Apps (Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs), which areas are receiving the most attention from maintainers and contributors?

MS: After a long stretch focused on making batch workloads run smoothly on Kubernetes, we've shifted attention to make sure serving workloads (DaemonSets, StatefulSets, etc) aren't left behind. This means performance and high-scale improvements to rollout and scaling behavior, plus working through our backlog of user-reported issues, prioritizing the ones with the strongest support from the user base.

Current focus areas

As Kubernetes workloads grow in scale and complexity, the challenges facing workload controllers evolve as well. We asked the SIG Apps chairs where contributors are focusing their efforts today and which resilience problems they believe are the highest priorities.

NF: From your perspective, what are the most important workload resilience problems SIG Apps is trying to solve today?

MS: Node lifecycle challenges have come up repeatedly across SIG Apps, SIG Node, and SIG Autoscaling discussions. DaemonSets and Jobs are just where the pain is most visible, since they're the workloads most directly bound to node state. Rather than solve it piecemeal within one SIG, we've settled on spinning up a dedicated Node Lifecycle Working Group to focus on this properly and hopefully land long-term solutions instead of one-off patches.

JK: From an AI perspective, resilience is critical. When you are running a massive distributed LLM training job that spans hundreds of GPUs, a single node failure can halt the entire pipeline. Similarly, if a DaemonSet that runs your logging or GPU monitoring agent gets stuck on a bad node, it impacts the entire cluster's health.

In addition to the work in the Node Lifecycle WG to handle infrastructure-level degradation, SIG Apps is addressing this at the orchestration layer through subprojects like JobSet (for distributed training) and LeaderWorkerSet (LWS) (for sharded LLM inference). These APIs introduce patterns like "all-or-nothing" failure handling, where a single pod or job failure triggers a coordinated group-level restart to resume from the last clean checkpoint, rather than letting stuck workloads hang in an inconsistent state.

Real-world impact

The work happening within SIG Apps extends far beyond controller implementations and API design. We wanted to understand what these improvements mean in practice for platform teams operating Kubernetes clusters in production.

NF: For platform teams operating Kubernetes in production, what practical improvements would they notice if the node lifecycle and workload resilience work currently under discussion is successfully delivered?

MS: I'm mostly looking from the sidelines, the folks actually in the Node Lifecycle Working Group would give you a sharper answer. But from where I sit, I'm hoping their work translates into fewer 3am pages that turn out to be "a DaemonSet rollout got stuck because node X was flaky, and someone had to manually cordon/delete/restart to unstick it."

JK: +1 to what Maciej said, and beyond reducing manual intervention, platform teams will also see much better resource predictability and cost efficiency. For example, in AI workloads where GPU idle time is extremely expensive, having Kubernetes automatically detect a degraded node and reschedule the training coordinator or agent before the job crashes means less wasted compute and more stable job execution.

Challenges and trade-offs

Evolving APIs that millions of workloads rely on requires careful engineering and even more careful decision-making. We asked the SIG Apps chairs about the technical and operational trade-offs they weigh when introducing changes to Kubernetes' core workload controllers.

NF: What are some of the hardest technical or operational trade-offs SIG Apps encounters when evolving core workload controllers?

MS: Honestly, a few tensions keep coming up: how aggressively a controller should give up on stuck pods, and what signals it actually needs to make that call correctly. At the same time, we always have to think about backward compatibility. Deployment, DaemonSet, and Job behavior has been depended on for a decade [by Kubernetes users, tooling, automation, and higher-level controllers], so even a change that's clearly "more correct" can break automation people built around the old behavior without meaning to.

JK: One of our hardest trade-offs is resisting the urge to make "elegant" design changes that break backward compatibility. Instead, we have to design opt-in features that let users adopt new behaviors without forcing them on legacy workloads. When we need to support completely new paradigms, we prefer introducing them as CRDs first rather than bloating the core APIs, like we are doing with Agent Sandbox, JobSet, and LWS.

Looking ahead

While much of SIG Apps' work focuses on maintaining the stability of existing workload APIs, the group is also shaping the future of Kubernetes through new enhancements and proposals. We concluded by asking about one proposal that recently returned to active development and what it represents for the future of workload management.

NF: The SIG recently discussed reviving KEP-4443 with a target release of Kubernetes 1.38. What opportunities or challenges does this proposal aim to address, and why is now the right time to revisit it?

KEP-4443 addresses a small but real gap in the Job API: a PodFailurePolicy can be configured to add a condition reason to the JobFailed condition, but different pod failure policy rules targeting different container exit codes all produce that same generic reason. The proposal is simple: an optional Name field on each PodFailurePolicyRule, which gets appended to the JobFailed condition reason, so higher-level tools like JobSet can finally react differently depending on which rule triggered the failure.

As for timing, the answer is as simple as it always is in open source: we lost the original contributor who was driving this. Now we've got someone new interested in picking it up, that's why we're targeting the next release.

Getting Involved

NF: For someone interested in contributing to SIG Apps, where would you recommend they start, especially if they are not yet a Kubernetes maintainer?

MS: The best place to start is the #sig-apps slack channel and our regular SIG Apps meetings. We've all started there, and if it feels intimidating, or nobody replies right away, that's completely normal. Everyone's busy. It's not personal.

JK: In addition to what Maciej answered, I'd suggest looking at our newer subprojects and initiatives. Contributing to stable APIs like Deployment or StatefulSet can be daunting because the barrier for making changes is very high due to backward compatibility, and there is much less low-hanging fruit.

If you are new to the community, projects like the Agent Sandbox are fantastic entry points. They are actively evolving, have a friendly group of maintainers, and offer plenty of greenfield development opportunities where you can make a significant impact quickly.

Summary

SIG Apps has shaped how Kubernetes applications are deployed and operated since the project's earliest days. While users often interact with Deployments, StatefulSets, Jobs, and DaemonSets without thinking about the controllers behind them, the work within SIG Apps continues to shape the reliability and scalability of workloads across the Kubernetes ecosystem.

From improving workload resilience and node lifecycle behavior to enabling new patterns for AI and distributed computing, the SIG is evolving Kubernetes while remaining committed to one of the project's core principles: preserving the stability and backward compatibility that users depend on. Whether you're interested in core workload APIs, emerging projects like Agent Sandbox, or helping improve the operational experience of Kubernetes users everywhere, SIG Apps offers many opportunities to get involved.

22 Sep 2026 6:00pm GMT

21 Sep 2026

feedKubernetes Blog

Kubernetes v1.37: Tracking When a PersistentVolumeClaim Was Last Used (Beta)

Kubernetes v1.37 promotes the PersistentVolumeClaimUnusedSinceTime feature gate to Beta (enabled by default). With this feature, the PersistentVolumeClaim (PVC) protection controller adds an Unused condition to each PVC, telling you whether any running pod currently references it - no custom tooling or cross-referencing required.

For the API definition of PVC conditions, see the PersistentVolumeClaim API reference. Read on to learn how the Unused condition works and how to use it.

Why track PVC usage?

In large-scale Kubernetes clusters, it is common for users to create PVCs and then delete the associated pods without cleaning up the storage, because Kubernetes does not automatically delete PVCs when their pods are removed (to protect against accidental data loss). Over time, these orphaned PVCs may accumulate, silently consuming storage capacity and driving up cloud costs.

Before Kubernetes v1.37, it was easy to identify an unused PersistentVolume, but much harder to determine whether a PVC was still being used. Doing so required cross-referencing pods, PersistentVolumes, and PVCs over a potentially large window of time. Administrators often resorted to custom monitoring pipelines or scripts to answer a seemingly simple question: "Is anything actually using this volume?"

The PersistentVolumeClaimUnusedSinceTime feature solves this by making the answer available natively in the PVC status. Once the feature is enabled, every PVC gets an Unused condition managed by the PVC protection controller.

User stories

How does it work?

The PVC protection controller - which already watches pods to enforce the storage object in use protection - now also manages a new Unused condition on PVCs.

The condition works as follows:

Scenario Condition status Reason
No non-terminal pods reference the PVC Unused=True NoPodsUsingPVC
At least one running or pending pod references the PVC Unused=False PodUsingPVC

A few details worth noting:

Using lastTransitionTime to find when a PVC became idle

Like every Kubernetes condition, the Unused condition carries a standard lastTransitionTime field. This means you get a useful bonus for free: when the condition transitions from False to True, the lastTransitionTime records exactly when the PVC became idle. You can use this timestamp to answer questions like "how long has this PVC been sitting unused?" - for example, to find PVCs that have been idle for more than 30 days (see the example query below).

What changed from Alpha to Beta?

Kubernetes v1.36 introduced this feature as Alpha, where you had to enable the PersistentVolumeClaimUnusedSinceTime feature gate explicitly. For Beta in v1.37, the feature gate is enabled by default, and the feature has full end-to-end test coverage.

How to use it

Since the feature is Beta and enabled by default in Kubernetes v1.37, the Unused condition will appear on PVCs automatically. Here is a walkthrough to see it in action:

  1. Create a PVC:

    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: my-data
spec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 1Gi

  2. After a short time, inspect the PVC conditions:

    kubectl get pvc my-data -o jsonpath='{.status.conditions[*]}' | jq .

    You should see an Unused condition with status True and reason NoPodsUsingPVC:

    {
 "lastProbeTime": null,
 "lastTransitionTime": "2026-09-14T12:03:11Z",
 "message": "No pods are currently referencing this PVC",
 "reason": "NoPodsUsingPVC",
 "status": "True",
 "type": "Unused"
}

  3. Create a pod that uses the PVC:

    apiVersion: v1
kind: Pod
metadata:
 name: my-app
spec:
 containers:
 - name: app
 image: busybox
 command: ["sleep", "3600"]
 volumeMounts:
 - name: data
 mountPath: /data
 volumes:
 - name: data
 persistentVolumeClaim:
 claimName: my-data

  4. Check the condition again - it should now show Unused=False:

    kubectl get pvc my-data -o jsonpath='{.status.conditions[?(@.type=="Unused")].status}'

    Output:

    False

  5. Delete the pod and wait for the condition to transition back to Unused=True:

    kubectl delete pod my-app
kubectl get pvc my-data -o jsonpath='{.status.conditions[?(@.type=="Unused")]}'

    The condition should show Unused=True with reason NoPodsUsingPVC again.

Finding unused PVCs across the cluster

To list all PVCs that have been unused for more than 30 days, you can use a command like:

Note:

This command uses jq, a command-line JSON processor.
kubectl get pvc -A -o json | jq -r '
 .items[]
 | select(.status.conditions[]? | select(.type=="Unused" and .status=="True"))
 | select(
 (.status.conditions[] | select(.type=="Unused") | .lastTransitionTime) as $t
 | (now - ($t | fromdateiso8601)) > (30 * 86400)
 )
 | "\(.metadata.namespace)/\(.metadata.name) unused since \(.status.conditions[] | select(.type=="Unused") | .lastTransitionTime)"
'

What's next?

Depending on feedback and adoption, the Kubernetes project intends to graduate this feature to General Availability (GA) in a future release. If you have feedback on this feature, please open an issue in the kubernetes/kubernetes repository.

To learn more about this enhancement, refer to KEP-5541: PersistentVolumeClaim last used time.

Getting involved

The Kubernetes project always welcomes new contributors. If you would like to get involved, you can join us at SIG Storage.

If you would like to share feedback, you can do so on our public Slack channel (visit https://slack.k8s.io/ for an invitation if you need one).

Special thanks to the contributors who helped design and implement this feature (alphabetical order):

21 Sep 2026 6:30pm GMT

16 Sep 2026

feedKubernetes Blog

Kubernetes v1.37: Hardening Container Storage with Bind Mount Options and EmptyDir Permissions

Kubernetes v1.37 brings important storage security features: emptyDir permission modes and bind mount options. They help application programmers and security professionals implement rigorous security policies, for example, prohibiting deletion of files across containers or execution of arbitrary binaries from writable volumes, directly in Kubernetes without any complicated circumvention.

Linux storage and permission fundamentals

Before diving into the new Kubernetes features, let us briefly review the low-level Linux security mechanisms that make them possible.

Bind mount flags

When Linux mounts or remounts a directory, Virtual File System (VFS) flags control what actions are permitted on that filesystem:

Directory permissions and the sticky bit

Standard Unix permissions regulate access across three scopes: Owner, Group, and Others (e.g., 0755 or 0777).

Beyond standard read, write, and execute bits, Linux supports the sticky bit (as in mode 01777). When applied to a directory, the sticky bit ensures that a file inside that directory can only be deleted or renamed by the file's owner or root. This is essential for shared writable directories like /tmp.

Motivation for the improvements

Why does Kubernetes need bind mount options and emptyDir permissions?

The primary goal of these features is to increase the security of Kubernetes workloads by allowing security-related bind mount options on volume mounts. By default, volumes are bind-mounted into containers by the container runtime and kubelet without noexec, nosuid, or nodev flags. This default can undermine security. For example, with noexec missing, a compromised process can use any writable volume (emptyDir, PersistentVolume, etc.) to download, chmod +x, and execute arbitrary binaries even when the container has a read-only root filesystem (readOnlyRootFilesystem: true). Supporting noexec, nodev, and nosuid gives users a native way to harden volume mounts to match security benchmarks and policy.

The gap is most visible with emptyDir volumes, which are the most common writable volume type and have been the subject of multiple security findings:

However, the same gap applies to all volume types. PersistentVolumes have a mountOptions field, but those options are filesystem-level flags applied by the CSI driver at the node, so they do not reliably translate into bind mount flags inside the container. Previously, there was no mechanism to set noexec, nosuid, or nodev on the bind mount that the container runtime creates for any volume type.

Additionally, the emptyDir volume type defaults to creating directories with a hardcoded mode of 0777. This previously meant that any process that can discover the volume could read, write, and delete anything in the volume, regardless of who created it.

You could - and still can - use an initial container to set a different access mode, but this is more complex, and hard to verify for compliance.

This causes real problems:

The emptyDir volume type was a notable gap. As one of the most common writable volume types in Kubernetes, it had no way to control its creation permissions.

Real-world use cases

Application developers, working closely with security engineers, are responsible for maintaining the security posture of their applications and ensuring workloads do not pose risks to the wider infrastructure. These features allow development teams to confidently address critical security scenarios:

Preventing Privilege Escalation on Writable Mounts: An application developer configuring temporary workspace volumes (like emptyDir or /tmp mounts) can ensure they are mounted with nosuid and noexec. This guarantees that even if the application is compromised and a malicious payload is downloaded, the workload cannot execute the payload or use it to escalate privileges on the node.

Securing Shared Scratch Space in Multi-Container Pods: A developer configuring CI/CD pipeline pods often needs multiple containers (e.g., a builder container and a sidecar logger) to share a workspace. By setting mode: 01777 on an emptyDir, the developer ensures the shared workspace behaves like a traditional Unix /tmp directory. Each container can write files independently, but a compromised process in one container cannot delete the build artifacts produced by another.

Enforcing Principle of Least Privilege for Application Data: An application developer deploying a database pod can lock down access to the database's temporary storage. By setting mode: 0750 on the emptyDir, the developer ensures that only the specific database user and group can read or write to the volume, explicitly denying access to any other processes or sidecars in the same pod.

Note: Both features are behind Alpha feature gates in Kubernetes v1.37. To use them, enable VolumeBindMountOptions and EmptyDirVolumeMode on the API server and kubelet.

Example 1: Enforcing bind mount options

This full Pod manifest mounts an emptyDir volume at /tmp with bindMountOptions: [noexec, nosuid].

apiVersion: v1
kind: Pod
metadata:
 name: hardened-bindmount-pod
 namespace: default
spec:
 os:
 name: linux
 containers:
 - name: hardened-app
 image: alpine:latest
 command: ["sleep", "3600"]
 securityContext:
 readOnlyRootFilesystem: true
 volumeMounts:
 - name: temp-storage
 mountPath: /tmp
 bindMountOptions:
 - noexec
 - nosuid
 volumes:
 - name: temp-storage
 emptyDir: {}

Example 2: emptyDir volume permission mode with sticky bit

This full Pod manifest creates an emptyDir volume using mode: 01777 to enforce standard Unix /tmp sticky bit protections across containers.

apiVersion: v1
kind: Pod
metadata:
 name: hardened-emptydir-pod
 namespace: default
spec:
 os:
 name: linux
 containers:
 - name: app-container
 image: alpine:latest
 command: ["sleep", "3600"]
 volumeMounts:
 - name: shared-tmp
 mountPath: /tmp
 volumes:
 - name: shared-tmp
 emptyDir:
 mode: 01777

Verifying the features in Linux

To verify that these features are actively enforcing restrictions, you can run kubectl exec into the container. The following examples simulate attempts to perform actions that are successfully blocked by these features.

Verifying noexec

Attempt to write and run a script on a volume mounted with noexec:

# 1. Exec into the pod
kubectl exec -it hardened-bindmount-pod -- sh

# 2. Create an executable script on the mounted volume
cd /tmp
echo '#!/bin/sh' > test.sh
echo 'echo "Executing untrusted code..."' >> test.sh
chmod +x test.sh

# 3. Attempt to run the script
./test.sh

Expected result:

sh: ./test.sh: Permission denied

Even if an executable file is created, the Linux kernel refuses execution because MS_NOEXEC is enforced at the bind mount level.

Verifying the sticky bit

Attempt to delete another user's file in an emptyDir with 01777 permission mode:

# 1. Exec into the pod
kubectl exec -it hardened-emptydir-pod -- sh

# 2. Verify directory permissions on /tmp
ls -ld /tmp
# Output: drwxrwxrwt 2 root root ... /tmp (Notice the 't' indicating sticky bit)

# 3. Create a file as the guest user
su -s /bin/sh -c "touch /tmp/guest_file" guest

# 4. Attempt to delete that file as nobody
su -s /bin/sh -c "rm /tmp/guest_file" nobody

Expected result:

rm: can't remove '/tmp/guest_file': Operation not permitted

The kernel blocks deletion because the sticky bit (01777) restricts file removal strictly to the owner of the file.

Things to know

Keep these key details in mind as you begin using these features. Full details are available in the official documentation for bind mount options, emptyDir volume mode, and emptyDir volumes.

How do I get involved?

These new features are driven by SIG Node and SIG Storage. You can find more details in the KEPs for these enhancements: KEP-5855 (bind mount options) and KEP-5502 (emptyDir permission mode).

Reach out to SIG Node:

Reach out to SIG Storage:

16 Sep 2026 6:30pm GMT