04 Sep 2026

feedKubernetes Blog

Kubernetes v1.37: KubeletInUserNamespace (aka Rootless mode) Graduates to Beta

Kubernetes v1.37 promotes the KubeletInUserNamespace feature gate to beta. With this feature enabled, all of the node components (kubelet, CRI and OCI runtimes, CNI plugins, and kube-proxy) can run as a non-root user on the host, using a Linux user namespace. This technique is also known as rootless mode. The work started as an experiment in 2018, and was merged into Kubernetes v1.22 (2021) as an alpha feature (Kubernetes Enhancement Proposal KEP-2033).

This feature should not be confused with user namespaces for pods (hostUsers: false with the UserNamespacesSupport feature gate, GA since v1.36), which puts pods in user namespaces but still runs the node components as root. These two features do not conflict. Moreover, they can be combined to nest Kubernetes inside Kubernetes without resorting to the full privileged: true.

Why run the node components in a user namespace?

Because the node components have historically had container-breakout vulnerabilities that could compromise full root privileges on the host.

Examples of such vulnerabilities include:

By running the node components in a user namespace, the potential damage is confined to the non-root user's account. Notably, an attacker cannot conceal their intrusion by modifying the kernel, the boot loader, or the firmware.

It should still be noted that user namespaces are not effective for mitigating vulnerabilities in the kernel itself. User namespaces should be used in conjunction with traditional hardening measures such as seccomp to prevent containers from invoking unnecessary system calls.

Use cases

How does it work?

A Linux kernel user namespace maps a host level non-root user (e.g., UID 1000) to a fake root user inside the namespace. The UID 0 privileges are limited to the inside of the namespace. The fake root is enough for most of the node components' tasks: mounting volumes, creating cgroups, and configuring the network namespaces of pods. It still comes with some caveats that may break compatibility with specific CNI and CSI drivers, though.

The user namespace has to be created outside of Kubernetes. For example, Rootless Docker can be used to prepare the user namespace in which Kubernetes runs.

The KubeletInUserNamespace feature gate itself is quite "boring": basically it just lets the kubelet ignore permission errors that occur when setting some sysctl values (e.g., vm.overcommit_memory and kernel.panic) and when watching kernel messages via /dev/kmsg.

See Running Kubernetes Node Components as a Non-root User for further information.

What changed from Alpha to Beta?

Several related improvements have also happened outside the promotion of the feature gate itself:

With these improvements, a Kubernetes cluster with KubeletInUserNamespace can now also be nested inside Kubernetes pods with hostUsers: false (UserNamespacesSupport).

How to use it

kind

The easiest way is to use kind (a Kubernetes SIG Testing project) to run a Kubernetes cluster in rootless Docker, rootless nerdctl, or rootless Podman:

# Example using Docker
dockerd-rootless-setuptool.sh install
kind create cluster

Depending on the host configuration, you may need additional configuration for systemd, kernel modules, sysctl, etc.

See the Docker documentation and the kind documentation for further information.

minikube

minikube (a Kubernetes SIG Cluster Lifecycle project) also supports running a Kubernetes cluster in rootless Docker or rootless Podman:

dockerd-rootless-setuptool.sh install
minikube start --driver=docker

See the minikube documentation for further information.

Usernetes

Usernetes (a third-party project) is a distribution of rootless Kubernetes, maintained by the author of this article. The project began in 2018, and it is where the KubeletInUserNamespace feature gate originally came from.

Unlike kind and minikube, Usernetes supports creating a cluster with multiple rootless Docker / Podman / nerdctl nodes, connected using VXLAN via the Flannel CNI plugin.

Usernetes also experimentally supports a Kubernetes-in-Kubernetes mode.

k3s

k3s (a CNCF Sandbox project) also supports rootless mode. Unlike kind, minikube, and the current generation of Usernetes, rootless k3s does not rely on an external runtime such as rootless Docker.

What's next?

Depending on feedback and adoption, the Kubernetes project plans to graduate this feature to General Availability (GA) in a future release. If you have feedback on this feature, please open an issue in the kubernetes/kubernetes repository.

The project is also discussing several Kubernetes Enhancement Proposals that may contribute to simplifying Kubernetes-in-Kubernetes with this feature:

Getting involved

We always welcome new contributors. If you would like to get involved, you can join the Node Special Interest Group (SIG Node).

If you would like to share feedback, you can do so on our public Slack channel (visit https://slack.k8s.io/ for an invitation if you need one).

Special thanks to everyone who helped design and implement this feature, including but not limited to (in alphabetical order):

04 Sep 2026 6:30pm GMT

03 Sep 2026

feedKubernetes Blog

Kubernetes v1.37: DRA Updates

Kubernetes 1.37 is here and Dynamic Resource Allocation (DRA) keeps pushing past where it started! This release brings DRA Extended Resource support to GA, a milestone the team has been building toward for three straight releases. Several more features graduate to Beta or GA. A fresh batch of alpha features rounds out the release.

I'll dive into what's new for DRA in Kubernetes 1.37!

What's stable in 1.37

DRA Extended Resource support has graduated to GA. This is the mechanism that lets DRA drivers satisfy requests made through the traditional extended resource API, think example.com/gpu in a Pod spec, without requiring a separate device plugin alongside the DRA driver. An extended resource name can be set directly on a DeviceClass, and Pods requesting it get matched to a device through DRA with no ResourceClaim needed on the workload's part.

It's been on a steady path since KEP acceptance in 1.34. Alpha landed in 1.35, Beta in 1.36, and now it's Stable. For cluster operators, this is what makes DRA adoption gradual. Existing workloads written against extended resources keep working unmodified while the backend allocation logic moves over to DRA.

ResourceClaims status with possible standardized network interface data adds a devices field to ResourceClaim .status, letting DRA drivers report per-device status, including, for network devices, the interface name, MAC address, and IP addresses. This gives users and controllers visibility into device state that was previously invisible once a device was configured in a Pod, and makes it possible to build things like network services that rely on a device's reported IPs.

DRA: device taints and tolerations is now Stable; DRA drivers can mark devices as tainted so they're skipped for new Pod scheduling, and cluster admins can apply the same taints cluster-wide via a DeviceTaintRule, without reconfiguring drivers. Pods already using a tainted device can be evicted automatically, unless their ResourceClaim explicitly tolerates the taint. This mirrors node taints and tolerations, letting operators take a single device offline for maintenance or mark it degraded, without disrupting the rest of the cluster.

Standard numaNode device attribute standardizes resource.kubernetes.io/numaNode as a shared attribute name, so devices from different drivers can be compared on the same NUMA node instead of each driver inventing its own name for it. It landed directly as stable in 1.37, since it's a naming/registration KEP with no feature gate or in-tree behavior change.

Feature promoted to Beta

ResourceClaim support for workloads graduates to Beta behind the DRAWorkloadResourceClaims feature gate, which stays disabled by default. In a cluster that has the feature enabled, Workloads and PodGroups can reference ResourceClaims directly, so a single claim can be shared across an entire group of Pods. This is instead of claims being capped at 256 Pods through the old per-Pod reservation limit.

The DRA Device Attributes Downward API is aimed at supporting device injection into KubeVirt VMs. Drivers populate a Metadata field when preparing a claim, and the framework writes it to a JSON file mounted into the container via CDI, letting workloads read a device's PCI bus address, MAC address, and other attributes directly instead of requiring custom controllers to watch and translate ResourceClaims and ResourceSlices.

Alpha features

List types for attributes moved into a second Alpha in 1.37, letting a device attribute hold more than one value instead of a single scalar, such as a CPU that's adjacent to more than one PCIe root. This makes it possible to match or distinguish devices based on overlapping or non-overlapping sets of values, while single-value attributes keep working as they do today.

Node allocatable resource requests moved into Alpha 2. It lets the scheduler and kubelet treat DRA-managed CPU, memory, and similar node resources the same way they treat ordinary resource requests, so a node doesn't get oversubscribed and users no longer have to duplicate the same request in both a ResourceClaim and the pod spec.

Resource availability visibility moved to a second Alpha in Kubernetes 1.37. Users create a ResourcePoolStatusRequest to get a point-in-time availability snapshot. To refresh it, delete and recreate the request; it is not a continuous monitoring API.

DRA: Optional Node Operations lets a driver skip kubelet's prepare and unprepare calls for allocations that don't need any setup on the node. This makes it possible to avoid an unnecessary dependency on the driver for allocations where there's genuinely nothing for it to do locally.

Derived Attributes is a new feature that lets you use CEL expressions to match up devices based on your own custom rules. Before this, pairing devices from different vendors (like a GPU/TPU and a NIC on the same NUMA node) only worked if both drivers used the exact same attribute name. If one used numa and the other used numaNode, the scheduler couldn't pair them together. Now, you can easily bridge these differences yourself inside your manifest, meaning you don't have to wait for hardware vendors to agree on standardized attribute names. Beyond just fixing naming differences, you can also use CEL to handle more complex scenarios like slicing a specific ID out of a long, monolithic topology string, or grouping devices into custom performance tiers based on their available capacity.

DRA Device Compatibility Groups lets drivers tag partitions of a device, like MIG vs vGPU profiles on the same GPU, with compatibility groups, so the scheduler rejects incompatible combinations up front instead of the driver failing at node preparation time. It's controlled by the DRADeviceCompatibilityGroups feature gate, disabled by default.

PreQueueingHint extension point is new as Alpha in 1.37. DRA ResourceClaim events used to trigger a full scan of every unschedulable pod, an O(N²) cost during large scale-ups. The DRA plugin now uses a pod informer index to narrow that to just the pods actually affected, cutting the requeue path to O(1) and roughly doubling scheduling throughput in early benchmarks. Controlled by the SchedulerPreQueueingHints feature gate.

DRA Consumable Capacity now supports fractional values in CapacityRequestPolicyRange, enabling more precise capacity requests and allocation for devices with fractional resources. This improves flexibility for workloads that require fine-grained resource allocation. The enhancement is gated by the DRAFractionalCapacityRange feature gate, which is in Beta in 1.37.

What's next

DRA continues to mature with every release. Several features currently in Alpha and Beta are on track to progress in the coming releases, and the community keeps working on DRA's performance, scalability, and reliability. Expect another ambitious set of DRA features in Kubernetes 1.38.

Getting involved

A good starting point is joining the WG Device Management Slack channel and meetings which happens at US/EU and EU/APAC friendly time slots.

Not all enhancement ideas are tracked as issues yet, so come talk to us if you want to help or have some ideas yourself! We have work to do at all levels, from difficult core changes to usability enhancements in kubectl which could be picked up by newcomers.

Acknowledgments

The following KEP owners added or promoted a feature in the 1.37 release (in alphabetic order):

This would not have been possible without the help of the reviewers and approvers. So a huge thanks to everyone else who helped shape this release, in ways big and small. Given enough eyeballs, all bugs are shallow and this release had plenty of them, watching closely and caring enough to make things better. DRA got better this cycle because of all of you.

03 Sep 2026 6:30pm GMT

02 Sep 2026

feedKubernetes Blog

Kubernetes v1.37: Scale Workloads to Zero with HorizontalPodAutoscaler

Kubernetes v1.37 includes API support for horizontal autoscaling of workloads down to zero replicas. This feature is now Beta and enabled by default. A HorizontalPodAutoscaler (HPA) that uses a suitable object metric or external metric can now scale a workload to zero replicas, then bring it back when the metric changes.

Before v1.37, you needed an add-on or external component, or you had to enable the Alpha feature gate, to scale from zero. It is now part of core Kubernetes.

Scaling to zero removes the last idle Pod from workloads such as queue consumers and batch processors. The savings are largest when each Pod reserves expensive resources, including dedicated CPUs or GPUs.

The trade-off is cold-start time: the HPA must observe the metric, schedule a Pod, and start the application. This works well when work can wait in a durable queue.

Kubernetes Services do not buffer requests while no Pods are ready, so HTTP and other request-driven workloads need a separate buffering layer.

Why scaling from zero needs a different metric

The HPA commonly scales on CPU or memory usage. Both metrics come from running Pods. Once the replica count reaches zero, there are no Pods left to measure and no signal that can tell the HPA to scale back up.

Object and external metrics do not have that limitation. A queue length, for example, exists independently of the workers that consume it. The HPA can continue reading the queue length while no workers are running.

The following example scales a queue consumer to and from zero using an external metric.

Configure an external metric

The following example uses a Prometheus metric named queue_consumer_lag. It assumes that Prometheus already collects a series similar to this one:

queue_consumer_lag{namespace="default",name="worker_tasks"}

Kubernetes needs a metrics adapter to make that value available through the External Metrics API. One implementation is the Prometheus Adapter, which can expose the series using an externalRules entry:

externalRules:
- seriesQuery: '{__name__="queue_consumer_lag",name!=""}'
 metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (name)
 resources:
 overrides:
 namespace:
 resource: namespace

The exact adapter installation and discovery rules depend on your monitoring setup. See the Prometheus Adapter guide to external metrics for the full configuration options.

Before creating the HPA, you can verify that Kubernetes can read the metric:

kubectl get --raw \
 '/apis/external.metrics.k8s.io/v1beta1/namespaces/default/queue_consumer_lag?labelSelector=name%3Dworker_tasks'

The request should return the current value for worker_tasks. If it does not, fix the metrics pipeline before configuring the HPA. An HPA cannot scale from zero when its metric is unavailable.

Configure the HPA

The following HPA targets a Deployment named queue-worker. It allows between zero and ten replicas, with one replica requested for each 30 queued tasks:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
 name: queue-worker
 annotations:
 kubernetes.io/description: "Scales queue-worker based on the number of queued tasks"
spec:
 scaleTargetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: queue-worker
 minReplicas: 0
 maxReplicas: 10
 metrics:
 - type: External
 external:
 metric:
 name: queue_consumer_lag
 selector:
 matchLabels:
 name: worker_tasks
 target:
 type: Value
 value: "30"

When the queue is empty, the HPA can reduce the Deployment to zero replicas. When tasks arrive, the external metric remains available and the HPA calculates a new replica count, capped at ten by maxReplicas.

Start the Deployment with at least one replica. Manually setting a Deployment to zero has always paused autoscaling. The HPA preserves that behavior and will not wake a workload that it did not scale down itself.

Normal HPA behavior still applies. In particular, the default downscale stabilization window is five minutes. The window prevents a short drop in queue length from immediately removing all workers. You can configure the window through spec.behavior.scaleDown if your workload needs different behavior.

How the HPA distinguishes zero from paused

Scaling from zero creates an ambiguity. A replica count of zero can mean that the HPA scaled the workload down, or that an operator manually paused it.

The controller resolves this with a ScaledToZero status condition. When the HPA scales a workload from one or more replicas to zero, it records ScaledToZero=True. The condition tells later reconciliation loops that the controller owns the zero state and should continue evaluating object or external metrics.

After scaling the workload back up, the controller changes the condition to ScaledToZero=False with the reason NotScaledToZero. A workload at zero without the ScaledToZero=True condition remains paused.

You can inspect the conditions with:

kubectl describe hpa queue-worker

If the adapter cannot return the configured metric, the HPA reports ScalingActive=False with a reason such as FailedGetExternalMetric. Restore the metric or manually scale the workload to recover capacity.

Before upgrading or rolling back

In Kubernetes v1.37, the HPAScaleToZero feature gate is enabled by default on both the kube-apiserver and kube-controller-manager. The API server accepts minReplicas: 0; the controller manager performs the condition-based scaling.

During a version-skewed control plane upgrade, wait until both components support the feature and have it enabled before creating HPAs with minReplicas: 0. A controller manager with the feature disabled treats replicas: 0 as a manual pause and may leave a workload at zero.

Before disabling the feature gate or downgrading to a version without the condition-based implementation:

minReplicas: 0 also requires at least one object or external metric. The API server rejects an HPA that only contains resource metrics such as CPU or memory.

From Alpha to Beta

The first Alpha implementation shipped in Kubernetes v1.16. Kubernetes v1.36 added the ScaledToZero condition and the controller behavior needed to distinguish an automatic scale-down from a manual pause.

Kubernetes v1.37 enables the feature by default after adding integration and end-to-end coverage for scaling down to zero and back up from an external metric. The next step is to gather operational feedback before considering graduation to GA.

How can I learn more?

How to get involved

This feature is owned by SIG Autoscaling. Join Kubernetes Slack and the #sig-autoscaling channel to share feedback from Beta usage.

Acknowledgements

Thanks to the SIG Autoscaling contributors who took this feature from the original v1.16 implementation to the condition-based redesign and Beta graduation. Thanks also to Guy Templeton and Adrian Moisey for reviewing the KEP, and to the release, documentation, and production-readiness reviewers who helped prepare it for Kubernetes v1.37.

02 Sep 2026 6:30pm GMT

01 Sep 2026

feedKubernetes Blog

Kubernetes v1.37: etcd RangeStream Cuts Memory Use on Large List Reads

I am excited to announce that etcd RangeStream is graduating to beta in Kubernetes v1.37. Paired with etcd v3.7, it reduces the memory the API server and etcd need to read a large collection, and makes peak usage more predictable.

The cost of large reads

The API server serves most list and watch requests from its in-memory watch cache. Populating that cache requires reading a resource's full state from etcd, at startup and on every re-initialization. For a resource with many objects, or large ones, such as Pods, that read is expensive.

The API server already paginated these reads, asking etcd for a fixed number of keys at a time rather than the whole collection at once. But a page bounded by key count has no awareness of object size, so a page of large objects can still be very large. That makes memory usage hard to predict, and a bad combination of object size and concurrent reads can be enough to trigger an OOM. etcd's unary Range assembles each page in full before sending it, and the API server holds it while decoding, so the same payload sits in memory on both sides at once. Most of that cost lands on etcd, which is also where streaming helps most.

Streaming reads with RangeStream

etcd v3.7 adds a streaming version of that read, the RangeStream RPC. It takes the same RangeRequest as Range and returns the same result set, but instead of building the whole response up front, etcd splits it into chunks and streams them. Chunk size is tuned adaptively to the values being returned, so a collection of large objects is bounded by bytes rather than by a key count, and memory is freed as the stream progresses instead of being held until a whole page is assembled.

When the feature is enabled, the API server uses RangeStream wherever it reads a whole collection out of etcd. This includes watch cache initialization, and the fallback paths where a list request cannot be served from the cache and reads etcd directly. In either case the API server decodes each chunk as it arrives and releases it before pulling the next one, so neither side ever holds the whole collection.

Requirements

RangeStream is used when the EtcdRangeStream feature gate is enabled on the kube-apiserver, which is beta and on by default in v1.37, and etcd is v3.7 or later. The API server resolves etcd's support at startup and also falls back at runtime if a call returns Unimplemented, so an API server paired with an older etcd keeps using the paginated Range path on its own. To turn it off, disable the gate:

--feature-gates=EtcdRangeStream=false

Confirming RangeStream is in use

The API server records streamed reads under their own operation label on its etcd metrics. A non-zero count here means RangeStream is in use:

etcd_request_duration_seconds_count{operation="listStream"}

If it stays at zero, the API server is still using the paginated Range path, most likely because etcd is older than v3.7.

Learn more

If you have questions or feedback, join the #sig-etcd channel on Kubernetes Slack.

01 Sep 2026 6:30pm GMT

31 Aug 2026

feedKubernetes Blog

Kubernetes v1.37: Storage Version Migration Enabled by Default

I am excited that storage version migration (SVM) has graduated to General Availability (GA) in Kubernetes v1.37!

After a number of releases of work and testing, the built-in StorageVersionMigration API (storagemigration.k8s.io/v1) and control plane controller are now fully stable and enabled by default across all v1.37 Kubernetes clusters.

The problem with stale storage versions

In Kubernetes, stored API resources are written using a specific storage version (schema representation). The way Kubernetes interacts with object storage fundamentally requires mutation of a resource in order to ensure that the latest storage version is used for all resources. This creates problems when you want to change the storage version of a resource.

One example of a scenario where you may want to change the storage version of a resource is when you are promoting a CRD to drop an older API version (such as v1alpha1) to a newer version (leaving just v1beta1 and v1). It's a problem to drop the older API version whilst there are still resources stored with the old alpha version.

To avoid problems, you designate v1 as the new storage version; but, on it's own, that's not enough. While new writes are stored as v1, any existing resource could remain stored as v1alpha1 or v1beta1 in storage. You cannot safely remove v1alpha1 from the CRD's .status.storedVersions or drop serving support until every single resource in storage has been re-written to not be serialized and stored with the alpha version.

Another relevant example is encryption at rest and, related, key rotation. When you configure encryption at rest or rotate encryption keys, existing resources in storage remain unencrypted (or encrypted under old keys) until they are actively re-written through the Kubernetes API server.

Historically, cluster administrators and CRD authors had to rely on manual kubectl get / kubectl replace scripts, or to deploy the out-of-tree kube-storage-version-migrator component to force re-writes. These approaches were often tedious, error-prone, and difficult to monitor.

How storage version migration works

Initiating a storage version migration is as simple as creating a declarative StorageVersionMigration object. The built-in StorageVersionMigrator controller in the Kubernetes control plane watches for these objects, and automatically migrates existing resources to the default storage version for that API.

Example: Migrating a custom resource API

Suppose you have updated a CustomResourceDefinition (crontabs.example.com) to use v1 as its storage version. To migrate all existing stored resources off older versions, create a StorageVersionMigration:

apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
 name: crontabs-migration
spec:
 resource:
 group: example.com
 resource: crontabs

Apply the manifest using kubectl:

kubectl apply -f crontabs-migration.yaml

Monitoring and verifying migrations

The StorageVersionMigrator controller updates the status of the StorageVersionMigration object as migration progresses. You can inspect the migration status using kubectl:

kubectl get storageversionmigration.storagemigration.k8s.io/crontabs-migration -o yaml

A successful migration will report a Succeeded condition set to True:

status:
 conditions:
 - type: Running
 status: "False"
 lastUpdateTime: "2026-08-02T10:05:00Z"
 reason: StorageVersionMigrationInProgress
 - type: Succeeded
 status: "True"
 lastUpdateTime: "2026-08-02T10:05:00Z"
 reason: StorageVersionMigrationSucceeded

Once the migration has succeeded, you can be confident that all instances of the resource in storage are stored in the current storage version. For CRDs, the stored version should be updated in the CRD's .status.storedVersions to only contain the preferred version. If the .status.storedVersions is not updated following a successful migration then that means that the CRD was updated during the migration. In that case, the migration should be retried in order to safely deprecate an older storage version.

Including migrations in your CRD manifests

Because StorageVersionMigration is a standard declarative Kubernetes API, CRD authors can bundle or trigger migrations directly alongside CRD upgrades. For example, you can include the migration in the same manifest as your updated CustomResourceDefinition:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
 name: crontabs.example.com
spec:
 group: example.com
 # Updated versions list where v1 has storage: true
 ...
---
apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
 name: crontabs-migration
spec:
 resource:
 group: example.com
 resource: crontabs

What's next?

SIG API Machinery would love to hear your feedback as you adopt built-in Storage Version Migration in your clusters. Reach out to us on the #sig-api-machinery Slack channel or participate in our community discussions!

31 Aug 2026 6:30pm GMT

28 Aug 2026

feedKubernetes Blog

Kubernetes v1.37: Pod Certificates and Cluster Trust Bundles

Pod Certificate / Cluster Trust Bundles Blog Post

Kubernetes brings a wealth of features that make it easy to run your production workloads securely and reliably. While aspects like scheduling, health checks and resource limits are probably at the front of your mind, one other important feature of Kubernetes is production identity - how your workload can authenticate to other systems in order to do its job.

Up until now, the primary production identity mechanism built into Kubernetes has been service account JWTs (JSON Web Tokens). These are cryptographically-signed tokens, issued by the control plane of your cluster, that let anyone in the world understand who is calling when your workload uses them.

In Kubernetes 1.37, the foundations of a new built-in production identity technology have gone GA. Pod Certificates (and the closely-associated Cluster Trust Bundles) build X.509 certificate issuance for TLS and mTLS directly into core Kubernetes.

Why?

Service account JWTs have a lot going for them:

However, service account JWTs have one big downside - they are bearer tokens. With bearer tokens, if you have the token, then you are the identity asserted by the token. And since you necessarily have to hand copies of the JWT to all your peers in order to authenticate to them, they can be you, too.

There are partial mitigations for this, and service account tokens make use of them (time-, object-, and audience-binding), but none are complete defences.

A solution to this problem lies in proof-of-possession credentials, where you don't send your entire credential to your peer, but only a proof that you possess the credential. In practice, these schemes are always built on asymmetric cryptographic signatures (RSA, ECDSA, and friends).

There are few different standard approaches, such as request signing (AWS SigV4, JWT DPoP, RFC 9421), but the most widely-deployed and understood solution is X.509 certificates, as used in TLS. In TLS, your credential is split into two pieces

The goal of Pod Certificates is to make using X.509 certificates from your Kubernetes workload just as easy as using service account JWTs, while maintaining Kubernetes' high security bar. I think we've hit this target.

As I'll cover in the architecture and example sections below, there are many similarities between the design of service account JWT issuance and Pod Certificates. One significant place they diverge, however, is that Pod Certificates is a much more flexible mechanism. Kubernetes only offers one flavor of service account JWTs, with standardized claims.

The X.509 ecosystem is significantly more varied than the JWT ecosystem, and X.509 certificates used for different purposes contain different extensions and information. For this reason, Pod Certificates has common machinery built into Kubelet, but offers a pluggable interface so that many different types of certificates can be issued within a single cluster, at the same time.

In the fullness of time, I expect Kubernetes to offer at least two built-in certificate providers:

In the remainder of this article, I'll take you through the overall architecture of a Kubernetes workload using Pod Certificates, as well as give you an example of installing and using a real (toy) Pod Certificates signer controller.

Architecture

When you use Pod Certificates and Cluster Trust Bundles, there are the following major components:

Block diagram of an application using Pod Certificates

Architecture of an application using Pod Certificates

The best way to get a sense of what these components each do is to follow the issuance process chronologically:

  1. Once your application pod is scheduled to a node, Kubelet identifies all of the podCertificate and clusterTrustBundle projected volumes sources in its spec.
  2. For each podCertificate source:
    1. Kubelet generates a new private key according to the keyType field.
    2. Kubelet creates a PodCertificateRequest addressed to the signer named in the source.
    3. The signer controller sees the PodCertificateRequest and decides whether or not to issue the certificate.
    4. The signer controller issues the certificate by filling out the status.certificateChain field.
    5. The signer controller also fills out the status.beginRefreshAt field to instruct Kubelet when it should begin trying to refresh the certificate.
      certificate to the container filesystem.
  3. For each clusterTrustBundle source:
    6) Kubelet retrieves the issued certificate, and writes the private key and
    1. Kubelet collects all the ClusterTrustBundles that match the signer name
    2. Kubelet unifies all of the certificates from all matching ClusterTrustBundles, and (stably) reorders them (to prevent applications from accidentally depending on a particular ordering).
      and label selectors in the source.
    3. Kubelet writes the certificates to the file path named in the source.
      and trust anchors from the filesystem.
  4. Your application pod starts up, and the application reads keys, certificates,
  5. Kubelet periodically updates the files from clusterTrustBundle sources as the contents of the selected ClusterTrustBundles changes. The application must pick up the changes using inotify or polling.
  6. As each certificate's beginRefreshAt time passes, Kubelet repeats the process in step 2 to refresh the certificates, and write the update private keys and certificate chains to the filesystem. As in step 5, the application must pick up changes using inotify or polling.

Some key takeaways:

Try it out

Because the Kubernetes project does not yet ship any Pod Certificate signers in core, in order to try these features out, you will need to install a third-party signer into your cluster. To make this easier, I have written Tinycert, which you can install into your cluster (or a Kind cluster).

Tinycert is not a full production solution, but it's a good starting point for experimenting with Pod Certificates, as well as a base for creating your own signers.

Tinycert provides:

What next?

Happy hacking!

28 Aug 2026 6:30pm GMT

27 Aug 2026

feedKubernetes Blog

Kubernetes v1.37: Metrics API graduates to stable

Kubernetes v1.37 promotes the metrics.k8s.io API to stable (v1). This API provides CPU and memory usage for nodes and Pods, and is the API behind commands such as kubectl top and resource-metrics-based autoscaling.

For cluster operators and application developers, this graduation means that the API now has the stability guarantees associated with a Kubernetes stable API. The v1 API has the same resource types and fields as v1beta1; this is an API-version graduation, not a change to the metrics that are collected or returned.

A long-lived API reaches stable

The resource Metrics API was introduced as alpha in Kubernetes v1.6 and became beta in v1.8. It has remained unchanged and has been used in production for years by clients including the HorizontalPodAutoscaler (HPA) and kubectl top. Kubernetes v1.37 formally graduates that proven API to metrics.k8s.io/v1.

The API exposes two resource types:

The API remains intentionally small. It provides the resource metrics needed for autoscaling and basic inspection; it is not a replacement for a full monitoring pipeline or the custom metrics (custom.metrics.k8s.io) API.

What changed with the v1.37 release?

The v1 API surface is identical to v1beta1, except for the API version. There are no renamed fields, new fields, or changes to the meaning of the returned CPU and memory values.

For example, a client can retrieve node metrics from the stable endpoint:

kubectl get --raw /apis/metrics.k8s.io/v1/nodes

Likewise, it can retrieve metrics for the pods in a namespace:

kubectl get --raw /apis/metrics.k8s.io/v1/namespaces/default/pods

kubectl top supports both API versions. It prefers v1 when available and automatically falls back to v1beta1 on clusters that do not yet serve v1. The HPA controller currently supports only v1beta1. Support for discovery-based selection between v1 and v1beta1 is planned, but is not available in Kubernetes v1.37.

What you need to do

You don't need to enable any feature gate. The Metrics API is served through the API aggregation layer, by an implementation such as metrics-server. You can choose any implementation of metrics.k8s.io; for the v1 metrics API to be available in your cluster, your chosen implementation must serve the v1.metrics.k8s.io API, and you need to register an associated APIService.

During the transition, implementations should serve both v1 and v1beta1. Keeping both versions available maintains compatibility with older clients. The v1beta1 API remains available in Kubernetes v1.37.

You can see which versions your cluster serves with:

kubectl get --raw /apis/metrics.k8s.io/ | jq .

Once your metrics implementation supports v1, you can also check that its APIService is available:

kubectl get apiservice v1.metrics.k8s.io

Learn more

Get involved

The Metrics API is maintained by SIG Instrumentation. To ask questions, share feedback, or contribute, join the #sig-instrumentation channel on Kubernetes Slack or attend a SIG Instrumentation meeting.

27 Aug 2026 6:30pm GMT

26 Aug 2026

feedKubernetes Blog

Kubernetes v1.37: Garhwal

Editors: Arsh Sharma, Christopher Tineo, Kirti Goyal, Sophia Ugochukwu, Swathi Rao, Troy Connor

Similar to previous releases, the release of Kubernetes v1.37 introduces new Stable, Beta, and Alpha features. The consistent delivery of high-quality releases underscores the strength of our development cycle and the vibrant support from our community.

This release consists of 67 enhancements. Of those enhancements, 16 have graduated to Stable, 23 have graduated to Beta, 27 are entering Alpha, and 1 is a deprecation/removal.

Release theme and logo

The theme for Kubernetes v1.37 is Garhwal (गढ़वाल, pronounced gaṛhvāl), a Himalayan region of Uttarakhand, India. The snow-capped peaks of the Garhwal Himalaya, deodar forests, terraced fields, rivers and streams, and mountain paths shape both the region and the logo. Together, these elements reflect a community in which every layer, route, and contribution is connected.

The logo is imagined as a window into Garhwal's landscape.1 Inside, terraced fields climb towards the snowy peaks, each level supported by the one below, much as every Kubernetes release depends on work carried forward. A river winds through the valley and gathers mountain streams, reflecting contributions from many SIGs and communities flowing into one project.

The deodar forest represents the wider Kubernetes ecosystem, where distinct projects share common ground and grow side by side. Stonework and woodcraft shape the path and mountain house, placing people at the centre and evoking shared foundations maintained for those who follow. Above the river, colourful flags catch the wind and bring the scene to life.

Encircling the scene is a patterned frame inspired by basketry woven from ringaal, a flexible dwarf Himalayan bamboo. Individual strips gain strength when interlaced, just as code, reviews, tests, documentation, and coordination come together to make a release.

Within the frame, the Himalayan monal, Uttarakhand's state bird, lives at high altitudes in the Himalaya. Its iridescent plumage holds many colours at once, much as the Kubernetes community brings many skills and perspectives into one project. Flowers of red buransh (Rhododendron arboreum), Uttarakhand's state tree, carry Kubernetes helms at their centres, linking a familiar bloom of Garhwal with the symbol shared by the community. The house bears १.३७ (1.37 in Devanagari numerals), grounding the release in the landscape.

1. Keep looking through the window (the logo). Watch the river flow and the flags catch the wind. In 37 seconds, the landscape reveals its magic. 😉

Spotlight on key updates

Kubernetes v1.37 is packed with new features and improvements. Here are a few select updates the Release Team would like to highlight!

Stable: Resilient watchcache initialization

Kubernetes v1.37 completes the work on resilient watch cache initialization: the ResilientWatchCacheInitialization feature gate reached Stable back in v1.34, and in v1.37 the remaining WatchCacheInitializationPostStartHook gate graduates to Stable and is locked on. It has defaulted to enabled since v1.36, hardening the API server at startup and during recovery. Watchcache initialization and reinitialization no longer create a traffic spike of requests against etcd, and requests are handled gracefully instead of piling up while the cache warms.

Instead of allowing expensive list and watch requests to overload etcd or exhaust API Priority and Fairness capacity, kube-apiserver now safely delegates bounded requests and rejects others with HTTP 429 responses. This reduces the risk of control plane outages in large clusters. Clients (including custom controllers and operators) should be designed to handle HTTP 429 Too Many Requests responses gracefully by respecting Retry-After headers and implementing exponential backoff.

This work was done as part of KEP #4568 led by SIG API Machinery.

Beta: HorizontalPodAutoscaler scale to zero

In Kubernetes v1.37, HorizontalPodAutoscaler scale to zero support is graduating to Beta. First introduced in Kubernetes v1.16, it is now enabled by default. For workloads that are using object or external metrics, this feature allows HorizontalPodAutoscalers to scale down to zero Pods when idle, then restore them when demand returns. Doing that can reduce costs for queue consumers, batch jobs, and GPU workloads. Setting spec.minReplicas: 0 applies this functionality for workloads.

Scaling to zero based on CPU and memory metrics is not supported because those metrics depend on active Pods. Instead, this feature is for situations such as leaving the replica count at zero until there is queued work to process.

While the HorizontalPodAutoscaler is holding a workload at zero replicas, it records a ScaledToZero condition with True in the HorizontalPodAutoscaler's status. The HorizontalPodAutoscaler controller then uses this condition to distinguish a workload that it scaled to zero (and will scale back up when the metric returns) from one that was manually deactivated by setting its replica count to 0. Once the workload is scaled back up, the condition is set to False with the reason NotScaledToZero.

This work was done as part of KEP #2021 led by SIG Autoscaling.

Beta: Manifest-based admission control configuration

Kubernetes v1.37 graduates manifest-based admission control configuration to Beta. Admission webhooks and CEL-based policies can now be loaded from manifest files on disk, via the staticManifestsDir field in AdmissionConfiguration, instead of living only in the Kubernetes API. Policies loaded this way are enforced from API server startup, keep working while etcd is unavailable, and can protect the API-based admission resources themselves from modification.

This work was done as part of KEP #5793 led by SIG API Machinery.

Alpha: Pod-level checkpoint and restore

Kubernetes v1.37 introduces Alpha support for Pod-level checkpoint and restore, extending the CRI with CheckpointPod and RestorePod RPCs, which allow the kubelet and compatible container runtimes to create a Pod checkpoint and restore a Pod from it. To use this feature, your container runtime(s) must also implement these new RPCs.

This work was done as part of KEP #5823 led by SIG Node.

Features graduating to Stable

This lists all the features that graduated to Stable (also known as General Availability). For a full list of updates including new features and graduations from Alpha to Beta, see the release notes.

This release includes a total of 16 enhancements promoted to Stable:

KYAML

KYAML is a safer and less ambiguous subset of YAML designed specifically for Kubernetes, not a replacement for it. Every KYAML file is valid YAML, so KYAML is a valid input for any version of kubectl, and spec files do not need to be written in KYAML for the input to be parsed. Your existing manifests, tooling, and pipelines don't need to change. Introduced as an Alpha feature in v1.34 and graduating to Beta in v1.35, KYAML graduates to Stable in v1.37 with conformance testing complete, and kubectl get -o kyaml is now Stable.

To learn more about KYAML, check out How to Pretty-Print Your Kubernetes YAML as KYAML and Why You'd Want To.

This work was done as part of KEP #5295 led by SIG CLI.

The metrics.k8s.io API

The metrics.k8s.io API graduates to Stable in Kubernetes v1.37 after spending nearly nine years in Beta. The API provides a standard way to retrieve CPU and memory usage for pods and nodes, powering widely used Kubernetes features such as the HorizontalPodAutoscaler (HPA) and commands like kubectl top.

The graduation follows the Kubernetes project's goal of avoiding permanent Beta APIs. Now that v1 exists, future Kubernetes releases will move over to it; v1beta1 remains usable throughout the transition, in line with the API deprecation policy, so you can adopt the Stable API without breaking existing workflows.

This work was done as part of KEP #5207 led by SIG Instrumentation.

SELinuxMount and SELinuxChangePolicy

In Kubernetes v1.37, SELinuxMount and SELinuxChangePolicy flags reach Stable and are enabled by default: this means that volumes get mounted with -o context=<label> (the MountOption default) instead of being recursively relabeled, but only when the volume's CSI driver opts in via .spec.seLinuxMount: true for the CSIDriver object.

A mount can only carry one SELinux context, so Pods with different SELinux labels sharing a volume on the same node, which used to coexist under recursive relabeling, can now fail to start. To retain the old behavior for a workload, it is advised to set the .spec.seLinuxChangePolicy to Recursive on a Pod.

This behavior itself also isn't locked until v1.38, so disabling it cluster-wide remains an option for one more release.

Clusters without SELinux enabled see no effect at all. To learn more, check SELinux Volume Label Changes goes GA (and likely implications in v1.37).

This work was done as part of KEP #1710 led by SIG Storage.

DRA features graduating to Stable

DRA: ResourceClaim status with possible standardized network interface data

The ResourceClaim .status.devices reaches Stable in Kubernetes v1.37, which allows drivers to report device-specific device status data for each allocated device in a resource claim. This makes it easier to see how a device is configured, troubleshoot problems, and use the device with other services.

This is particularly useful for network devices; before this field was added, if a Pod requested a network device via DRA, there was no way for any other component in the system to learn the IP address that was assigned to that network device. The new status field provides a standardized way for the DRA driver to export that information to components that need it, making DRA fully usable for attaching secondary network interfaces to Pods.

This work was done as part of KEP #4817 led by SIG Node and SIG Network.

DRA: handle extended resource requests via DRA Driver

DRA Extended Resource support reaches Stable in Kubernetes v1.37. This feature allows DRA drivers to fulfill requests made through the traditional extended resource mechanism, such as abc.example/gpu: 3 in a Pod spec, without requiring a separate device plugin.

With this mechanism, an extended resource name can be assigned directly to a DeviceClass. Pods requesting that resource can then have a device allocated through DRA without needing to define a ResourceClaim in the workload.

This work was done as part of KEP #5004 led by SIG Scheduling.

DRA: device taints and tolerations

Support for taints and tolerations for physical devices managed through DRA is now Stable in Kubernetes v1.37. By default, any available device can be considered for scheduling. This enhancement provides greater control over device scheduling by allowing DRA drivers to mark specific devices as tainted, preventing them from being selected for workloads. Alternatively, cluster administrators can create a DeviceTaintRule to taint devices based on specific selection criteria, such as all devices managed by a particular driver.

This work was done as part of KEP #5055 led by SIG Scheduling.

DRA: standard numaNode device attribute

Kubernetes v1.37 defines a new standard NUMA node device attribute. It standardizes resource.kubernetes.io/numaNode as a shared attribute name for device NUMA node information, allowing devices managed by different DRA drivers to be compared based on the same NUMA node. This avoids each driver defining its own attribute name and provides a consistent way to identify NUMA placement across devices. The enhancement lands directly as Stable because it is a naming and registration KEP with no feature gate or in-tree behavior changes.

This work was done as part of KEP #6072 led by SIG Node.

Node declared features

Node declared features graduate to Stable in Kubernetes v1.37, providing a framework to declare the availability of specific, feature-gated Kubernetes features for Nodes. This would then be used by control plane components (such as the kube-scheduler, admission controllers, or the API server itself) to manage version skew.

The feature introduces a new .status.declaredFeatures field for Nodes, which is used to declare a feature graduating through the Alpha → Beta → Stable stages. The control plane can use this to adopt the correct behavior even in a cluster running a mixture of different node versions.

Once features graduate to Stable and the control plane can assume all nodes support them across the supported version skew window, nodes stop reporting them.

The kubelet determines its declared features when it starts, based only on feature gates and the node's static configuration (so any changes require a kubelet restart).

This work was done as part of KEP #5328 led by SIG Node.

Storage version migrator

Kubernetes v1.37 sees the StorageVersionMigration API (storagemigration.k8s.io/v1) graduate to Stable and become enabled by default. It helps migrate existing resources, both built-in and custom, from an older storage version to the new storage version after an API upgrade, such as when the preferred storage version changes from v1beta1 to v1. It can also be used to rewrite existing data after a change to encryption at rest, so that stale data is stored using the new encryption settings.

Historically, cluster administrators and CustomResourceDefinition authors had to use manual kubectl get or kubectl replace scripts, or deploy the out-of-tree kube-storage-version-migrator component to rewrite existing resources. These approaches were often tedious, error-prone, and difficult to monitor.

To start a storage version migration, users would need to create a declarative StorageVersionMigration object. The built-in StorageVersionMigrator controller in the Kubernetes control plane watches for these objects and automatically migrates existing resources to the default storage version for that API. Since StorageVersionMigration is a standard Kubernetes API, CRD authors can trigger migrations as part of a CRD upgrade instead of managing the migration separately.

This work was done as part of KEP #4192 led by SIG API Machinery.

Stable: Pod certificates and Cluster Trust Bundles

Pod certificates and the closely related ClusterTrustBundles both graduate to Stable in Kubernetes v1.37, providing first-class support for distributing private keys, X.509 certificates, and trust bundles to Pods.

To use this, the developer or administrator chooses a signer name and deploys a signer controller that watches PodCertificateRequest objects, issues and refreshes certificates for eligible Pods, and maintains the corresponding ClusterTrustBundle objects containing the trust anchors needed to verify those certificates. A workload then opts into this identity by defining a podCertificate projected volume with the chosen signer name. Workloads can also mount a ClusterTrustBundle projected volume to load the trust anchor information.

This work was done as part of two KEPs - KEP #4317 and KEP #3257 led by SIG Auth.

Features graduating to Beta

Gang scheduling support in Kubernetes

As Kubernetes becomes the de facto standard for managing AI/ML workloads at scale, scheduling workloads such as AI/ML training jobs and HPC simulations becomes more important than ever. However, scheduling becomes challenging because the default Kubernetes scheduler schedules Pods individually, which can result in some Pods being scheduled while others remain pending due to insufficient resources. This partial scheduling can lead to deadlocks and inefficient use of cluster resources.

Gang scheduling graduates to Beta in Kubernetes v1.37, improving upon native support for gang scheduling through the Workload API and PodGroup concept. This feature implements an all-or-nothing scheduling strategy, ensuring that a defined group of Pods is scheduled only when the cluster has sufficient resources to accommodate the entire group. The Beta graduation of this enhancement also introduces workload-aware preemption to avoid premature preemptions that do not help a workload make progress, along with PodGroup queueing to better coordinate competing workloads.

Importantly, it addresses livelock scenarios that can occur when multiple workloads are being scheduled simultaneously by the kube-scheduler, preventing them from repeatedly interfering with one another without making progress.

This work was done as part of KEP #4671 led by SIG Scheduling.

Native histogram support for Kubernetes metrics

Kubernetes exposes hundreds of histogram metrics in Prometheus format across its control plane components, which are essential to monitor cluster health and debug performance issues. However, classical Prometheus histograms relied on static, pre-defined buckets that forced a compromise between data accuracy and memory usage. To mitigate this, Prometheus introduced native histograms that use dynamic exponential bucket boundaries instead of fixed boundaries, providing significant storage efficiency, improved query performance, and finer-grained visibility into distributions while maintaining full backward compatibility with existing monitoring infrastructure.

Kubernetes v1.37 graduates native histogram support for Kubernetes metrics to Beta. Building on the Alpha implementation, which introduced the NativeHistograms feature gate, the Beta phase improves the implementation and rollout experience. When enabled, Kubernetes components expose histograms in both classic and native formats when the requested scrape protocol supports Native Histograms, (specifically PrometheusProto), allowing existing dashboards and alerts to continue working while users migrate at their own pace. The implementation also refactored histograms created in init() functions to use lazy initialization, ensuring native histogram options are correctly applied after feature gates are parsed. These changes provide a more reliable implementation while retaining safe rollout and rollback through the feature gate or Prometheus-side configuration for Prometheus 3.x users.

This work was done as part of KEP #5808 led by SIG Instrumentation.

WAS: Features graduating to Beta

Workload-aware preemption

Kubernetes traditionally performs preemption at the Pod level, which can be inefficient for workloads made up of multiple tightly coupled Pods. In Kubernetes v1.37, workload-aware preemption graduates to Beta, allowing the scheduler to consider a PodGroup when making preemption decisions. This helps the scheduler consider the workload as a whole when preempting lower priority workloads, reducing cases where individual Pods are disrupted without providing enough capacity for the workload to make progress.

This work was done as part of KEP #5710 led by SIG Scheduling.

DRA: ResourceClaim support for workloads

Dynamic Resource Allocation (DRA) allows Pods to request specialized resources through ResourceClaims. In Kubernetes v1.37, DRA ResourceClaims support for workloads graduates to Beta, allowing Workload and PodGroup APIs to associate ResourceClaims and ResourceClaimTemplates with the groups of Pods. This allows ResourceClaims to be shared across a workload rather than reserved individually for each Pod, while ResourceClaimTemplates can create claims for PodGroups automatically.

This work was done as part of KEP #5729 led by SIG Scheduling.

cAdvisor-less, CRI-full container and Pod stats

The kubelet has historically obtained container and Pod statistics from cAdvisor, while the Container Runtime Interface (CRI) exposes statistics of its own. Having two sources for the same metrics makes it harder to tell where a particular value came from.

In Kubernetes v1.37, the cAdvisor-less, CRI-full Container and Pod Stats enhancement graduates to Beta. The enhancement expands the CRI to provide the container and pod statistics needed by Kubernetes, allowing the kubelet to get these metrics directly from the container runtime instead of relying on cAdvisor for them.

This moves container and pod metrics toward a single source of truth, while reducing duplicated metric collection and simplifying how the kubelet gathers and exposes these statistics.

This feature is Beta in v1.37 but off by default; enable the PodAndContainerStatsFromCRI feature gate to try it.

This work was done as part of KEP #2371 led by SIG Node.

Support memory QoS with cgroups v2

Kubernetes is improving its quality of service mechanisms to cover memory protection and isolation for Kubernetes workloads. For nodes running Linux, the memory QoS feature uses memory requests and limits to configure cgroup controls that can protect requested memory from reclamation and throttle memory usage before workloads reach their hard limits. This can help reduce the impact of memory pressure on memory-sensitive workloads and improve node stability.

In Kubernetes v1.37, memory QoS support is graduating to Beta. The feature uses cgroups v2 memory controls such as memory.min, memory.low and memory.high to provide different levels of memory protection and throttling. For example, memory requests can be used to protect memory from reclamation, while memory.high can be used to throttle workloads that exceed their configured threshold.

The MemoryQoS feature gate is enabled by default in v1.37. Cluster operators can control memory protection through the kubelet's memoryReservationPolicy setting and configure memory throttling with memoryThrottlingFactor. The defaults are designed to avoid introducing unexpected memory throttling for existing workloads when upgrading to v1.37, while allowing operators to opt into the additional memory protection capabilities.

This work was done as part of KEP #2570 led by SIG Node.

Pod-level resource managers

In Kubernetes v1.37, Pod-level resource managers graduate to Beta behind the PodLevelResourceManagers feature gate,

which stays disabled by default. Enabling it allows the topology, CPU, and memory resource

managers to use the resources defined for an entire Pod when making

allocation and NUMA alignment decisions. This makes it possible to manage a Pod as a single resource unit while still supporting different resource requirements between the containers within it.

With pod-level resource management, a Pod can reserve a NUMA-aligned pool of CPU and memory based on its overall resource budget. Containers that require dedicated resources can receive exclusive portions of that pool, while other containers, such as sidecars or supporting workloads, can share the remaining resources. This is particularly useful for performance-sensitive workloads such as AI/ML and high-performance computing, where keeping resources close to each other on the same NUMA node can improve performance without requiring every container in the Pod to have dedicated resources.

The feature also supports a container scope, where containers can continue to receive independent NUMA-aligned allocations. This provides more flexibility for workloads that combine a performance-sensitive container with other containers that have different resource requirements.

This work was done as part of KEP #5526 led by SIG Node.

Watch-based route controller reconciliation

The route controller in the cloud-controller-manager library previously reconciled routes on a fixed interval, by default every 10 seconds. This could result in unnecessary requests to infrastructure providers, even when nothing had changed and could also delay route updates when a new Node is added.

Watch-based route controller reconciliation graduated to beta in Kubernetes v1.37. This release also adds observability for this work: the route controller's Alpha route_sync_total metric gains two labels, trigger (periodic or node_change) and outcome (changed, noop, or error), so operators can see whether periodic reconciliation is actually correcting route drift or just running as a no-op, and can track failed reconciles.

With watch-based route controller reconciliation, the route controller can reconcile routes from watch events instead of waiting for the next fixed interval: a reconciliation can start as soon as relevant Node changes occur, such as a Node being added or removed or when its addresses or assigned Pod CIDRs change. A less frequent periodic reconciliation still runs to catch outdated routes and keep the state consistent. This behavior sits behind the CloudControllerManagerWatchBasedRoutesReconciliation feature gate and is disabled by default, so the transition has not changed default behavior.

This reduces unnecessary requests to infrastructure providers while allowing routes for newly added Nodes to be reconciled sooner. The change does not alter the route reconciliation logic itself; it changes when reconciliation is triggered.

This work was done as part of KEP #5237 led by SIG Cloud Provider.

Storage capacity scoring of Nodes

The VolumeBinding scheduler plugin has always been able to score nodes for statically bound PVs based on free capacity, but that scoring never extended to dynamic provisioning.

When a CSI driver provisions a new volume on demand, the scheduler had no way to prefer a node with more or less free space.

This was a gap for local storage, as an admin might want pods landing on the node with the most free capacity to leave room for a later volume expansion or on the node with the least (but still sufficient) free capacity to bin-pack workloads and cut down on the number of nodes a cloud cluster needs to run.

Kubernetes v1.37 graduates storage capacity scoring for dynamic provisioning to Beta behind the StorageCapacityScoring feature gate. First introduced in Alpha in v1.33, this feature consolidates (and deprecates) the older VolumeCapacityPriority gate from KEP #1845. When enabled, the VolumeBinding plugin's Score extension point reads CSIStorageCapacity objects published by a driver's external provisioner sidecar and scores nodes for dynamic provisioning the same way it already does for static bindings. Admins choose the strategy via the Shape setting in VolumeBindingArgs, defaulting to "prefer the node with the maximum allocatable" so there is room for expansion later.

The feature depends solely on the StorageCapacityScoring gate: scoring for statically bound PVs runs as soon as it's enabled, independent of any CSI driver. A driver only needs StorageCapacity: true on its CSIDriver object so that its dynamically-provisioned volumes also get capacity-aware scoring. The feature is fully reversible, and disabling the gate stops all VolumeBinding capacity scoring - static and dynamic alike - without affecting already scheduled pods.

This work was done as part of KEP #4049 led by SIG Storage.

Integrate CSI volume attach limits with Cluster Autoscaler

Kubernetes v1.37 improves Cluster Autoscaler's integration with CSI volume attach limits, so that when it creates new nodes for pending Pods, Cluster Autoscaler can more accurately determine how many new nodes are required to attach all pending Pods that use CSI volumes. Cluster Autoscaler already had visibility into CSI volume attach limits for existing nodes, but not for the nodes it was about to create, which means it could undershoot scale-ups and leave volume-backed Pods pending even after adding capacity. The problem compounds on the scheduling side: the NodeVolumeLimits plugin treats a node with no published CSI driver info as having no limits at all, so a freshly created node that hasn't yet reported its CSINode object can get crowded with more volume-backed pods than it can actually mount, which is a race condition that, until now, cluster admins had no way to close.

Kubernetes v1.37 graduates CSI-aware autoscaling to Beta behind the VolumeLimitScaling feature gate, first introduced in Alpha in v1.35. Cluster autoscaler now runs its scale-up simulations against templated CSINode objects, so it correctly accounts for attach limits whether it's scaling an existing node group or scaling one from zero. On the scheduler side, admins can opt in per CSIDriver, via a new PreventPodSchedulingIfMissing field, to block pod placement on nodes that haven't reported their driver yet, with dedicated CSIDriverMissingOnNode and CSINodeMissing errors making those scheduling failures easier to debug. The Beta phase adds e2e coverage for scale-down behavior and CSI opt-in scenarios, and updates the failed_scale_ups_total and scaled_up_nodes_total metrics to include CSI driver information. Both the autoscaler and scheduler changes stay strictly opt-in: disabling the feature gate restores today's default of unlimited pod placement on nodes without CSINode data, so distros and admins running autoscalers that aren't CSI-aware yet (e.g. Karpenter) aren't forced into the new behavior.

This work was done as part of KEP #5030 led by SIG Autoscaling.

Report last used time on a PVC

PersistentVolumeClaims tend to outlive the workloads that created them. When an app gets deleted or migrated, its PVC remains behind, consuming storage and increasing costs.

Kubernetes v1.37 graduates PVC "last used" tracking to Beta behind the PersistentVolumeClaimUnusedSinceTime feature gate, which shipped disabled by default in Alpha (v1.36) and is now enabled by default. The feature adds a new Unused condition to PersistentVolumeClaimStatus, managed by the existing PVC protection controller: Status=True (Reason=NoPodsUsingPVC) once the last non-terminal Pod referencing the PVC goes away, and back to Status=False (Reason=PodUsingPVC) as soon as a Pod starts referencing it again. The condition's lastTransitionTime doubles as an "unused since" timestamp, so admins can query how long a PVC has actually been idle without Kubernetes tracking which Pod used it last or making any deletion decision itself; that's left entirely to the admin. One thing worth noting is that the timestamp reflects when the controller observed no Pods using the PVC, not the exact moment the volume unmounted at the infrastructure level, so the reported idle time may run a little short of the true figure but should never overstate it.

This work was done as part of KEP #5541 led by SIG Storage.

etcd RangeStream support

etcd's unary Range RPC builds an entire response in memory before sending it back, which becomes a problem at scale. On a large list, say kube-apiserver's watch cache warming up on a big cluster, the raw key-value slice, its serialized protobuf form, and the gRPC send buffer all have to coexist in memory at once, and the resulting spikes ripple through kube-apiserver too. Pagination doesn't really fix the underlying cost either because each paginated page still walks the entire B-tree index to recompute the total result count, turning what should be an O(limit) operation into an O(total_keys) one on every single page.

Kubernetes v1.37 ships etcd RangeStream support directly at Beta, behind the EtcdRangeStream feature gate (kube-apiserver only, on by default). This release adds a new server-streaming RangeStream RPC that reuses the existing RangeRequest but returns chunks instead of one buffered blob: the server paginates internally with adaptive chunk sizing (each chunk's target size adjusts based on MaxRequestBytes and the value sizes observed so far), pins a single MVCC revision so the merged stream stays snapshot-consistent, and derives the total key count from the running tally it builds while streaming, rather than a separate index walk. kube-apiserver's watch cache initialization is the primary consumer, and it now decodes each chunk into synthetic created events inline as they arrive instead of assembling the full list in memory first, with the same treatment applied to direct GetList calls when WatchList is disabled.

The feature requires etcd 3.7+; against older etcd, kube-apiserver detects the Unimplemented response and falls back to unary Range automatically, with zero behavior change. If the pinned revision gets compacted mid-stream, kube-apiserver treats it the same as any other watch cache init failure and retries, which is no worse than the compaction races a paginated List call can already hit today. Beta graduation criteria include a scalability test measuring large-list latency on a 5000-node cluster, and etcdctl get --stream ships alongside it for anyone who wants to poke at the new RPC directly.

This work was done as part of KEP #5966 led by SIG etcd.

Concurrent watch object decode

kube-apiserver decodes and transforms every watch event from etcd one at a time on a single goroutine, so one slow per-event transform, most notably a CRD conversion webhook call, blocks every event queued behind it. That's mostly a nuisance for built-in resources, but for a CRD whose served version differs from its stored version, converting a cold cache serially can take minutes. If that exceeds etcd's default 5-minute compaction interval, the revision the cache started reading from gets compacted before initialization finishes, the watch can't resume, and init just restarts and never converges for a large enough resource, with every client trying to list or watch it getting errors in the meantime.

The ConcurrentWatchObjectDecode gate has actually been in Beta, off by default, since v1.31, and Kubernetes v1.37 flips it on by default. Enabling it moves the decode/transform step onto a bounded pool of worker goroutines (10 by default, tuned from a sweep that showed gains flattening out around 8-12) instead of a single one, with a collector reassembling events back into their original order before delivery, so event ordering is preserved exactly. In benchmarks over 150k pods, concurrent decode alone cuts cache initialization about 40%, and about 55% combined with the new EtcdRangeStream feature also landing this release (see KEP 5966). The main tradeoff to watch is conversion webhook load. With the feature on, up to 10 conversions can now run concurrently against a webhook during cache init instead of one at a time. The total call volume is unchanged, only how many run at once, so this mainly matters for webhooks that cap their own concurrency below 10.

This work was done as part of KEP #6178 led by SIG API Machinery.

Stale controller mitigation

Every controller in kube-controller-manager works off a local cache built from watching the kube-apiserver, and that watch stream is only eventually consistent. A change can show up in milliseconds, or it can take seconds or even minutes under load. Today operators have no visibility into that lag and no way to tell a normal delay from a controller that's fallen dangerously out of sync, so a controller can keep reconciling against a view of the world that's already stale.

Stale controller mitigation has been Beta since v1.36, enabled by default per controller behind a StaleControllerConsistency<Controller> feature gate; Kubernetes v1.37 extends it to the HorizontalPodAutoscaler controller and adds the circuit-breaking variant and extra metrics described below. The core mechanism is a read your writes guarantee: client-go's ResourceEventHandlerFuncs gets a new BookmarkFunc callback so a controller can reliably track the resource version of objects it cares about, even through edge cases the existing add/update/delete callbacks miss. A controller records the resource version of its own writes and, on its next reconcile, skips and requeues until its informer cache has actually caught up to that write. The DaemonSet controller is a good example of this. It tracks DaemonSet → Pod resource versions so it won't re-reconcile against its own stale pod cache. A second, circuit-breaking variant targets latency-sensitive controllers like node-lifecycle, which can otherwise read a stale node lease from cache and wrongly decide it's expired; instead, it does a live GET on the disruptive decision and marks its cache "not ready" until it's caught up, rather than acting on a stale read. StaleControllerConsistency gates the mitigation itself (scoped initially to controllers KCM has flagged as high-scale), MonitorInformerStaleness is a separate, observation-only gate that polls the apiserver directly every 5 seconds purely to surface how far behind an informer's cache actually is, and AtomicFIFO / UnlockWhileProcessingFIFO are the underlying client-go workqueue plumbing the mitigation depends on. None of this changes default reconciler behavior; a paused-and-requeued controller can look stuck when it's really just waiting on its cache, and it rolls back cleanly since nothing it does is irreversible.

This work was done as part of KEP #5647 led by SIG API Machinery.

Manifest-based admission control config

In Kubernetes, admission control is responsible for enforcing policies on resources before they are accepted by the API server. However, admission webhooks and policies configured through the Kubernetes API are dependent on the API server and etcd during cluster startup and cannot protect the admission configuration resources themselves. This creates a gap during cluster bootstrap and allows critical admission policies to be modified or removed by users with sufficient privileged access.

In Kubernetes v1.37, manifest-based admission control configuration graduates to Beta, allowing admission webhooks and CEL-based

policies to be loaded from manifest files on disk and enforced from API server startup. Because the configuration is managed independently of the Kubernetes API, it can also protect API-based admission resources from modification. Manifest files are watched for changes and valid updates are reloaded automatically, while invalid updates leave the previously loaded configuration in place.

This work was done as part of KEP #5793 led by SIG API Machinery

Improved handling for undecryptable resources

Kubernetes stores resources in etcd, where encryption at rest can be used to protect sensitive data. However, when encrypted resources can no longer be decrypted, for example because the encryption key is unavailable, the API server cannot read or manage those resources normally. This can leave resources in the cluster that cannot be accessed through the Kubernetes API, requiring administrators to manually modify the underlying etcd data to recover them.

Kubernetes v1.37 includes Beta support for cluster administrators to identify and remove resources that cannot be decrypted by the API server.

Previously Alpha, and introduced in Kubernetes v1.32, this support allows problem API resources to be removed via

the Kubernetes API rather than directly manipulating the etcd file. This feature also provides safeguards for administrators to verify affected resources before deletion.

This work was done as part of KEP #3926 led by SIG Auth.

New features in Alpha

New Recreate strategy for StatefulSet rollouts

Kubernetes v1.37 introduces the Recreate strategy for StatefulSet rollouts. The StatefulSet API previously only offered two update strategies: OnDelete (manual) and RollingUpdate (automatic, default). Similar to Deployments, the Recreate update strategy deletes all of the StatefulSet's Pods before creating new Pods that reflect modifications made to a StatefulSet's .spec.template. Using this strategy requires the StatefulSetRecreateStrategy feature gate to be enabled.

This work was done as part of KEP #3541 led by SIG Apps.

DRA: Alpha features to look out for

DRA: Node allocatable resource request

Kubernetes v1.37 improves Alpha support for managing node resources such as CPU, memory, and huge pages through DRA. It unifies standard and DRA resource accounting, helping prevent the same node capacity from being counted twice.

This update introduces distinct API fields for mapping (for devices directly modeling core resources, like CPU/memory DRA drivers) and overhead (like auxiliary host memory for accelerator devices). The kubelet now enforces these allocations across pod and container cgroups, integrates them with Memory QoS, OOM score calculations, and in-place pod resizing.

This work was done as part of KEP #5517, led by SIG Scheduling with participation from SIG Node.

DRA: derived attributes

Kubernetes v1.37 introduces Alpha support for derived attributes in DRA. Workloads can use CEL expressions to create virtual attributes from device information and use them when selecting related devices.

This makes it easier to co-locate devices such as GPUs and network interfaces, even when their drivers use different attribute names or formats. For example, a workload can derive a shared NUMA identifier and use it to select devices with matching topology.

This work was done as part of KEP #6080, led by SIG Scheduling with participation from SIG Network.

DRA: device compatibility groups

DRA can be used to manage devices that support different partitioning or virtualization schemes. However, some of these configurations cannot be used together on the same physical device, such as MIG and vGPU on a GPU. Previously, these incompatibilities could only be detected during device preparation, after the scheduler had already made its decision.

In Kubernetes v1.37, DRA adds device compatibility groups, allowing resource drivers to describe which devices can be allocated together. The scheduler can use this information when making allocation decisions, preventing incomplete devices from being assigned together and avoiding Pod startup failures caused by incomplete device configurations.

This work was done as part of KEP #5963, led by SIG Scheduling.

Scheduler preemption for in-place Pod resize

Kubernetes v1.37 introduces scheduler preemption for in-place pod resize, behind the (opt-in, Alpha) InPlacePodVerticalScalingSchedulerPreemption feature gate. This change addresses an important feature gap that remained after the core in-place Pod vertical scaling feature graduated to Stable: if a running pod requested additional resources that exceeded the node's available capacity, kubelet marked the request as Deferred, leaving the pod waiting until sufficient resources became available on the node. With this enhancement, the Kubernetes control plane can actively free up capacity on a fully-utilized node and preempt lower-priority workloads, enabling the pending in-place resizes of critical, higher-priority applications to succeed.

This work was done as part of KEP #5836 led by SIG Scheduling.

Dynamic resize of memory-backed volumes

Also building upon in-place Pod vertical scaling, the Alpha in-place scaling for memory backed volumes feature extends the pod

/resize subresource, which previously only enabled dynamic CPU and memory adjustments without restarting containers, to support updating the sizeLimit of memory-backed (medium: Memory) emptyDir volumes on running pods. When a volume's sizeLimit is explicitly adjusted via the /resize subresource, Kubelet dynamically updates the underlying tmpfs mount without container disruption while safely preventing out-of-memory errors or false-positive eviction triggers. This is particularly useful for stateful and memory-intensive workloads that rely on in-memory ephemeral storage, allowing them to dynamically scale storage limits alongside container memory capacity without incurring Pod restarts or application downtime.

This is an opt-in, off-by-default Alpha feature. To try it out, enable the

InPlacePodVerticalScalingMemoryBackedVolumes feature gate.

This work was done as part of KEP #6030 led by SIG Node and SIG Storage.

Specialized lifecycle management for Nodes

Several Kubernetes components need to understand a Node's lifecycle state, and today each one infers it from a different mix of Node readiness, taints, Pod state, labels, annotations, and provider APIs. This enhancement introduces well-known lifecycle conditions on Nodes, giving administrators a single Kubernetes-owned place to publish lifecycle state that core controllers and ecosystem tooling can consume. These new Node Conditions are: DrainInProgress, Drained, MaintenancePlanned, MaintenanceInProgress, and GracefulNodeShutdownInProgress

This work was done as part of KEP #5683 led by SIG Node.

WAS: Alpha features to look out for

CompositePodGroup API

While previous releases introduced support for gang scheduling of workloads with a flat structure, modern AI/ML workloads are complex and have more sophisticated scheduling requirements. In Kubernetes v1.37, the new Alpha CompositePodGroup API allows Kubernetes to describe complex workloads as a hierarchy of groups instead of a flat set of Pods. This enables multi-level gang scheduling, workload-aware preemption and topology-aware scheduling.

This work was done as part of KEP #6012 led by SIG Scheduling

Workload Aware Scheduling Controller APIs

As an Alpha feature Kubernetes v1.37 provides a common framework for integrating workload controllers (such as JobSet, TrainJob, LWS, and RayJob, along with core workloads such as Job) with Workload-aware Scheduling (WAS).

The framework provides reusable scheduling.k8s.io API primitives, such as topology constraints and disruption policies, along with shared libraries that handle the creation of scheduling resources. This allows controllers to expose WAS features natively within their APIs in a consistent way without implementing the same scheduling logic separately.

This work was done as part of KEP #6089 led by SIG Scheduling.

Integrate workload APIs with the Job controller

Initially introduced in Kubernetes v1.36 with limited functionality, this feature builds on top of the Workload Aware Scheduling Controller APIs adding a new user-facing spec.scheduling field to the batch/v1 Job in Kubernetes v1.37, allowing users to explicitly configure scheduling policies, topology constraints, disruption modes, and resource claims. If spec.scheduling is omitted, the Job defaults to Basic scheduling, preserving existing behavior while still creating a Basic Workload/PodGroup for workload-aware scheduling, without enforcing a minCount gate. Users can explicitly opt into Gang scheduling, where minCount defaults to the Job's parallelism, and the controller uses the shared workloadbuilder library to translate the scheduling configuration into the corresponding Workload and PodGroup objects instead of implementing custom translation logic.

This work was done as part of KEP #5547 led by SIG Scheduling.

localhost NodePort userspace proxy for nftables

Kubernetes v1.37 adds an opt-in userspace proxy to the nftables kube-proxy backend, allowing NodePort services to be accessed through localhost over IPv4 and IPv6. This closes a gap between the nftables and iptables backends, as nftables could not previously serve localhost NodePorts.

The proxy is enabled when localhost or a loopback address is included in the kube-proxy --nodeport-addresses configuration. This can be useful for workloads such as local container registries that rely on localhost:<NodePort> connections. The existing behavior of the iptables and ipvs backends is unchanged.

This work was done as part of KEP #6032 led by SIG Network.

Other notable changes

maxUnavailable for StatefulSets back on by default

The maxUnavailable field for StatefulSets has been re-enabled by default in Kubernetes v1.37 (after a bug was observed in v1.36).

The bug occurred where a faulty initial StatefulSet revision created a Pod that never became ready, and with MaxUnavailableStatefulSet enabled, the StatefulSet controller failed to update that Pod to the newer, corrected revision. When the bug triggered, the affected Pod could end up stuck in a CrashLoopBackOff state indefinitely (see kubernetes#137409).

Improved nftables performance

kube-proxy now uses the kernel's netlink interface for nftables rule operations, bypassing the nft command-line tool. This makes kube-proxy more efficient when inspecting and managing its nftables rules, improving rule-management performance.

Context handling and contextual logging in client-go

Support for context propagation and contextual logging in client-go is complete, with the exception of a small number of authentication plugin log calls that still rely on the global klog logger because the underlying APIs do not support context passing.

Graduations, deprecations, and removals in v1.37

Graduations to Stable

This lists all the features that graduated to Stable (also known as general availability). For a full list of updates including new features and graduations from Alpha to Beta, see the release notes.

This release includes a total of 16 enhancements promoted to Stable:

Deprecations, removals and community updates

As Kubernetes develops and matures, features may be deprecated, removed, or replaced with better ones for the project's overall health. See the Kubernetes deprecation and removal policy for more details on this process. Many of these deprecations and removals were announced in the Deprecations and Removals blog

Deprecation of kube-dns

CoreDNS has been the default cluster DNS add-on since Kubernetes v1.13, and kube-dns has not kept pace since then; features like EndpointSlices and dual-stack Services aren't available in it.

Kubernetes has already retired the kube-dns subproject and has split node-local-dns out into its own repository, where it continues to be maintained and works with CoreDNS. It is expected that no new packages will be built for kube-dns after v1.40.

If you still run kube-dns, start planning to migrate your clusters to CoreDNS.

Deprecating kube-proxy's support for ipvs mode

kube-proxy support for ipvs mode was introduced in v1.8 to resolve iptables performance bottlenecks. However, since the kernel ipvs API alone cannot fully implement Kubernetes Services, ipvs mode continues to use iptables underneath (KEP-3866, "The ipvs mode of kube-proxy will not save us").

Clusters running kube-proxy in ipvs mode (or mode: ipvs in KubeProxyConfiguration) now log a deprecation warning on startup. The deprecation timeline looks like this:

kubectl -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}' | grep 'mode:'

To understand the rationale behind this deprecation, see KEP #5495.

kubectl: kubectl run --filename/-f to be deprecated

The --filename (or -f) flag for kubectl run is being deprecated as the generated pod is always built purely from CLI arguments like NAME and --image.

See kubernetes/kubernetes#138671 for the original issue and discussion.

kubelet: static Pods can no longer reference Secrets or ConfigMaps

Static Pods were never meant to read API resources directly, since they aren't created through the API server - but a bug let them reference Secrets or ConfigMaps via fields like configMapRef or secretRef. That bug is now fixed: as of v1.37 these references are strictly prohibited, and the PreventStaticPodAPIReferences feature gate that previously let you opt out of the restriction has been removed.

See kubernetes/kubernetes#140226 for the original issue and discussion.

Ongoing major change: Future removal of cgroup v1 support

As modern Linux distributions and container runtimes use cgroup v2 as the default, support for the legacy cgroup v1 is officially being phased out. Since the v1.35 release, the failCgroupV1 setting has defaulted to true. Consequently, the kubelet will fail to initialize on any nodes that still rely on cgroup v1 unless an explicit configuration override is applied.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failCgroupV1: false # temporary override

Using this override should be considered a short-term fix. Advanced resource management capabilities, such as memory QoS and in-place scaling for memory-backed volumes, work only on cgroups v2. While the override remains available in Kubernetes v1.37, users are encouraged to migrate to cgroups v2, as support for cgroups v1 is planned to be removed in a future release.

To learn more about this deprecation, refer to KEP #5573.

Release notes

Check out the full details of the Kubernetes v1.37 release in our release notes.

Availability

Kubernetes v1.37 is available for download from the Kubernetes download page or direct from on GitHub.

To get started with Kubernetes, check out these tutorials or run local Kubernetes clusters using minikube. You can also easily install v1.37 using kubeadm.

Release team

Kubernetes is only possible with the support, commitment, and hard work of its community. Each release team is made up of dedicated community volunteers who work together to build the many pieces that make up the Kubernetes releases you rely on.

This requires the specialized skills of people from all corners of our community, from the code itself to its documentation and project management.

We would like to thank the entire release team for the hours spent hard at work to deliver the Kubernetes v1.37 release to our community.

The Release Team's membership ranges from first-time shadows to returning team leads with experience forged over several release cycles.

A very special thanks goes out to our release lead, Dipesh Rawat, for supporting us through a successful release cycle, advocating for us, making sure that we could all contribute in the best way possible, and challenging us to improve the release process.

Project velocity

The CNCF K8s DevStats project aggregates a number of interesting data points related to the velocity of Kubernetes and various sub-projects.

This includes everything from individual contributions to the number of companies that are contributing and is an illustration of the depth and breadth of effort that goes into evolving this ecosystem.

In the v1.37 release cycle, which ran for 15 weeks from May 18th, 2026, to August 26th, 2026, contributions to Kubernetes reached a maximum of 212 different companies and 1,754 individuals.

Source for this data:

By contribution we mean when someone makes a commit, code review, comment, creates an issue or PR, reviews a PR (including blogs and documentation), or comments on issues and PRs.

If you are interested in contributing, check out our getting started page.

Event updates

Explore the upcoming KubeCons worldwide:

Explore the upcoming Kubernetes Community Days (KCDs) taking place for the rest of 2026:

September 2026

October 2026

November 2026

December 2026

You can find the latest event details at the CNCF Events Page.

Upcoming release webinar

Join members of the Kubernetes v1.37 Release Team on Wednesday, September 23rd, 2026 at 4:00 PM (UTC) to learn about the release highlights of this release. For more information and registration, visit the event page on the CNCF Online Programs site.

Get involved

The simplest way to get involved with Kubernetes is by joining one of the many Special Interest Groups (SIGs) that align with your interests.

If you don't know where to start, join our monthly New Contributor Orientations where we teach the community how the project is structured, and we'll guide you on how to make your first contribution to the project.

26 Aug 2026 12:00am GMT

11 Aug 2026

feedKubernetes Blog

How to Pretty-Print Your Kubernetes YAML as KYAML and Why You'd Want To

YAML has been the standard way to write Kubernetes manifests for years. Every example, tutorial, and configuration file you come across is written in it. The problem isn't that YAML is a bad format. It's that YAML gives you a lot of choices, and not all of them are equally good for writing Kubernetes manifests. Some features make files harder to read, some are easy to misuse and others can lead to surprising behavior.

The interesting part is that Kubernetes doesn't actually need most of those features. It only relies on a small subset of YAML. This led to a simple question: if Kubernetes only needs a small part of YAML, why not standardize on that part and avoid the rest? Instead of introducing a new configuration language, SIG CLI introduced KYAML, a stricter, more consistent way to write YAML.

What is KYAML?

KYAML is a strict subset (or "dialect") of standard YAML, designed to be parseable by the existing ecosystem without any changes, as proposed in KEP 5295. It does not introduce a new format or a new parser. It just narrows the scope of choices you make when writing YAML, so everyone ends up making the same ones.

Think of it less like a new language and more like an agreed-upon style. Everything valid in KYAML is valid YAML.

How KYAML solves it

Standard YAML has a few well-known traps and JSON is not without its own.

Whitespace sensitivity. Indentation defines structure in YAML, which means a wrongly indented file can remain syntactically valid while representing a different object than intended. This gets especially painful with templating tools like Helm, where you are manipulating indentation from outside the YAML context.

Silent type coercion. String quoting is optional in YAML, which sounds convenient until it is not. Some values that look like strings get coerced into other types without warning. The classic example is the "Norway Bug".

country: NO

In standard YAML, NO is parsed as a boolean false, not the string "NO" and it has caught more than a few people off guard.

JSON is not the answer either. It lacks comment support, is strict about trailing commas, and requires every key to be quoted, none of which makes for a good config writing experience.

KYAML addresses all of these by making structure and types explicit:

YAML calls this flow style, as opposed to the conventional block style most people use. KYAML sits halfway between JSON and YAML, more explicit than default YAML, friendlier than JSON.

Here is the same Pod manifest written in both formats for comparison.

Standard YAML

apiVersion: v1
kind: Pod
metadata:
 name: my-pod
 labels:
 app: demo
spec:
 containers:
 - name: nginx
 image: nginx:1.20

KYAML

---
{
 apiVersion: "v1",
 kind: "Pod",
 metadata: {
 name: "my-pod",
 labels: {
 app: "demo",
 },
 },
 spec: {
 containers: [{
 name: "nginx",
 image: "nginx:1.20",
 }],
 },
}

Notice the double-quoted string values, the braces around every mapping, the brackets around the list and the trailing commas. The additional syntax makes the document structure explicit instead of relying on indentation.

How to pretty print YAML as KYAML

There are different ways to get KYAML output.

Option 1: kubectl -o kyaml

Since Kubernetes 1.34, kubectl supports KYAML as a native output format.

# Kubernetes 1.35+ (beta; feature enabled by default, still requires -o kyaml CLI param)
kubectl get deployment my-app -o kyaml

# Kubernetes 1.34 (alpha, opt-in)
export KUBECTL_KYAML=true
kubectl get deployment my-app -o kyaml

To save the output to a file:

kubectl get deployment my-app -o kyaml > my-app.yaml

There are currently no plans to make KYAML the default output format. If you prefer using KYAML by default, you can configure your preferred default with kuberc. For more details, see the kuberc documentation.

# Kubernetes 1.36+
kubectl kuberc set --section defaults --command get --option output=kyaml

# Kubernetes 1.33-1.35 (alpha prefix still required)
kubectl alpha kuberc set --section defaults --command get --option output=kyaml

Option 2: Kubernetes' yamlfmt

sigs.k8s.io/yaml ships a yamlfmt tool that can convert files to KYAML.

Install via Go:

go install sigs.k8s.io/yaml/yamlfmt@latest

Running it against a file prints the KYAML version to stdout. It also accepts a directory, in which case it converts and prints every file in that directory. So you'll need to redirect the output to a file (or files) if you want the conversion to stick.

yamlfmt -o=kyaml my-deployment.yaml

It can also show you a diff instead of a full conversion:

yamlfmt -o=kyaml -d my-deployment.yaml

Option 3: Google's yamlfmt

For converting existing files, Google's yamlfmt added a dedicated kyaml formatter in v0.21.0.

Install via Go, or grab a binary from the releases page:

go install github.com/google/yamlfmt/cmd/yamlfmt@latest

It is also available as a pre-commit hook and as a Docker image for CI pipelines.

Add a .yamlfmt config to your project root:

formatter:
 type: kyaml

Preview the output without modifying your file:

yamlfmt -dry my-deployment.yaml

then apply:

yamlfmt my-deployment.yaml

To convert an entire directory:

yamlfmt ./k8s/

The kyaml formatter takes no additional configuration and does not share options with the default formatter so mixing them will cause an error.

For more on the available modes and flags, check the command usage docs.

Is KYAML worth adopting?

Every valid KYAML file is a valid YAML file. So whatever you write in KYAML, your existing tools, your kubectl, your CI pipelines, none of them need to change. You can even pass KYAML as input to any version of kubectl, not just 1.34+, because at the end of the day it is just YAML.

KYAML is not strictly necessary. You can keep writing block-style YAML and things will work. But it is a deliberate choice to make your configs less error-prone and more consistent especially across a team or a larger repo.

It is less of a migration and more of a better habit.

11 Aug 2026 6:00pm GMT

03 Aug 2026

feedKubernetes Blog

Gateway API v1.6: TCPRoute and UDPRoute Graduate to Standard

Gateway API logo

The Kubernetes SIG Network community is thrilled to share the release of Gateway API v1.6.0, which was released on June 30th of this year!

Gateway API has become the standard for modern, role-oriented, and expressive service networking in Kubernetes. In previous releases, Gateway API established a production-grade foundation for HTTP and TLS layer 7 traffic. With version 1.6.0, Gateway API takes a major step forward by expanding standard layer 4 protocol routing and introducing cleaner API boundaries for experimental innovation.

Here is a quick summary of what's new in Gateway API v1.6.0:

Let's dive into the details!

TCPRoute and UDPRoute graduate to Standard

Leads: Nick Young, Ricardo Katz and Zac Nixon

Until now, Gateway API only offered a stable routing model for HTTP and TLS traffic. Workloads that speak a raw protocol over TCP or UDP - databases, DNS, VoIP, gaming, IoT telemetry - had no portable way to plug into a Gateway. Users either fell back to a plain Kubernetes Service, or to an implementation-specific CRD that doesn't travel between Gateway controllers.

TCPRoute and UDPRoute close that gap: they route traffic to backends based on protocol and port alone, no L7 awareness required. With this release, both have graduated from the Experimental channel to Standard, and moved to the v1 API version. The v1alpha2 version of each was deprecated as of the v1.6 release, and will be removed in a future release.

How it works

A Gateway needs a listener that allows TCPRoute attachment:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: example-gateway
spec:
 gatewayClassName: example-gateway-class
 listeners:
 - name: foo
 protocol: TCP
 port: 12345
 allowedRoutes:
 kinds:
 - kind: TCPRoute

A TCPRoute then attaches to that listener and forwards traffic to a backend:

apiVersion: gateway.networking.k8s.io/v1
kind: TCPRoute
metadata:
 name: tcp-app
spec:
 parentRefs:
 - name: example-gateway
 sectionName: foo
 rules:
 - backendRefs:
 - name: my-foo-service
 port: 6000

Traffic arriving on the Gateway's port 12345 is proxied to the endpoints of my-foo-service on port 6000. Omitting sectionName and port from parentRefs attaches the route to every TCP listener on the Gateway instead of a single one.

UDPRoute follows the same pattern; swap the listener protocol and the route kind:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: example-gateway
spec:
 gatewayClassName: example-gateway-class
 listeners:
 - name: foo
 protocol: UDP
 port: 12345
 allowedRoutes:
 kinds:
 - kind: UDPRoute
---
apiVersion: gateway.networking.k8s.io/v1
kind: UDPRoute
metadata:
 name: udp-app
spec:
 parentRefs:
 - name: example-gateway
 sectionName: foo
 rules:
 - backendRefs:
 - name: my-foo-service
 port: 6000

XBackend arrives in Experimental

Leads: Keith Mattix II

Gateway API v1.6 introduces the new XBackend resource, which is a general-purpose decorator for Service (and other backend types) within Gateway API.

The Service resource is an amazing, stable, and flexible object, but that comes with some costs: The flexibility creates a lot of edge cases that Gateway API needs to handle, and the stability makes it impossible to add new concepts to Service.

The XBackend resource builds on the ideas in the upstream EndpointSelector KEP, to add a Gateway API-native object that still targets the backend app, while allowing the community to extend it to handle use cases that are difficult or dangerous to handle with Service.

The first version of XBackend includes support for ExternalHostname destinations, which are ruled out from Service support in Gateway API because of the possibility of confused deputy attacks.

For XBackend, this support is an Extended/Optional feature, allowing implementations and users to opt in once they understand the security tradeoffs.

This support is very useful for egress use cases (which are most commonly used for cluster-hosted agentic workloads), which the community is also working towards formalizing in GEPs about Gateways for Egress (work in progress, stay tuned!)

The XBackend API is experimental and its behavior can change, do not assume it is ready for production

An example of a Gateway with an ExternalName backend that can be used for egress to a cloud AI API is as follows:


# Gateway-level TLS remains authoritative for incoming connections
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
spec:
 listeners:
 - name: https
 protocol: HTTPS
 tls:
 certificateRefs:
 - name: gateway-cert
---
# Backend resource for external destination
apiVersion: gateway.networking.x-k8s.io/v1alpha1
kind: XBackend
metadata:
 name: ai-provider-api
 namespace: ai-apps
spec:
 type: ExternalHostname
 externalHostname:
 hostname: api.ai-provider.com

---
# HTTPRoute referencing XBackend
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
spec:
 rules:
 - backendRefs:
 - name: ai-provider-api
 kind: XBackend
 group: gateway.networking.x-k8s.io

The community is also working on moving Session Persistence config from XBackendTrafficPolicy into XBackend, along with other use cases like retries, TLS origination and similar config that is useful to be able to configure per-application rather than per-Route.

Experimental resources move off the standard API group

Previously, experimental resources shared the same API group as standard ones - gateway.networking.k8s.io - distinguished only by a v1alpha2-style version. TCPRoute and UDPRoute were the last resources to graduate under that scheme.

Going forward, new experimental resources are defined in a separate group, gateway.networking.x-k8s.io, and the names of their API types get an X prefix - for example XBackend and XMesh. When one of these graduates to Standard, it's renamed into the gateway.networking.k8s.io group and drops the X prefix, the same way XMesh is expected to become Mesh.

This separation makes the experimental/standard boundary explicit at the API group level, rather than relying on version strings alone.

What's next & getting involved

The graduation of TCPRoute and UDPRoute to Standard marks an essential milestone in making Gateway API a complete, universal ingress and mesh networking API for Kubernetes workloads across layer 4 and layer 7 protocols.

Try it out

You can start using Gateway API v1.6.0 today with your favorite Gateway controller implementation:

Gateway API relies on an extensive conformance test suite to ensure consistent, portable behavior across all implementations. Here is a list of the implementations that are conforment with v1.6 on the day we published the article:

Get involved

Gateway API is an open, community-driven project built under Kubernetes SIG Network. We welcome contributions, feedback, and participation from everyone!

Acknowledgments

A huge thank you to all the contributors, reviewers, maintainers, and implementation authors whose hard work made Gateway API v1.6.0 possible!

03 Aug 2026 4:00pm GMT

31 Jul 2026

feedKubernetes Blog

Kubernetes v1.37 Sneak Peek

As we get closer to the release date for Kubernetes v1.37, the project develops and matures, features may be deprecated, removed, or replaced with better ones for the project's overall health. This blog outlines some of the planned changes for the Kubernetes v1.37 release that the release team feels you should be aware of for the continued maintenance of your Kubernetes environment and keeping up to date with the latest changes. The information below reflects the current status of the v1.37 release and may change before the actual release date.

Deprecations and removals for Kubernetes v1.37

Kubectl: kubectl run --filename/-f to be deprecated

The --filename (or -f) flag for kubectl run is being deprecated as the generated pod is always built purely from CLI arguments like NAME and --image.

See kubernetes/kubernetes#138671 for the original issue and discussion.

Kubelet: Static Pods can no longer reference Secrets or ConfigMaps

Static Pods were never meant to read API resources directly, since they aren't created through the API server - but a bug let them reference Secrets or ConfigMaps via fields like configMapRef or secretRef. That bug is now fixed: as of v1.37 these references are strictly prohibited, and the PreventStaticPodAPIReferences feature gate that previously let you opt out of the restriction has been removed.

See kubernetes/kubernetes#140226 for the original issue and discussion.

Deprecating kube-proxy's support for ipvs mode

kube-proxy support for ipvs mode was introduced in v1.8 to resolve iptables performance bottlenecks. However, since the kernel ipvs API alone cannot fully implement Kubernetes Services, ipvs mode continues to use iptables underneath (KEP-3866, "The ipvs mode of kube-proxy will not save us").

Clusters running kube-proxy in ipvs mode (or mode: ipvs in KubeProxyConfiguration) would now be logging a deprecation warning on startup. The deprecation timeline looks like this:

kubectl -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}' | grep 'mode:'

To understand the rationale behind this deprecation, see KEP-5495: Deprecate ipvs mode in kube-proxy.

Ongoing major changes

Future removal of cgroup v1 support

As modern Linux distributions and container runtimes use cgroup v2 as the default, support for the legacy cgroup v1 is officially being phased out. Since the v1.35 release, the failCgroupV1 setting has defaulted to true. Consequently, the kubelet will fail to initialize on any nodes that still rely on cgroup v1 unless an explicit configuration override is applied.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failCgroupV1: false # temporary override

Using this override should be considered a short-term fix. Advanced resource management capabilities, such as In-Place Pod Resizing and Tiered Memory Protection, depend entirely on cgroup v2. While the override remains available in Kubernetes v1.37, users are encouraged to migrate to cgroup v2, as support for cgroup v1 is planned to be removed in a future release.

To learn more about this deprecation, refer to KEP-5573: Remove cgroup v1 support.

Breaking changes in Kubernetes v1.37

SELinux volume relabeling ("SELinuxMount") graduates to GA

SELinuxMount is expected to reach GA and be enabled by default in v1.37. Volumes would then be mounted with -o context=<label> (the mount option default) instead of being recursively relabeled, but only when the volume's CSI driver opts in via a CSIDriver that sets .spec seLinuxMount: true.

Because a single mount can only hold one SELinux context, pods with different SELinux labels sharing a volume on the same node (which previously coexisted under recursive relabeling) may now fail to start. To retain the previous recursive behavior for a specific workload, set seLinuxChangePolicy: Recursive in the Pod spec.

Clusters without SELinux enabled see no effect at all. To learn more, check SELinux Volume Label Changes goes GA (and likely implications in v1.37)

Featured enhancements of Kubernetes v1.37

Metrics API goes GA

The metrics.k8s.io API is expected to graduate to Stable (GA) in Kubernetes v1.37 after spending nearly nine years in Beta. The API provides a standard way to retrieve CPU and memory usage for pods and nodes, powering widely used Kubernetes features such as the Horizontal Pod Autoscaler (HPA) and commands like kubectl top.

This graduation recognizes the API's stability and widespread adoption, with no functional changes expected. Both v1 and v1beta1 will remain usable during the transition, enabling developers to adopt the stable API at their own pace without breaking existing workflows.

To learn more about this enhancement, refer to KEP-5207: metrics.k8s.io API definition.

Kubelet in UserNS a.k.a. Rootless Mode

Traditionally, Kubernetes node components such as the kubelet run with root privileges on the host. While necessary for many deployments, this also means that a vulnerability in one of these components could potentially have a greater impact on the underlying system.

With Kubernetes v1.37, kubelet in User Namespace (Rootless Mode) is expected to graduate to Beta. This enhancement allows Kubernetes node components to run inside a Linux user namespace as an unprivileged user on the host while still behaving as root within the namespace. By reducing the need for host-level root privileges, it adds an extra layer of isolation and helps limit the impact of potential vulnerabilities affecting node components.

To learn more about this enhancement, refer to KEP-2033: Kubelet in UserNS(aka Rootless Mode).

Volume health monitor

Historically, Kubernetes has lacked an API for CSI drivers to report storage failures, which become evident only through failed mounts or hung I/O. Since remediation controllers had nothing machine-readable to act upon, the only way to figure out the root cause behind this failure was to cross-reference Kubernetes objects alongside external vendor dashboards.

In Kubernetes v1.37, this KEP resets graduation to Alpha after an initial implementation in v1.21 and introduces four new CSI RPCs. The controller plugin reports the health of storage volumes using ControllerListVolumeHealth (lists unhealthy volumes) and ControllerGetVolumeHealth (checks a specific volume). A controller-side health monitor polls these CSI controllers and stores the results in PersistentVolumeClaim.status.healthStatus.

On the node side, the kubelet calls NodeGetVolumeHealth to obtain the health of individual volumes on that node and records it in Pod.status.volumeHealth, while NodeGetStorageHealth reports the health of the drivers registered to a node in CSINode.status.storageHealth.

The error vocabulary is kept simple, extensible, and machine-parsable (Inaccessible, Degraded, etc.), with further driver-specific elaboration available via reason and message. Finally, the controller-side and node-side reports are kept independent and are hence displayed separately, providing a more holistic view of storage health to consumers.

To learn more about this enhancement, refer to KEP-1432: Volume Health Monitor.

Want to know more?

New features and deprecations are also announced in the Kubernetes release notes. We will formally announce what's new in Kubernetes v1.37 as part of the CHANGELOG for that release.

Kubernetes v1.37 release is planned for Wednesday, August 26th, 2026. Stay tuned for updates!

You can see the announcements of changes in the release notes for:

Get involved

The simplest way to get involved with Kubernetes is by joining one of the many Special Interest Groups (SIGs) that align with your interests.

If you don't know where to start, join our monthly New Contributor Orientations where we teach the community how the project is structured, and we'll guide you on how to make your first contribution to the project.

31 Jul 2026 4:00pm GMT

29 Jul 2026

feedKubernetes Blog

How the controller-runtime Cache Actually Works, and Why Your Controller Does Not Crash the API Server

This article has been revised since it was first published, to correct several significant technical inaccuracies in the original text.

Kubernetes has long been the default platform for distributed workloads, and writing your own controller for it is now a matter of a few hours. The common path - Golang, using kubebuilder on top of controller-runtime - gives you a project scaffold, types, and a reconciler. For typical scenarios that is more than enough. But as soon as load grows or the controller starts behaving in ways you did not expect, a whole class of edge cases shows up. Most of them trace back to the same root cause: a fuzzy mental model of how controller-runtime works inside. If you write Kubernetes controllers in Go, this article should help you build a coherent picture and avoid expensive surprises in production.

This article walks through the internals of controller-runtime and, along the way, shows which architectural decisions are baked into Kubernetes itself. The starting point is how controllers actually read objects from the Kubernetes API.

A common misconception goes like this: r.Get() inside Reconcile queries kube-apiserver directly; r.List() returns a fresh, live view of the world; and after r.Update() you can re-read the object and immediately see the new state. In practice the model is the opposite: controller-runtime operates against a local copy of the data populated through list + watch. Reads inside a reconciler cost almost nothing and do not load the control plane even at hundreds of calls per second - but the price of this design is that a controller can quietly consume gigabytes of memory, perform hidden O(n) scans, and regularly trip over stale reads.

This post is aimed at engineers who already write controllers in Go with controller-runtime but want to consolidate the pieces into a single mental model rather than carry around a bag of isolated observations. The focus is the practical impact on production clusters: memory, network traffic, read consistency, and reconciler behavior.

TL;DR

If you take only one idea from this article, take this:

r.Get() and r.List() inside a reconciler typically do not read from the API server. They read from a local in-memory cache, which the manager warms up with list and then keeps current through watch.

Almost every other property of the system follows from that one fact:

The rest of the article unpacks why this is so and how the model is wired underneath.

A bit of context: what a reconciliation loop is

To avoid arguments about terminology, start with the basic model.

A controller in Kubernetes lives inside a reconciliation loop: it continuously compares the desired state of an object with the actual state and tries to bring one in line with the other. The idea is described in the original architectural notes on Kubernetes. In practice it looks like this:

What matters here is not that the controller "does something" - it is where it learns about changes from and where it reads state from. That is exactly where the cache comes in.

On a live cluster, the easiest way to see this in action is:

kubectl get pods --watch

In watch mode, kubectl subscribes to the same event stream that controllers consume. You create or delete a Pod and you see not a single "final" object but a chain of states: the scheduler assigns a node, the kubelet updates status, other controllers contribute their changes. Kubernetes controllers do not poll continuously - they consume an event stream and maintain a local state that is kept current.

For a visual walkthrough, see Reconciliation loop pattern in visual representation, a talk that shows how the reconciliation loop plays out on a real Pod and the states it passes through.

Why the cache exists in controller-runtime at all

Imagine the simplest possible controller:

func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
 var pod corev1.Pod
 if err := r.Get(ctx, req.NamespacedName, &pod); err != nil {
 return ctrl.Result{}, err
 }
 // ... meaningful logic ...
}

Looks straightforward. But what happens when you call r.Get? Does it fire an HTTP request at the API server? If it did, picture the scene: a dozen controllers, each issuing a get and a list per reconcile, with hundreds of reconciles per second. The API server and etcd would be writing each other farewell letters within minutes.

To prevent that, Kubernetes was built around a watch model rather than polling from the very beginning. The standard mechanism works like this: a client takes a snapshot of the slice of the world it cares about once, then subscribes to a stream of changes and keeps a local copy current. This is the list + watch pattern, and there is no "what is in the world right now?" loop anywhere in it.

This idea has lived in client-go since the very first controllers in kube-controller-manager. controller-runtime wraps it in a friendly framework so that you do not have to glue the Reflector, the delta queue, and the Indexer together yourself (more on those below).

So when people talk about "the controller-runtime cache", they are not talking about a clever optimization. They are describing the foundation of the entire model: you read from memory, you write to the API server, and you receive feedback through a watch.

The rest of this article walks through how each piece is wired up.

Glossary

A few terms collected up front, so you do not have to jump back and forth later. Skim or skip if any of them are already familiar.

With those in hand, you can dive in.

Anatomy: what lives under the cache package

If you peek into sigs.k8s.io/controller-runtime/pkg/cache, you will see that it is a thin wrapper over k8s.io/client-go/tools/cache. The same primitives that power the rest of Kubernetes live underneath:

At a glance the pipeline looks like this:

A vertical flow chart with seven labeled boxes connected by arrows, from API server at the top to Reconcile at the bottom.

Pipeline diagram: API server to Reflector to the delta queue to Indexer to Event handlers

Now walk through each link.

Reflector and resourceVersion

Within the cache, the Reflector is the only component that talks to the API server. It has exactly two jobs: fetch the initial snapshot at startup, then keep a watch open from there on. (Writes and APIReader reads bypass the cache entirely and reach the API server on their own - more on those later.)

This is where the resourceVersion earns its keep. Along with the objects, the API server reports the version at which the snapshot was produced. The Reflector then says to the API server, "open a watch from version X", and receives a stream of events for everything that happened after that version. That is the basis of consistency: there is no gap between the snapshot and the stream, because the stream resumes exactly where the snapshot ended.

That snapshot no longer arrives as a separate list call by default. Current versions use a streaming list instead: the Reflector opens the watch with sendInitialEvents=true, and the API server begins the stream with synthetic ADDED events for the whole current state before switching to live changes. One request instead of two, and a plain list as the fallback. The pattern is unchanged - snapshot, then stream - which is why this article keeps saying list + watch, the way the Kubernetes documentation does.

If the connection drops, the Reflector reconnects with the last known resourceVersion. If the API server replies with 410 Gone ("that version is no longer in the history, you are too far behind"), the Reflector fetches a fresh snapshot and starts over. This is called a relist, and it does not happen on a schedule - only in those failure scenarios.

The delta queue

This piece is worth pausing on, and it is also the piece that changed most recently.

Historically the buffer between the Reflector and the rest of the informer was DeltaFIFO, which held deltas in a map keyed by namespace/name: deltas for one object accumulated in a slot, Pop() handed back the whole slice for that key at once, and a dedupDeltas helper collapsed consecutive Deleted entries. If you have read about informer internals before, that is probably the picture you are carrying.

That is no longer the default. Shared informers now use RealFIFO, and since client-go 1.36 DeltaFIFO cannot be switched back on at all. Which version you compile against is what decides this, not the version of the cluster you point at. The new queue is deliberately simpler - a flat, strictly ordered slice of deltas:

type RealFIFO struct {
 // ...
 items []Delta
}

Its own documentation states the design goal plainly: every notification from the Reflector is passed, in order, through Pop. Which means:

  1. Order is preserved globally, not just per object. Deltas come out in exactly the sequence they arrived.
  2. One Pop, one delta. There is no per-key slot and no slice - Pop takes items[0]. (There is also a PopBatch for processing several deltas in one pass, on by default since client-go 1.35, but it is a batching optimization, not a merge: each delta is still delivered.)
  3. No deduplication whatsoever. RealFIFO has no dedupDeltas equivalent. Nothing is collapsed - including consecutive deletes, and including intermediate states.

So the worked example gets simpler than it used to be. Suppose three events for default/my-deploy arrive in quick succession:

  1. Added - the Deployment is created (say, with spec.replicas=1).
  2. Updated - somebody bumps spec.replicas to 2.
  3. Updated - and immediately to 3.

All three are appended to the queue and popped one at a time, and the informer dispatches them in order: first OnAdd, then two OnUpdate calls (the intermediate 1→2, then the final 2→3). The event handler runs three times, no shortcuts.

The store is written before the handlers are notified, and handler delivery is asynchronous - the informer writes the indexer, then hands the notification to a per-subscriber buffer that the subscriber's own goroutine drains later. So your handler never sees an indexer lagging behind its own event, but it can see one that has moved well past it. Handling the 1→2 update, a Get from the cache can legitimately return 3 - or NotFound, if the object has since been deleted. Never treat the object in the store as "the state at the time of my event".

Deduplication does exist - but it lives one layer up, in the controller's workqueue, and with RealFIFO that is now the only place it happens. The mechanic is straightforward: for each delta, the controller's event handler extracts the namespace/name key from the object and enqueues it. Re-inserting the same key silently coalesces with the existing entry; the workqueue does not care about the object itself.

A concrete picture: you create a Pod. Within a second or two a flurry of Updated deltas arrives - the scheduler assigns a node, the kubelet sets Pending, then ContainerCreating, Running, Ready. Five deltas in a row, and the event handler fires on every one of them - but throughout this window the workqueue holds a single entry with the key default/my-pod. By the time Reconcile pops it, the cache already holds the final state, and Reconcile runs once.

So you get two layers with cleanly separated responsibilities:

If you keep that two-layer picture in your head, it becomes clear why a flood of events against a single object barely affects controller throughput - the workqueue absorbs them.

Indexer: the local copy of the cluster

The Indexer, backed by a ThreadSafeStore, is the local copy of the cluster. Underneath it is a plain map[string]interface{} keyed by namespace/name, plus a single sync.RWMutex, plus a dictionary of registered indexes (covered in their own section below).

An uncontended r.Get is cheap: a map lookup followed by a DeepCopy of the object. The part of that structure that matters most at scale, though, is not the map - it is the one sync.RWMutex, which guards the store and every index at once. Readers hold it for shared access, the informer needs it exclusively to write, so the two genuinely compete: a List holds the read lock while it walks every object of that kind, and the next store write waits behind that walk. This was a real bottleneck in kube-controller-manager at scale (kubernetes#130767); recent client-go releases hold the write lock for much less time.

SharedIndexInformer and subscriptions

A SharedIndexInformer fuses the Reflector, the delta queue, and the Indexer together and exposes two interfaces to the rest of the world:

"Outside" here means your controllers. When a controller registers Watches(...), under the hood it asks the informer: "add a handler that, on every change, enqueues the key into my workqueue". The controller's workers then pop keys one at a time and call your Reconcile(ctx, ctrl.Request{NamespacedName: ...}).

The keyword in the name is Shared. The manager creates one informer per GVK, and every controller, webhook, and event source within that manager subscribes to it:

A single Pod informer at the top with three arrows fanning out to two controllers and a webhook, all inside a ctrl.Manager box.

Shared informer diagram: a single list / watch per GVK, feeding multiple subscribers

In other words: an informer is the thing that subscribed to Pods once, holds them locally, and serves every interested party in the process. From the API server's perspective, that is one snapshot and one watch per GVK, regardless of how many reconcilers live inside your process.

What happens at startup and on the very first r.Get

Step by step, here is what happens between the moment the manager starts and the first r.Get inside your reconciler:

  1. The manager's mgr.Start(ctx) brings up every registered informer.
  2. For each GVK, the Reflector fetches a full snapshot: every object of that type that falls within your scope.
  3. The snapshot is loaded into the informer's store, registered indexes are rebuilt, and the informer is marked as synced.
  4. The same stream then continues as an ordinary watch from the resourceVersion the snapshot synced to.
  5. Only then does the controller start invoking Reconcile - specifically, once every source it owns reports synced, which includes its event handlers having processed the initial snapshot. Until that point, workers do not drain the workqueue, even if events have already started piling up.

So "the reconciler is running but the cache is still empty" is not a state you can observe - the warm-up happens before the first Reconcile. (The one exception: a Get for a type nothing registered a watch for starts a new informer on the spot, and blocks until it is warm.)

What happens during the first r.Get? Suppose your reconciler contains:

var obj appsv1.Deployment
err := r.Get(ctx, req.NamespacedName, &obj)

Under the hood it boils down to roughly this:

item, exists, err := indexer.GetByKey("default/my-deploy")
if !exists {
 return apierrors.NewNotFound(...)
}
// DeepCopy into obj

No HTTP, no TLS, no protobuf serialization, no etcd. A map lookup plus a deep copy of the object, and no I/O at all.

To repeat, because it matters: even the very first Get for a registered type reads from a fully warmed-up, fully indexed snapshot. There is no "first time slow, then fast".

Note: This applies specifically to mgr.GetClient(). If for some reason you need to read objects before mgr.Start() (for example, during initialization), use mgr.GetAPIReader(), which goes straight to the API server. The regular client does not hand you an empty result at that point - it fails fast with ErrCacheNotStarted. More on this later.

Client ≠ Cache: read from memory, write to the API server

Another point that often gets lost. client.Client in controller-runtime is a composite object:

This is not a hack - it is a deliberate design choice:

It is worth dwelling on "should be exact". This is where resourceVersion shows up again.

When you read an object from the cache, you do not get its current state in etcd - you get the state as the Reflector last observed it. That state carries a resourceVersion. You then mutate the object and call r.Update(ctx, &obj). The request goes to the API server right now, and the API server checks:

This is optimistic concurrency control. No real locks are taken; everybody writes in parallel; but only one of the racing Update calls wins - the one that arrives with the current version. Everyone else gets a 409 and is expected to re-read and try again.

Why does this matter for the cache? If you naively send a PUT with "your" resourceVersion from the cache and somebody has updated the object since you read it, you will get 409. That is not a bug. It is exactly the protection the system is supposed to give you. Writing without the resourceVersion check (via Patch without an optimistic lock, or via Server-Side Apply) is also possible, but that is a separate conversation.

The "write → visibility" cycle now looks like this:

A vertical diagram showing how a write travels from user code through the API server and back into the controller's cache via a watch event.

Write visibility diagram: client.Update to API server to watch event to cache

Between "you executed Update" and "the cache reflects the new state" there is a window - usually milliseconds, but with no guaranteed upper bound. Inside that window, an r.Get for the same object returns the previous version. The next section is essentially a list of mistakes that grow out of that window.

Common mistakes that everyone makes

Mistake 1: expecting read-after-write

A familiar pattern:

obj.Spec.Replicas = ptr.To(int32(5))
if err := r.Update(ctx, &obj); err != nil {
 return ctrl.Result{}, err
}

// re-read and confirm it is now 5
var fresh appsv1.Deployment
_ = r.Get(ctx, key, &fresh)
fmt.Println(*fresh.Spec.Replicas) // surprise: 3

This is not a controller-runtime bug. It is a property of an eventually consistent system: the cache catches up asynchronously, through the watch.

The right pattern is to never rely on instant freshness. Reconcile must be idempotent and must always look at the current state. If it does not match the desired state, the next reconcile fixes it. You do not need to "wait 100ms" or "re-trigger". You need to write the logic so that one or two extra invocations break nothing.

If a stale read is a genuine correctness problem for you, a live read does not fix it - a concurrent write can be mid-commit anyway. See the controller-runtime FAQ for patterns that do.

Mistake 2: DeepCopy and who owns the memory

To make sense of this, a quick word on event mechanics inside a controller. When you register a source via Watches(...), two layers sit between the indexer and your Reconcile:

Here is the critical part. Predicates and handlers receive the same objects that live in the informer's shared store. The same *corev1.Pod is seen by every controller subscribed to Pods.

Because Go has no immutable structs, nothing prevents you from doing pod.Labels["foo"] = "bar" directly inside a handler - and that Pod is the one in the store. Raw client-go listers have always worked this way; ThreadSafeStore's own documentation puts it bluntly: you must not modify anything returned by Get or List as it will break the indexing feature. Patch a status "for convenience" in a handler and you break the world view of an unrelated controller next door.

The cache-backed client from controller-runtime shields you from that on the read path: Get and List deep-copy by default, and have since its earliest releases. You can opt out with UnsafeDisableDeepCopy, which is named that way on purpose. The event path is not shielded - there is no DeepCopy anywhere between the informer and your predicate. The simple rule:

A concrete review heuristic: if predicate.Funcs{UpdateFunc: ...} or handler.EnqueueRequestsFromMapFunc(...) contains expressions like e.ObjectNew.SetLabels(...) or obj.Status.X = Y, stop and ask whether a DeepCopy is missing before that mutation.

Mistake 3: resync is not relist

An informer has a resync period (cache.Options.SyncPeriod, 10 hours by default in controller-runtime), and many people read it as meaning: rebuild the cache from the API server every n hours, fetching every resource once again.

It does not. A resync does not perform a list. It re-emits everything currently in the indexer back through the delta queue, and the informer dispatches an update per object, calling OnUpdate(old, old) for each one. This is for controllers that manage state outside the Kubernetes API (a cloud provider resource, for example): out-of-band changes produce no watch event, and a periodic resync is the only way to notice them. It generates no traffic to the API server.

One caveat before you rely on resync as a safety net: because both sides of the synthetic update are the same object, predicates that compare old and new - such as GenerationChangedPredicate - will drop it.

A real relist happens only in two cases: when the watch died with 410 Gone, and when you explicitly recreate the informer.

Mistake 4: do not confuse RequeueAfter with a timer

A small note that often saves time. Sometimes you want to wait inside a reconciler - "we just called the provider's API; if it is not ready yet, retry in a minute". The temptation is to spin up time.Sleep or your own goroutine.

Resist it. controller-runtime already provides a built-in mechanism:

return ctrl.Result{RequeueAfter: 30 * time.Second}, nil

The controller puts your req back into the workqueue with a delayed trigger 30 seconds out. If a real event for the same object arrives within that window, the reconcile fires immediately, without waiting for the timer (the key is deduplicated in the queue). This is both cheaper and more correct than a hand-rolled timer: you do not hold a worker, and you do not risk missing a real event.

cache + index = almost SQL

Now you get to what is, arguably, the most useful capability of the cache - and the one most controllers leave unused.

By default, a List from the cache looks like this:

var pods corev1.PodList
_ = r.List(ctx, &pods)
for _, p := range pods.Items {
 if p.Spec.NodeName == "node-1" {
 // do something
 }
}

It works - until the cluster has 50,000 Pods and reconciles run hundreds of times per second. Then the loop turns slow: every trigger walks all 50,000 Pods under the store's read lock, then deep-copies each one after the lock is released, doing O(n) work per reconcile, and it is the walk, not the copying, that blocks writers into the store.

The Indexer in client-go can do much better. You declare up front which field you want to index on:

// Index by spec.nodeName for Pods
if err := mgr.GetFieldIndexer().IndexField(
 ctx,
 &corev1.Pod{},
 "spec.nodeName",
 func(obj client.Object) []string {
 pod := obj.(*corev1.Pod)
 if pod.Spec.NodeName == "" {
 return nil
 }
 return []string{pod.Spec.NodeName}
 },
); err != nil {
 return err
}

Two things about that call are worth making explicit, because the tidy example hides them behind a convention.

The index name is arbitrary. That second argument, "spec.nodeName", is only a string key the index is registered under. controller-runtime does not parse it as JSONPath and does not check it against the object's schema - you could write "by-node" or "xyzzy" and it would behave identically. The only rule is that the exact same string comes back in MatchingFields at query time. Naming the index after the field it happens to read is a readability convention, nothing more.

But remember: this only works for reads served from the cache.

The indexed value is computed, not read. The function returns whatever strings you build; they need not be the verbatim contents of any single field. You can lowercase a value, join several fields into one composite key, bucket a timestamp (the time-bucket trick below does exactly this), or emit a string that appears nowhere in the object literally. Whatever the function returns becomes a key in the inverted dictionary, and a MatchingFields lookup for that exact key is what finds the objects again. The only constraint is that the value has to be derivable from the object you are indexing.

What is an inverted index? The term comes from search engines. Normally you have documents and each document has a list of words in it. "Inverted" means the relationship is flipped: a dictionary in which the key is a word and the value is the list of documents that contain it. Same idea here: the key is the value of a field (for example, node-1), and the value is the list of object keys whose field has that value:

map["node-1"] = {"default/pod-a", "kube-system/pod-b", ...}
map["node-2"] = {"default/pod-c", ...}

What the indexer does:

And now you can write:

var pods corev1.PodList
_ = r.List(ctx, &pods,
 client.MatchingFields{"spec.nodeName": "node-1"},
)

This is not "fetch the full list, then filter". It is a lookup in the inverted index → a ready set of keys → a fetch of the corresponding objects. A different code path entirely.

The comparison to SQL is more accurate than it might look at first:

SQL controller-runtime
CREATE INDEX idx_node ON pods(node_name) IndexField(&Pod{}, "spec.nodeName", fn)
SELECT * FROM pods WHERE node_name = 'node-1' List(&pods, MatchingFields{"spec.nodeName": "node-1"})
SELECT * FROM obj WHERE owner_uid = $1 List(&list, MatchingFields{"metadata.ownerReferences.uid": uid}) (requires an IndexField for that field)

Note the last row: MatchingFields does not make magic out of thin air. For every field you want to look up via MatchingFields you need a corresponding IndexField registered during manager setup. Without one, controller-runtime rejects the query and returns an error.

A few things worth keeping in mind:

Note: An index is built at registration time and is populated as part of the initial snapshot. By the time the first Reconcile runs, List with MatchingFields already works - the index is not built lazily. (Get never consults a field index; it is a direct lookup by store key.)

Selective cache: do not pull the whole cluster into your controller

By default, an informer pulls every object of its type from every namespace. For Pod, Secret, ConfigMap, and Event in a large cluster, that is a multi-gigabyte surprise delivered on the first list at startup.

It hurts especially with:

In controller-runtime, caching policy lives in cache.Options, passed when constructing the manager:

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
 Cache: cache.Options{
 ByObject: map[client.Object]cache.ByObject{
 // Cache Secrets only from your own namespace, and only by label
 &corev1.Secret{}: {
 Namespaces: map[string]cache.Config{
 "my-controller": {},
 },
 Label: labels.SelectorFromSet(labels.Set{
 "app.kubernetes.io/managed-by": "my-controller",
 }),
 },
 // Cache all Pods, but trim noise on the way into the store
 &corev1.Pod{}: {
 Transform: func(obj any) (any, error) {
 pod := obj.(*corev1.Pod)
 pod.ManagedFields = nil
 return pod, nil
 },
 },
 },
 },
})

A subtle point: this is a manager-level setting and it affects every controller in the process that reads the corresponding type. If you narrow the cache for Secrets to a single namespace and another controller in the same binary needs all secrets in the cluster, that controller will not see them. Before you tighten the scope, audit who else is reading the type.

A short tour of the options:

Caveat: A selector limits what is cached, not what exists. If an object does not match your selector, then as far as your controller is concerned, it does not exist in either Get or List. This bites people: somebody mislabels a single Secret and then spends half a day figuring out why their controller "cannot see it".

Metadata-only: when spec and data are not needed

A separate pattern: you need to know that an object exists, but you do not need its spec or data. Typical examples: a controller that waits for a Secret with a particular name to appear but never reads it; one that counts PersistentVolume objects by the topology.kubernetes.io/zone label; one that reacts to ConfigMap objects in a namespace by name and does not care about contents.

Caveat: PartialObjectMetadata by definition gives you nothing from spec or status - only ObjectMeta. So you cannot filter through it on spec fields (such as a PersistentVolume's storageClassName or a Pod's nodeName); those fields do not exist in the local copy. Everything covered by metadata-only is labels, annotations, ownerReferences, finalizers, creationTimestamp, and the rest of metadata.

For this case there is PartialObjectMetadata:

var list metav1.PartialObjectMetadataList
// controller-runtime infers the list shape from the variable type.
list.SetGroupVersionKind(schema.GroupVersionKind{
 Group: "",
 Version: "v1",
 Kind: "Secret",
})
if err := r.List(ctx, &list, client.InNamespace("my-ns")); err != nil {
 return err
}

Under the hood this is a separate watch that asks the API server for metadata only. The store keeps such objects without Data, Spec, or Status - only ObjectMeta. For Secrets the memory difference can reach an order of magnitude.

APIReader: when the cache is not enough

mgr.GetAPIReader() returns a client.Reader that goes straight to the API server, around the cache. When you actually need it:

The price is a real network request, and it is easy to underestimate: on top of the round trip you pay to deserialize whatever comes back, which for a large collection is not cheap. So the trade is less obvious than it looks - reading from the API server is not automatically cheaper just because it avoids keeping objects in memory. Measure before you "optimize" a cached read into a live one. One thing to avoid outright: do not build "look in the cache, and if missing, fall back to the API" logic. That is exactly the split-brain pattern the cache is meant to protect you from.

Disabling the cache for a type entirely

If you do not need a local cache for a given type at all - say, the type is "fat", read rarely, and the list + watch overhead is not worth paying - you can tell the manager not to cache it. This is configured through client.Options.Cache.DisableFor:

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
 Client: client.Options{
 Cache: &client.CacheOptions{
 DisableFor: []client.Object{
 &corev1.Secret{},
 },
 },
 },
})

With this configuration, mgr.GetClient().Get(...) and List(...) for Secret go straight to the API server, bypassing the cache. No informer is started for that type, which means no list at startup and no permanent memory pressure from a store. That also means no events: nothing will trigger your controller when such an object changes. If you need those triggers, pair the direct reads with a metadata-only watch. This is a more radical alternative to APIReader: where APIReader is reached for ad hoc, individual requests, DisableFor turns the cache off for the type wholesale.

Real-world projects use this: external-secrets, for instance, has flags that disable caching for Secrets and ConfigMaps, trading memory for API traffic.

Aside: If you want to avoid a watch on the API server entirely, you can feed the controller events from a source of your own design, bypassing list + watch. In controller-runtime this is done with WatchesRawSource / source.Channel: you can wire the controller to events from any place - an internal queue, a kubelet, a custom watch. Niche, but a perfectly valid pattern when the API server should not be touched.

Good practices

A short checklist worth running through before you ship a controller into a live cluster:

Wrapping up

In one breath:

And the single sentence to remember: r.Get inside a reconciler reads from memory, not from the API server - not even the first time. The exceptions are the ones you opt into yourself: APIReader, Cache.DisableFor, and unstructured reads. Once that becomes a reflex, half the questions on controller code reviews answer themselves.

29 Jul 2026 6:00pm GMT

14 Jul 2026

feedKubernetes Blog

Building a Custom Metrics Exporter for Kubernetes

Kubernetes ships with built-in awareness of CPU and memory, but most real-world scaling decisions depend on signals that live entirely outside that narrow window: how many messages are waiting in a queue, how long the last batch job took, how many active WebSocket connections a pod is holding. When the built-in metrics are not enough, a metrics exporter bridges that gap.

This post walks through writing one from scratch, packaging it as a container, and wiring it into a cluster so that Prometheus - and ultimately the HorizontalPodAutoscaler - can consume it.

What a metrics exporter actually does

An exporter is a small HTTP server with a single responsibility: expose application state as text on a /metrics endpoint. Prometheus scrapes that endpoint on a regular interval, stores the time-series data, and makes it available for queries, alerts, and autoscaling rules.

In some cases you can instrument your application directly - embedding the Prometheus client library and exposing /metrics from within the same process - rather than running a separate exporter. A standalone exporter makes more sense when the data source is external to your application or when you do not control the application code.

The format Prometheus expects is plain text - one metric per line, with a name, optional labels, and a numeric value. Client libraries handle the serialization for you, so in practice you only need to decide what to measure and call the right function when that value changes.

Choosing what to measure

Before writing any code, it helps to decide what kind of signal you are dealing with. The Prometheus data model has three main types:

Once you know which type fits your signal, choose a name that follows the convention <namespace>_<name>_<unit> in snake_case. A job processor might expose worker_jobs_processed_total (counter), worker_queue_depth (gauge), and worker_job_duration_seconds (histogram). Clear names save everyone debugging time later.

Setting up the project

The Go Prometheus client is the most common choice for exporters in the Kubernetes ecosystem, largely because the same library powers most of the official Kubernetes components. Start by creating a module and pulling in the dependency:

mkdir my-exporter && cd my-exporter
go mod init example.com/my-exporter
go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promhttp

Registering metrics

Create main.go. The first thing to do is declare the metrics and register them with Prometheus's default registry. Registration tells the library that these metrics exist so they appear in the output even before the first observation is recorded:

package main

import (
 "log"
 "net/http"

 "github.com/prometheus/client_golang/prometheus"
 "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
 jobsProcessed = prometheus.NewCounterVec(
 prometheus.CounterOpts{
 Name: "worker_jobs_processed_total",
 Help: "Total number of jobs processed, partitioned by status.",
 },
 []string{"status"},
 )

 queueDepth = prometheus.NewGauge(prometheus.GaugeOpts{
 Name: "worker_queue_depth",
 Help: "Current number of jobs waiting in the queue.",
 })

 jobDuration = prometheus.NewHistogram(prometheus.HistogramOpts{
 Name: "worker_job_duration_seconds",
 Help: "Time spent processing a single job.",
 Buckets: prometheus.DefBuckets,
 })
)

func init() {
 prometheus.MustRegister(jobsProcessed, queueDepth, jobDuration)
}

prometheus.MustRegister panics on a duplicate registration, which makes misconfigurations obvious at startup rather than silently at runtime. If you are embedding this exporter inside a library that other packages will also instrument, prefer prometheus.Register and handle the error yourself.

Collecting real values

With the metrics registered, the next step is to keep them current. You can either continually update the data as the data change, or run your own internal refresh loop. The pattern below shows a polling loop - a goroutine that periodically reads from whatever data source your application owns and updates the registered metrics. Replace the simulated values with real calls to your database, internal API, or message broker:

import (
 "math/rand"
 "time"
)

func collectMetrics() {
 for {
 // Replace these with real reads from your application.
 depth := float64(rand.Intn(50))
 queueDepth.Set(depth)

 start := time.Now()
 time.Sleep(time.Duration(rand.Intn(200)) * time.Millisecond)
 jobDuration.Observe(time.Since(start).Seconds())
 jobsProcessed.WithLabelValues("success").Inc()

 time.Sleep(5 * time.Second)
 }
}

The polling interval (here five seconds) should be shorter than Prometheus's scrape interval so that each scrape sees a fresh value. The default scrape interval in most cluster deployments is fifteen seconds, which gives you comfortable headroom.

Exposing the endpoint

Wire the collection loop and the HTTP handler together in main. A /healthz path alongside /metrics gives Kubernetes a liveness probe target without exposing metric data on the health route:

func main() {
 go collectMetrics()

 http.Handle("/metrics", promhttp.Handler())
 http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
 w.WriteHeader(http.StatusOK)
 })

 log.Println("Listening on :8080")
 if err := http.ListenAndServe(":8080", nil); err != nil {
 log.Fatalf("server error: %v", err)
 }
}

Verify the output locally before building the image:

go run .
curl http://localhost:8080/metrics | grep worker_

You should see three # HELP and # TYPE blocks followed by the current metric values. If those lines appear, the exporter is working correctly and is ready to be containerized.

Build a container image

A multi-stage build keeps the final image small and avoids shipping a Go toolchain to production. The first stage compiles a statically linked binary; the second stage copies only that binary into a minimal base. The example below uses Docker, but the same pattern works with any OCI-compatible build tool such as Buildah or Podman:

FROM golang:1.21-alpine AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /exporter .

FROM gcr.io/distroless/static:nonroot
COPY --from=builder /exporter /exporter
EXPOSE 8080
ENTRYPOINT ["/exporter"]

distroless/static:nonroot contains no shell, no package manager, and runs as a non-root user by default, which satisfies most cluster security policies without extra configuration.

Build and push the image, replacing <registry> with your own registry address:

docker build -t <registry>/my-exporter:v1.0.0 .
docker push <registry>/my-exporter:v1.0.0

(Note: Using a CI/CD pipeline to automate this is generally a better pattern than running these commands manually.)

Deploying to the cluster

Two manifests are enough to run the exporter: a Deployment that manages the pod lifecycle, and a Service that gives Prometheus a stable address to scrape. (You might prefer to have Prometheus scrape from every Pod; if that makes sense for your use case, then it's OK to configure instead).

The examples below use the monitoring namespace, which is a common convention when running Prometheus and related components together. Adjust the namespace to match your own cluster setup.

The Deployment sets conservative resource limits appropriate for a lightweight sidecar-style process, and uses the /healthz route for its liveness probe:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: my-exporter
 namespace: monitoring
 labels:
 app.kubernetes.io/name: my-exporter
spec:
 replicas: 1
 selector:
 matchLabels:
 app.kubernetes.io/name: my-exporter
 template:
 metadata:
 labels:
 app.kubernetes.io/name: my-exporter
 spec:
 containers:
 - name: exporter
 image: <registry>/my-exporter:v1.0.0
 ports:
 - name: metrics
 containerPort: 8080
 livenessProbe:
 httpGet:
 path: /healthz
 port: 8080
 initialDelaySeconds: 5
 periodSeconds: 10
 resources:
 requests:
 cpu: 50m
 memory: 32Mi
 limits:
 cpu: 100m
 memory: 64Mi

The Service names the port metrics, which the ServiceMonitor in the next section will reference by that name:

apiVersion: v1
kind: Service
metadata:
 name: my-exporter
 namespace: monitoring
 labels:
 app.kubernetes.io/name: my-exporter
spec:
 selector:
 app.kubernetes.io/name: my-exporter
 ports:
 - name: metrics
 port: 8080
 targetPort: metrics

Apply both:

kubectl apply -f deployment.yaml -f service.yaml

Telling Prometheus where to look

How you configure scraping depends on how Prometheus was installed.

Option 1: Prometheus Operator (ServiceMonitor)

If you installed Prometheus using the Prometheus Operator or the kube-prometheus-stack Helm chart, the operator must be running in your cluster before you create a ServiceMonitor. The release label must match the label selector configured on your Prometheus resource - kube-prometheus-stack is the default for a standard Helm install:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 name: my-exporter
 namespace: monitoring
 labels:
 release: kube-prometheus-stack
spec:
 selector:
 matchLabels:
 app.kubernetes.io/name: my-exporter
 endpoints:
 - port: metrics
 interval: 15s
 path: /metrics

Option 2: Annotation-based discovery

If your Prometheus uses annotation-based pod discovery instead, you will need a matching scrape_config rule in your Prometheus configuration - check with whoever manages your Prometheus installation to confirm it is in place.

You can add the following three annotations to the Pod template regardless of which scraping method you use. They are ignored by the Prometheus Operator but picked up automatically by annotation-based setups:

annotations:
 prometheus.io/scrape: "true"
 prometheus.io/port: "8080" # omit if not using annotation-based discovery
 prometheus.io/path: "/metrics" # omit if not using annotation-based discovery

If you are unsure which setup your cluster uses, the ServiceMonitor approach is more explicit and easier to debug.

Verifying the scrape

Port-forward to the Prometheus service and open the targets page to confirm the exporter has been discovered:

kubectl port-forward svc/prometheus-operated 9090 -n monitoring

Navigate to http://localhost:9090/targets. The my-exporter target should appear with state UP. If it shows DOWN, check that the ServiceMonitor's release label matches and that the pod is running:

kubectl get pods -n monitoring -l app.kubernetes.io/name=my-exporter
kubectl describe servicemonitor my-exporter -n monitoring

Once the target is healthy, run a quick query in the expression browser to confirm data is flowing:

rate(worker_jobs_processed_total{status="success"}[2m])

A non-zero result here means the full pipeline is working: your application is producing data, Prometheus is scraping it, and the time-series are stored and queryable.

What comes next

A working exporter is the foundation, not the destination. The natural next step is surfacing these metrics to the HorizontalPodAutoscaler so that your workload scales on the signals that actually drive load, not just CPU. That requires a metrics adapter - the Prometheus Adapter is the most widely deployed option - which registers your custom metrics with the Kubernetes Custom Metrics API. Once registered, any HorizontalPodAutoscaler in the cluster can reference worker_queue_depth or worker_jobs_processed_total directly in its metrics block.

For a walkthrough of that setup, see Autoscaling on multiple metrics and custom metrics. For a catalog of ready-made exporters covering databases, message brokers, and cloud services, the Prometheus exporters and integrations page is a good starting point.

14 Jul 2026 6:00pm GMT

13 Jul 2026

feedKubernetes Blog

Operating AI/ML Workloads on Kubernetes: A Headlamp Plugin for Kubeflow

Kubernetes has quietly become the default platform for AI and machine learning. Whether you run notebook servers for data scientists, schedule distributed training jobs, tune hyperparameters, or orchestrate multi-step ML pipelines, those workloads increasingly land on a Kubernetes cluster. Kubeflow is one of the most popular ways to assemble that stack, and it does so the Kubernetes-native way: every capability is exposed as a Custom Resource Definition (CRD).

That design is a gift to cluster operators, because it means ML workloads can be observed and managed with the same primitives as everything else in the cluster. But in practice the specialized ML dashboards that ship with these platforms hide the Kubernetes layer underneath. When a notebook is stuck or a training run fails, the operator is often left dropping back to kubectl to find out what actually happened at the Pod level.

This post introduces the Headlamp Kubeflow plugin, which closes that gap by surfacing Kubeflow's custom resources directly inside a general-purpose Kubernetes UI. It is a worked example of a pattern any CRD-heavy platform can follow: meet operators where they already work, and show them the cluster-level truth.

Headlamp itself is an extensible Kubernetes web UI maintained under Kubernetes SIG UI and licensed under Apache 2.0. It runs as a desktop app or in-cluster, and its plugin system lets anyone add first-class views for custom resources.

Why operators need a different view

Purpose-built ML dashboards help data scientists submit experiments, pipelines, and notebooks. Cluster operators and site reliability engineers (SREs) troubleshoot the Kubernetes resources underneath, and they ask different questions:

The Headlamp Kubeflow plugin helps answer these questions by reading directly from the Kubernetes API server. It shows Pod conditions, Kubernetes failure reasons, and resources across namespaces without requiring an intermediary ML service or database.

What the plugin covers

Kubeflow is modular, and teams often install only the components they need. The plugin discovers the Kubeflow API groups on a cluster and displays only the corresponding sections.

The plugin supports the following component families and API resources:

Kubeflow components and API resources supported by the Headlamp plugin
Component Purpose API resources
Notebooks Provides development environments such as Jupyter, VS Code, and RStudio Notebook, Profile, PodDefault
Pipelines Defines and tracks pipelines, versions, experiments, runs, and schedules Pipeline, PipelineVersion, Run, RecurringRun, Experiment
Katib Automates hyperparameter tuning and neural architecture search Experiment, Trial, Suggestion
Training Runs distributed training workloads such as PyTorch and TensorFlow jobs TrainJob, TrainingRuntime, ClusterTrainingRuntime
Spark Runs large-scale data processing with Apache Spark SparkApplication, ScheduledSparkApplication

What you can see

Inspect notebook Pods

The Notebook detail view shows Pod conditions and their reason and message fields. It also shows CPU, memory, and GPU requests and limits; volume mounts and their backing types, such as PersistentVolumeClaim, ConfigMap, Secret, or emptyDir; environment variables that reference Secret or ConfigMap objects; sidecar containers; and node tolerations. This view consolidates information that would otherwise require several kubectl describe commands.

Inspect hyperparameter tuning

The Katib views show the tuning algorithm, search space, every Trial with its live status, and the current best Trial with its metric values and parameter assignments. They also show the early-stopping configuration and the number of Trial resources that stopped early, so you can follow the search without leaving the cluster UI.

Inspect pipeline state without the backend database

The Pipelines views read Kubernetes API resources directly and do not query the Kubeflow Pipelines API service or backend database. You can inspect stored pipeline state even when that service is unavailable. The Pipeline detail view compares the latest and previous PipelineVersion specifications in a side-by-side YAML diff. Run views show state and duration, RecurringRun views show human-readable schedules, and the artifacts view aggregates pipelineRoot values from recent Run resources.

Map ML resources

The plugin registers a Headlamp map source that renders Notebook, Profile, PodDefault, Experiment, Pipeline, SparkApplication, and TrainJob resources as graph nodes. It draws edges between supported resources based on .metadata.ownerReferences. Headlamp also shows inline summaries for these resource types when you hover over them.

Try it

The Kubeflow plugin README explains installation and local-cluster setup, including a lightweight CRD-only path for evaluation. Because the plugin discovers installed API groups, you can use it with an existing modular Kubeflow installation or create an evaluation cluster with only the CRDs and sample resources.

Apply the pattern to other platforms

Kubeflow illustrates a broader pattern. Platforms often model domain-specific workflows with custom resources. Their dashboards focus on those workflows, while Kubernetes operators also need the state of the underlying API resources and Pods. A CRD-driven plugin in a general Kubernetes UI can expose that state without making operators switch between unrelated tools.

The plugin uses the Apache 2.0 license and is developed under Kubernetes SIG UI. To report a problem or contribute an improvement, use the Headlamp plugins repository's issue tracker or pull requests.

13 Jul 2026 8:00pm GMT

Kubernetes Dashboard to Headlamp: A Step-by-Step Guide

1. Before you start: know what is changing

Kubernetes Dashboard and Headlamp both show what is running in a cluster, but they work differently. When Headlamp runs on the desktop, it uses your existing kubeconfig to connect to one or more clusters and can be extended with plugins. When Headlamp runs inside a cluster, it uses a Kubernetes ServiceAccount to access the API and follow RBAC rules. Kubernetes Dashboard, in contrast, only runs in-cluster and always relies on service account tokens. Understanding these models early helps you choose the right setup and permissions.

1.1 How Kubernetes Dashboard works

Dashboard is a web app that runs inside your cluster.

It feels like this: a UI that lives with the cluster.

1.2 How Headlamp works

Headlamp acts more like a Kubernetes client with a UI.

Headlamp is a UI that follows your identity, not your cluster.

1.3 What stays the same

Many workflows will feel familiar:

1.4 What changes

A few things will feel different:

2. Pre-migration checklist

This checklist helps you avoid surprises during the switch. It makes sure Headlamp can use the same identity and permissions you already trust in Kubernetes. It also gives you a quick way to prove the migration worked before you turn off Dashboard.

2.1 Write down what you use today

List the basics:

This is your baseline.

2.2 Check that kubeconfig works

Headlamp uses kubeconfig, especially on desktop. Make sure yours works before you install anything.

Run:

kubectl config current-context

Then try:

kubectl get nodes

If you cannot list nodes, test in a namespace you can access:

kubectl get pods -n <namespace>

If these work, Headlamp can use the same identity and RBAC.

2.3 Pick a rollout plan

There is no need to rush. Most teams choose one of these:

Parallel rollout (recommended)

Cutover

Parallel rollout is safer for shared clusters.

2.4 Decide where Headlamp will run

You can use either option. Many teams use both.

Desktop

In-cluster

2.5 Note optional dependencies

These are common. You can handle them later.

3. Choose where Headlamp will run (desktop or in-cluster)

Headlamp can run on your desktop or inside a cluster. Both work well, but they fit different needs. Desktop is the fastest way to start because it uses your kubeconfig and does not run in the cluster. In-cluster is best when you need a shared URL and want the platform team to manage upgrades and access.

Option A: Desktop (user-managed)

Desktop Headlamp runs on each user's machine. It reads the same kubeconfig you use with kubectl. This keeps access tied to each user's identity and RBAC.

Why teams pick it

Option B: In-cluster (best for shared access)

In-cluster Headlamp is installed as a Kubernetes workload (often via Helm). This lets cluster admins manage it like other in-cluster apps.

4. Install Headlamp (desktop and in-cluster)

This section gets Headlamp running. Follow the path you chose in Section 3.

4.1 Desktop install (fastest way to start)

Install Headlamp on your machine. Then open it like any other app. Headlamp reads your kubeconfig and uses the same identity and RBAC rules as kubectl.

Windows

Install with WinGet:

winget install headlamp

Or with Chocolatey:

choco install headlamp

macOS

Install with Homebrew:

brew install --cask headlamp

Linux

Install with Flatpak (Flathub):

flatpak install flathub io.kinvolk.Headlamp

Quick check

  1. Launch Headlamp.
  2. Confirm you can see a cluster context.
  3. Open a namespace you can access and confirm you can list workloads. Headlamp will only show actions your RBAC allows.

4.2 In-cluster install (shared access)

Use this path when you want a shared UI that the platform team can manage. Headlamp supports in-cluster deployment with Helm or a YAML manifest.

Install with Helm

Add the repo and update:

helm repo add headlamp https://kubernetes-sigs.github.io/headlamp/
helm repo update

Create a namespace (example):

kubectl create namespace headlamp

Install the chart:

helm install headlamp headlamp/headlamp --namespace headlamp

Install with a YAML manifest (optional)

Headlamp also provides a YAML manifest you can apply and then adjust to your needs.

Check the install

Confirm the pod is running:

kubectl get pods -n headlamp

Confirm the service exists:

kubectl get svc -n headlamp

Access it (two common ways)

Quick test with port-forward

This is the fastest way to verify the service works:

kubectl port-forward -n headlamp svc/headlamp 8080:80

Then open: http://localhost:8080

Shared access with ingress

If you want a stable URL, expose the service through your ingress controller. Your exact ingress YAML depends on your setup. Headlamp's OIDC callback URL is your public URL plus /oidc-callback, so ingress and TLS settings matter.

4.3 Updating Headlamp

Updates depend on how you installed Headlamp. Package managers upgrade in place. DMG or EXE installs update by reinstalling the newer download.

macOS

If you installed with Homebrew, run:

brew upgrade headlamp

If you installed from a DMG, download the newest DMG and drag Headlamp into /Applications, replacing the old version. DMG installs do not auto upgrade.

Windows

If you installed with WinGet, run:

winget upgrade headlamp

If you installed with Chocolatey, run:

choco upgrade headlamp

If you installed from the EXE, download the newest installer and run it again. EXE installs do not auto upgrade.

Linux

If you installed with Flatpak, run:

flatpak update io.kinvolk.Headlamp

If you installed with AppImage, download the newest AppImage and run that file instead.

If you installed with a tarball, download the newest tarball, extract it, and run the new headlamp binary.

4.4 Notes for in-cluster access (keep it safe)

Treat an in-cluster UI like any other cluster-facing service. Use TLS, lock down who can reach it, and rely on Kubernetes auth and RBAC to control what users can do.

5. Authentication and RBAC

Headlamp uses the Kubernetes API the same way kubectl does. Your cluster still decides who can do what. Headlamp only shows actions your identity is allowed to take.

This section covers two setups: desktop and in-cluster.

5.1 Desktop: use kubeconfig

On desktop, Headlamp reads your kubeconfig and uses the same credentials you use with kubectl. There is no separate token login flow to manage.

Step 1: Confirm your kubeconfig works

Run:

kubectl config current-context

Then test access:

kubectl get nodes

If you cannot list nodes, test a namespace you can access:

kubectl get pods -n <namespace>

If these commands work, your kubeconfig and credentials are valid for Headlamp too.

Step 2: Point Headlamp at the right kubeconfig (if needed)

Headlamp can use the default kubeconfig path. It can also use a custom file path. You can set KUBECONFIG to choose a specific file.

Example:

KUBECONFIG=/path/to/config headlamp

You can also use more than one kubeconfig file at once. On Unix systems, separate paths with :. On Windows, separate paths with ;.

What to expect in the UI

Headlamp adapts to your RBAC permissions. If you do not have permission to edit or delete a resource, Headlamp will not offer those actions.

5.2 In-cluster: shared access needs a sign-in plan

In-cluster Headlamp is shared by many users. You need a clear plan for sign-in and access. Headlamp supports OpenID Connect (OIDC) for a "Sign in" flow.

You will usually choose one of these patterns:

A. Built-in OIDC (Headlamp)

To use OIDC, Headlamp needs:

Your OIDC provider must also allow Headlamp's callback URL. The callback is your Headlamp URL plus:

Example:

Ingress note

If Headlamp is behind an ingress or load balancer, make sure it forwards X-Forwarded-Proto. If it does not, Headlamp may generate an http callback URL instead of https. That can break login.

B. Auth layer in front of Headlamp

Some teams protect Headlamp with an identity-aware proxy or a platform auth system. This keeps sign-in consistent across tools. Headlamp docs include an example using OpenUnison, which can deploy Headlamp with hardened defaults and integrate with identity providers.

5.3 RBAC: keep it least privilege

Kubernetes security starts with API authentication and authorization (RBAC). Headlamp respects those rules.

Practical guidance:

5.4 Quick troubleshooting

Desktop: "I do not see my cluster"

Your kubeconfig may not be in the default location. Point Headlamp to the file with KUBECONFIG or a file path.

In-cluster: "OIDC login fails after redirect"

Confirm your provider allows https://YOUR_URL/oidc-callback. If you use ingress, make sure it forwards X-Forwarded-Proto.

6. Manage multiple clusters

Kubernetes Dashboard is usually tied to one cluster at a time. Headlamp is built for multi-cluster work. It is a client that follows your kubeconfig, not a single cluster install. That means you can keep one UI open and switch clusters as you work.

Clusters come from your kubeconfig

Headlamp reads clusters from your kubeconfig files. That means the clusters you can access with kubectl can also show up in Headlamp.

Switch clusters in the UI

Once Headlamp loads your kubeconfig, you can switch clusters using the cluster selector. This makes it easier to move between dev, staging, and prod without changing tools.

Optional: use more than one kubeconfig file

If you keep separate kubeconfig files, you can load them together. Headlamp supports multiple kubeconfig paths in KUBECONFIG.

Unix/macOS/Linux (: separator):

KUBECONFIG=~/.kube/dev:~/.kube/prod headlamp

Windows (; separator):

$env:KUBECONFIG="$HOME\.kube\dev;$HOME\.kube\prod"

Optional: add a cluster from inside Headlamp

You can also add clusters by loading additional kubeconfig files from the UI.

Permissions stay the same

Multi-cluster does not change security rules. Each cluster still enforces its own RBAC. Headlamp shows only what your identity can do in the selected cluster.

7. Navigate and understand resources

If you used Kubernetes Dashboard, this part will feel familiar. Headlamp keeps the same core resource views, but makes it easier to move around and understand what is connected.

Find resources in familiar places

Headlamp groups resources in a way that maps closely to Dashboard:

You can filter by namespace at the top of the UI, just like in Dashboard.

Inspect and edit resources

From any list, you can click into a resource to see details:

If your RBAC allows it, you can edit YAML directly from the UI. If it does not, Headlamp shows the resource as read-only. This matches how kubectl behaves.

Use search and filters to move faster

Headlamp adds faster search and filtering across lists. This helps when clusters or namespaces get large. You can narrow views without jumping between pages.

Understand relationships with Map View

Dashboard mostly shows resources as lists. Headlamp also includes a Map View.

Map View shows how resources relate to each other:

This helps when you are troubleshooting. Instead of clicking through several pages, you can see the connections at once. You can spot missing links or broken relationships faster.

When to use lists vs Map View

Both views work on the same data. You are just choosing how much context you want at that moment.

8. Deploy applications with YAML

This is the biggest change for most Kubernetes Dashboard users. Dashboard relied on forms. Headlamp relies on manifests. The goal is not to slow you down. It is to align the UI with how Kubernetes is usually run in practice.

From forms to manifests

In Kubernetes Dashboard, you often deployed an app by filling in a form:

Headlamp does not include the same wizard. Instead, it lets you apply YAML directly from the UI.

This matches how most teams deploy today:

Headlamp fits into that flow rather than replacing it.

Create resources using YAML

To deploy an application in Headlamp:

  1. Select a cluster and namespace.
  2. Click Create.
  3. Paste or upload a YAML manifest.
  4. Review it.
  5. Click Apply.

Create button highlight

The resource appears immediately in the UI.

If the manifest is not valid, Headlamp shows the same errors you would see from the Kubernetes API.

Generate YAML the easy way

If you miss the Dashboard wizard, you can still generate YAML quickly.

For example:

kubectl create deployment nginx \
 --image=nginx \
 --dry-run=client \
 -o yaml > nginx.yaml

You can edit the file if needed, then paste it into Headlamp and apply it.

This gives you a repeatable manifest instead of an object created only through a UI.

What if you use Helm or GitOps?

That works well with Headlamp.

Headlamp does not replace those tools. It gives you visibility into what they create.

What to expect compared to Dashboard

9. Deploy and debug workloads

One of the main reasons people used Kubernetes Dashboard was day-to-day debugging. Headlamp covers the same tasks and adds a few useful upgrades.

View logs

You can view pod logs directly in the UI.

To check logs:

  1. Open Workloads.
  2. Select Pods.
  3. Click a pod.
  4. Open the Logs tab.

Workloads view

If the pod has more than one container, you can switch between containers. Logs stream live, which helps during rollouts or active incidents.

Exec into running pods

Headlamp also lets you open a shell inside a container.

From a pod view:

This opens an interactive session inside the container. It replaces the need to switch back to the terminal for quick checks.

This action follows RBAC rules. If you cannot run kubectl exec, Headlamp will not allow it either.

Check metrics and resource usage

Headlamp can show CPU and memory usage for pods and nodes. This works the same way it did in Dashboard.

A few things to know:

This makes it easy to answer simple questions:

View events when something goes wrong

Events are often the fastest way to understand failures.

In Headlamp, you can:

This is often the first place to look when a workload is stuck or crashes.

How this compares to Dashboard

What stays the same:

What improves:

10. Remove Kubernetes Dashboard

After Headlamp is working and your team is comfortable using it, you can remove Kubernetes Dashboard. This is the final cleanup step.

Removing Dashboard reduces clutter and avoids keeping unused access paths around.

Confirm Headlamp covers your needs

Before uninstalling anything, make sure:

Once these checks pass, you are ready to remove Dashboard.

Uninstall the Dashboard

If you installed Kubernetes Dashboard with Helm, remove it with:

helm uninstall kubernetes-dashboard -n kubernetes-dashboard

If Dashboard was installed by a manifest or addon, remove it using the same method you used to install it.

After removal, confirm the resources are gone:

kubectl get pods -n kubernetes-dashboard

Clean up access artifacts (recommended)

Many Dashboard setups used dedicated service accounts and cluster-wide roles.

Review and remove anything that was created only for Dashboard access, such as:

This reduces long-lived credentials and unused permissions.

Communicate the change

Make sure your team knows:

11. Post-migration checklist

This final checklist helps you confirm the migration is complete. It gives you confidence that Headlamp is working as expected and that nothing important was left behind.

Access and visibility

Authentication and RBAC

Core workflows

Operational confidence

Cleanup confirmation

Team alignment

You've now completed the move from Kubernetes Dashboard to Headlamp. Your team can use the same Kubernetes access model, work across clusters, and rely on workflows that match how Kubernetes is used today. From here, Headlamp becomes your default UI, whether on the desktop or in shared environments. As your needs grow, you can keep using it as-is or extend it with plugins and new views over time.

If you want to help shape what comes next, join the Headlamp community and contribute at headlamp.dev.

13 Jul 2026 6:00pm GMT

08 Jul 2026

feedKubernetes Blog

Announcing etcd v3.7.0

This article is a mirror of the original announcement

Today, SIG etcd is releasing etcd v3.7.0, the latest minor release of the popular distributed key-value store and core Kubernetes component. v3.7 ships the long-requested RangeStream feature, delivers several other performance improvements, removes the last remnants of the legacy v2store, and completes a major protobuf overhaul.

You can download etcd v3.7.0 here:

This release also includes new versions of the two core etcd dependencies, bbolt v1.5.0 and raft v3.7.0.

For instructions on installing etcd, see the install documentation. For the full list of changes, see the etcd v3.7 changelog.

A heartfelt thank you to all the contributors who made this release possible!

Major features

The most significant changes in v3.7.0 include:

Features

RangeStream

In etcd v3.6 and earlier, it is challenging to work with requests that return large result sets. The database would buffer the full result set before sending, leading to unpredictable latency and memory usage, both on the server and the client. The RangeStream RPC lets calling applications accept result sets in chunks, reducing latency and making buffering memory usage more predictable.

Instructions on how to use RangeStream in gRPC calls and in etcdctl can be found in the etcd documentation. Users should try it out for their own applications.

In coordinated releases, the RangeStream feature will become available to users running the upcoming v1.37 of Kubernetes by enabling the EtcdRangeStream feature gate. This early and planned adoption is possible thanks to the merger of etcd and Kubernetes development in 2023.

Performance improvements

v3.7 delivers multiple specific performance improvements, both for the Kubernetes control plane and for other use cases. Kubernetes users should see a significant decrease in overall CPU usage by the etcd members, compared with v3.6.

Keys-only range optimization

etcd v3.7.0 includes a keys-only Range optimization (#21791: keys-only Range optimization). When processing a keys_only Range request or etcdctl get --keys-only, etcd reads solely from its in-memory index. It returns the matched keys without loading all serialized values from bbolt as it did previously. The only exception where loading from bbolt is still required is when keys_only Range requests must be sorted by value (i.e., when SortTarget is set to VALUE).

This reduces unnecessary backend reads and memory use for workloads that only need key names, making large keys-only range requests more efficient.

Faster, more reliable etcd leases

v3.7 improves lease expiration and renewal:

Faster find() operations

etcd 3.7 improves the performance of concurrent watches on keys by making find() operations faster (#19768: adt: split interval tree by right endpoint on matched left endpoints).

Other features

Protobuf overhaul

v3.7 migrates and replaces multiple outdated protobuf libraries with fully supported dependencies. This includes replacing github.com/golang/protobuf and github.com/gogo/protobuf with the fully-supported google.golang.org/protobuf (#14533: Protobuf: cleanup both golang/protobuf and gogo/protobuf), and migrating grpc-logging to grpc-middleware v2 (#20420: Migrate grpc-logging to grpc-middleware v2).

As well as improving security and maintainability, this refactor has been shown to reduce CPU usage by etcd components.

While these changes are not expected to directly affect users running etcd via official binaries or container images, they may affect users who depend on etcd Go modules, such as the client SDK or packages under api/ or pkg/. These consumers may need to update their code or dependencies due to protobuf and related API changes introduced in this release. More detailed information is available from the API change tracking issue.

Unix socket support

etcd now supports Unix socket endpoints (#19760: Add Support for Unix Socket endpoints), enabling local communication without a TCP port. Since this is restricted to single-member clusters, it is mainly aimed at development, testing, and edge device use-cases.

Bootstrap from v3store

One of the major changes in etcd v3.7 is that the server now bootstraps entirely from the v3 store (#20187 Bootstrap etcdserver from v3store), eliminating its dependency on the legacy v2 store during startup.

This milestone is the result of a long-term effort spanning multiple releases, from v3.4 through v3.7. It resolves a long-standing technical debt, significantly simplifies the bootstrap workflow, and lays the foundation for future improvements to etcd.

To maintain backward compatibility, etcd v3.7 continues to generate v2 snapshots. As a result, the --snapshot-count flag is also retained in v3.7. This is the last remaining dependency on the legacy v2 store, and both the v2 snapshot generation and the --snapshot-count flag will be removed in v3.8.

etcdutl timeouts

All etcdutl commands now have a timeout command line argument (#20708: etcdutl: enable timeout functionality for all commands), so offline utility commands no longer block indefinitely when holding a lock.

Setting the authentication token directly

Client v3 now allows users to set the JWT directly, offering more flexibility in authentication options (#16803: clientv3: allow setting JWT directly, #20747: clientv3: disable auth retry when token is set),

Retrieve AuthStatus without authenticating

Clients can check their AuthStatus without attempting to authenticate first, eliminating some application overhead (#20802: etcdserver: remove permission check on AuthStatus api).

New watch metrics

v3.7 adds optional watch send-loop metrics (#21030: Instrument watchstream send loop) for better observability of the watch path:

There is also a new etcd_server_request_duration_seconds metric (#21038: Add metric etcd_server_request_duration_seconds).

etcdctl command cleanup

etcdctl commands were reorganized for clarity (#20162: etcdctl: organize etcdctl subcommand) and global command line arguments are now hidden to streamline help output (#20493: etcdctl: hide global flags).

Upgrading

This release contains breaking changes, particularly around the removal of legacy v2 components. Users should review the upgrade guide before upgrading their nodes. As with all minor releases, perform a rolling upgrade one member at a time and confirm cluster health between steps.

Experimental flags removed

All deprecated experimental flags have been removed (#19959: Cleanup the deprecated experimental flags). Features in etcd now follow the Kubernetes-style feature-gate lifecycle (Alpha → Beta → GA) introduced in v3.6, rather than the old --experimental prefix. If your configuration still relies on --experimental-* command line arguments, migrate to using the corresponding feature gates or stable command line arguments before you upgrade to etcd 3.7.

Legacy V2 API packages and code cleanup

To remove the dependencies on v2store, the following components have been removed:

These changes may create some breakage for users, particularly those who have not already updated to v3.6.11 or later. Users should report any blockers encountered, or cases that need better upgrade documentation.

Non-blocking client creation

etcd no longer honors the deprecated grpc.WithBlock dial option ( #21942: Make the etcd client creation non-blocking). To preserve the previous blocking behavior when needed, follow the guidance in grpc-go's anti-patterns documentation.

Multiarch container images only

For users relying on the official etcd container images, v3.7 will be distributed only as multiarch containers. Architecture-tagged images will not be available, so adjust deployments accordingly.

API changes

As with every etcd release, there are a number of API changes. These are designed to be backwards-compatible to the extent possible, but may require adjustment by some users. See our API documentation page for full information.

bbolt v1.5.1

etcd v3.7 depends on, and includes, v1.5.1 of the bbolt storage engine. v1.5 includes several improvements to functionality and performance, including:

raft v3.7.0

etcd 3.7 depends on, and includes, v3.7.0 of the raft consensus engine. v3.7 includes several improvements, including:

raft v3.7.0 also includes the same protobuf library updates and refactoring as etcd does.

Dependency updates

Other dependency updates include a bump to golang.org/x/crypto v0.52.0 for CVE resolution (#21903: [release-3.7] Bump golang.org/x/crypto to v0.52.0), an OpenTelemetry contrib update to v0.61.0 (#20017: Update otelgrpc to v0.61.0), and compilation with Go 1.26.4 (#21891: [release-3.7] Update Go to 1.26.4).

Contributors

etcd v3.7.0 is the product of more than a hundred contributors across the community. Thank you to everyone who wrote code, reviewed PRs, filed and triaged issues, and helped test the alpha, beta, and release candidates.

Leads

The SIG etcd leads for the v3.7 release are ivanvc, serathius, ahrtr, fuweid, siyuanfoundation, and jberkus. Ivan leads our release team.

Other contributors

ah8ad3, ajaysundark, aladesawe, amosehiguese, ArkaSaha30, ashikjm, AwesomePatrol, dims, Elbehery, gangli113, henrybear327, Jille, jmhbnz, joshuazh-x, kishen-v, lavishpal, liggitt, marcelfranca, miancheng7, mmorel-35, MrDXY, mrueg, purpleidea, qsyqian, redwrasse, ronaldngounou, skitt, spzala, tcchawla, tjungblu, vivekpatani, wenjiaswe

New contributors

A special welcome to the contributors who made their first etcd contribution in this cycle - including Jeffrey Ying, whose work drove the RangeStream feature. New contributors can have a substantial impact on etcd; if you'd like to get involved, see the contributor guide.

1911860538, 4rivappa, aaronjzhang, abdurrehman107, ABin-Huang, adeptvin1, aditya7880900936, AHBICJ, akstron, alliasgher, aman4433, aojea, apullo777, AR21SM, arturmelanchyk, AshrafAhmed9, asttool, asutorufa, BBQing, beforetech, boqishan, caltechustc, carsontham, christophsj, chuanye-gao, cnuss, cuiweixie, dmvolod, Dogacel, dongjiang1989, EduardoVega, evertrain, eyupcanakman, gaganhr94, goingforstudying-ctrl, greenblade29, Himanshu-370, HossamSaberX, huajianxiaowanzi, hwdef, ishan-gupta2005, ishan16696, ivangsm, JasonLove-Coding, Jefftree, jihogh, jonathan-albrecht-ibm, joshjms, kairosci, kei01234kei, kjgorman, kovan, kstrifonoff, Kunalbehbud, letreturn, lorenz, m4l1c1ou5, madhav-murali, madvimer, majiayu000, marcus-hodgson-antithesis, mattsains, mcrute, mingl1, MohanadKh03, mstrYoda, NAM-MAN, neeraj542, nicknikolakakis, nihalmaddala, niuyueyang1996, notandruu, ntdkhiem, nwnt, olamilekan000, pigeio, pjsharath28, progmem, Qian-Cheng-nju, quocvibui, ravisastryk, robin-vidal, robinkb, rockswe, roman-khimov, rsafonseca, sahilpatel09, SalehBorhani, SebTardif, seshachalam-yv, shashwat010, shivamgcodes, shuan1026, silentred, sneaky-potato, socketpair, srri, subrajeet-maharana, sxllwx, tchap, tsujiri, tzfun, upamanyus, uzairhameed, varunu28, vihasmakwana, wendy-ha18, xiaoxiangirl, xigang, xUser5000, yagikota, yajianggroup, yedou37, Zanda256, zechariahkasina, zhijun42, zhoujiaweii

Feedback can be shared through:

08 Jul 2026 12:00pm GMT