01 Sep 2026

feedKubernetes Blog

Kubernetes v1.37: etcd RangeStream Cuts Memory Use on Large List Reads

I am excited to announce that etcd RangeStream is graduating to beta in Kubernetes v1.37. Paired with etcd v3.7, it reduces the memory the API server and etcd need to read a large collection, and makes peak usage more predictable.

The cost of large reads

The API server serves most list and watch requests from its in-memory watch cache. Populating that cache requires reading a resource's full state from etcd, at startup and on every re-initialization. For a resource with many objects, or large ones, such as Pods, that read is expensive.

The API server already paginated these reads, asking etcd for a fixed number of keys at a time rather than the whole collection at once. But a page bounded by key count has no awareness of object size, so a page of large objects can still be very large. That makes memory usage hard to predict, and a bad combination of object size and concurrent reads can be enough to trigger an OOM. etcd's unary Range assembles each page in full before sending it, and the API server holds it while decoding, so the same payload sits in memory on both sides at once. Most of that cost lands on etcd, which is also where streaming helps most.

Streaming reads with RangeStream

etcd v3.7 adds a streaming version of that read, the RangeStream RPC. It takes the same RangeRequest as Range and returns the same result set, but instead of building the whole response up front, etcd splits it into chunks and streams them. Chunk size is tuned adaptively to the values being returned, so a collection of large objects is bounded by bytes rather than by a key count, and memory is freed as the stream progresses instead of being held until a whole page is assembled.

When the feature is enabled, the API server uses RangeStream wherever it reads a whole collection out of etcd. This includes watch cache initialization, and the fallback paths where a list request cannot be served from the cache and reads etcd directly. In either case the API server decodes each chunk as it arrives and releases it before pulling the next one, so neither side ever holds the whole collection.

Requirements

RangeStream is used when the EtcdRangeStream feature gate is enabled on the kube-apiserver, which is beta and on by default in v1.37, and etcd is v3.7 or later. The API server resolves etcd's support at startup and also falls back at runtime if a call returns Unimplemented, so an API server paired with an older etcd keeps using the paginated Range path on its own. To turn it off, disable the gate:

--feature-gates=EtcdRangeStream=false

Confirming RangeStream is in use

The API server records streamed reads under their own operation label on its etcd metrics. A non-zero count here means RangeStream is in use:

etcd_request_duration_seconds_count{operation="listStream"}

If it stays at zero, the API server is still using the paginated Range path, most likely because etcd is older than v3.7.

Learn more

If you have questions or feedback, join the #sig-etcd channel on Kubernetes Slack.

01 Sep 2026 6:30pm GMT

31 Aug 2026

feedKubernetes Blog

Kubernetes v1.37: Storage Version Migration Enabled by Default

I am excited that storage version migration (SVM) has graduated to General Availability (GA) in Kubernetes v1.37!

After a number of releases of work and testing, the built-in StorageVersionMigration API (storagemigration.k8s.io/v1) and control plane controller are now fully stable and enabled by default across all v1.37 Kubernetes clusters.

The problem with stale storage versions

In Kubernetes, stored API resources are written using a specific storage version (schema representation). The way Kubernetes interacts with object storage fundamentally requires mutation of a resource in order to ensure that the latest storage version is used for all resources. This creates problems when you want to change the storage version of a resource.

One example of a scenario where you may want to change the storage version of a resource is when you are promoting a CRD to drop an older API version (such as v1alpha1) to a newer version (leaving just v1beta1 and v1). It's a problem to drop the older API version whilst there are still resources stored with the old alpha version.

To avoid problems, you designate v1 as the new storage version; but, on it's own, that's not enough. While new writes are stored as v1, any existing resource could remain stored as v1alpha1 or v1beta1 in storage. You cannot safely remove v1alpha1 from the CRD's .status.storedVersions or drop serving support until every single resource in storage has been re-written to not be serialized and stored with the alpha version.

Another relevant example is encryption at rest and, related, key rotation. When you configure encryption at rest or rotate encryption keys, existing resources in storage remain unencrypted (or encrypted under old keys) until they are actively re-written through the Kubernetes API server.

Historically, cluster administrators and CRD authors had to rely on manual kubectl get / kubectl replace scripts, or to deploy the out-of-tree kube-storage-version-migrator component to force re-writes. These approaches were often tedious, error-prone, and difficult to monitor.

How storage version migration works

Initiating a storage version migration is as simple as creating a declarative StorageVersionMigration object. The built-in StorageVersionMigrator controller in the Kubernetes control plane watches for these objects, and automatically migrates existing resources to the default storage version for that API.

Example: Migrating a custom resource API

Suppose you have updated a CustomResourceDefinition (crontabs.example.com) to use v1 as its storage version. To migrate all existing stored resources off older versions, create a StorageVersionMigration:

apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
 name: crontabs-migration
spec:
 resource:
 group: example.com
 resource: crontabs

Apply the manifest using kubectl:

kubectl apply -f crontabs-migration.yaml

Monitoring and verifying migrations

The StorageVersionMigrator controller updates the status of the StorageVersionMigration object as migration progresses. You can inspect the migration status using kubectl:

kubectl get storageversionmigration.storagemigration.k8s.io/crontabs-migration -o yaml

A successful migration will report a Succeeded condition set to True:

status:
 conditions:
 - type: Running
 status: "False"
 lastUpdateTime: "2026-08-02T10:05:00Z"
 reason: StorageVersionMigrationInProgress
 - type: Succeeded
 status: "True"
 lastUpdateTime: "2026-08-02T10:05:00Z"
 reason: StorageVersionMigrationSucceeded

Once the migration has succeeded, you can be confident that all instances of the resource in storage are stored in the current storage version. For CRDs, the stored version should be updated in the CRD's .status.storedVersions to only contain the preferred version. If the .status.storedVersions is not updated following a successful migration then that means that the CRD was updated during the migration. In that case, the migration should be retried in order to safely deprecate an older storage version.

Including migrations in your CRD manifests

Because StorageVersionMigration is a standard declarative Kubernetes API, CRD authors can bundle or trigger migrations directly alongside CRD upgrades. For example, you can include the migration in the same manifest as your updated CustomResourceDefinition:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
 name: crontabs.example.com
spec:
 group: example.com
 # Updated versions list where v1 has storage: true
 ...
---
apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
 name: crontabs-migration
spec:
 resource:
 group: example.com
 resource: crontabs

What's next?

SIG API Machinery would love to hear your feedback as you adopt built-in Storage Version Migration in your clusters. Reach out to us on the #sig-api-machinery Slack channel or participate in our community discussions!

31 Aug 2026 6:30pm GMT

28 Aug 2026

feedKubernetes Blog

Kubernetes v1.37: Pod Certificates and Cluster Trust Bundles

Pod Certificate / Cluster Trust Bundles Blog Post

Kubernetes brings a wealth of features that make it easy to run your production workloads securely and reliably. While aspects like scheduling, health checks and resource limits are probably at the front of your mind, one other important feature of Kubernetes is production identity - how your workload can authenticate to other systems in order to do its job.

Up until now, the primary production identity mechanism built into Kubernetes has been service account JWTs (JSON Web Tokens). These are cryptographically-signed tokens, issued by the control plane of your cluster, that let anyone in the world understand who is calling when your workload uses them.

In Kubernetes 1.37, the foundations of a new built-in production identity technology have gone GA. Pod Certificates (and the closely-associated Cluster Trust Bundles) build X.509 certificate issuance for TLS and mTLS directly into core Kubernetes.

Why?

Service account JWTs have a lot going for them:

However, service account JWTs have one big downside - they are bearer tokens. With bearer tokens, if you have the token, then you are the identity asserted by the token. And since you necessarily have to hand copies of the JWT to all your peers in order to authenticate to them, they can be you, too.

There are partial mitigations for this, and service account tokens make use of them (time-, object-, and audience-binding), but none are complete defences.

A solution to this problem lies in proof-of-possession credentials, where you don't send your entire credential to your peer, but only a proof that you possess the credential. In practice, these schemes are always built on asymmetric cryptographic signatures (RSA, ECDSA, and friends).

There are few different standard approaches, such as request signing (AWS SigV4, JWT DPoP, RFC 9421), but the most widely-deployed and understood solution is X.509 certificates, as used in TLS. In TLS, your credential is split into two pieces

The goal of Pod Certificates is to make using X.509 certificates from your Kubernetes workload just as easy as using service account JWTs, while maintaining Kubernetes' high security bar. I think we've hit this target.

As I'll cover in the architecture and example sections below, there are many similarities between the design of service account JWT issuance and Pod Certificates. One significant place they diverge, however, is that Pod Certificates is a much more flexible mechanism. Kubernetes only offers one flavor of service account JWTs, with standardized claims.

The X.509 ecosystem is significantly more varied than the JWT ecosystem, and X.509 certificates used for different purposes contain different extensions and information. For this reason, Pod Certificates has common machinery built into Kubelet, but offers a pluggable interface so that many different types of certificates can be issued within a single cluster, at the same time.

In the fullness of time, I expect Kubernetes to offer at least two built-in certificate providers:

In the remainder of this article, I'll take you through the overall architecture of a Kubernetes workload using Pod Certificates, as well as give you an example of installing and using a real (toy) Pod Certificates signer controller.

Architecture

When you use Pod Certificates and Cluster Trust Bundles, there are the following major components:

Block diagram of an application using Pod Certificates

Architecture of an application using Pod Certificates

The best way to get a sense of what these components each do is to follow the issuance process chronologically:

  1. Once your application pod is scheduled to a node, Kubelet identifies all of the podCertificate and clusterTrustBundle projected volumes sources in its spec.
  2. For each podCertificate source:
    1. Kubelet generates a new private key according to the keyType field.
    2. Kubelet creates a PodCertificateRequest addressed to the signer named in the source.
    3. The signer controller sees the PodCertificateRequest and decides whether or not to issue the certificate.
    4. The signer controller issues the certificate by filling out the status.certificateChain field.
    5. The signer controller also fills out the status.beginRefreshAt field to instruct Kubelet when it should begin trying to refresh the certificate.
      certificate to the container filesystem.
  3. For each clusterTrustBundle source:
    6) Kubelet retrieves the issued certificate, and writes the private key and
    1. Kubelet collects all the ClusterTrustBundles that match the signer name
    2. Kubelet unifies all of the certificates from all matching ClusterTrustBundles, and (stably) reorders them (to prevent applications from accidentally depending on a particular ordering).
      and label selectors in the source.
    3. Kubelet writes the certificates to the file path named in the source.
      and trust anchors from the filesystem.
  4. Your application pod starts up, and the application reads keys, certificates,
  5. Kubelet periodically updates the files from clusterTrustBundle sources as the contents of the selected ClusterTrustBundles changes. The application must pick up the changes using inotify or polling.
  6. As each certificate's beginRefreshAt time passes, Kubelet repeats the process in step 2 to refresh the certificates, and write the update private keys and certificate chains to the filesystem. As in step 5, the application must pick up changes using inotify or polling.

Some key takeaways:

Try it out

Because the Kubernetes project does not yet ship any Pod Certificate signers in core, in order to try these features out, you will need to install a third-party signer into your cluster. To make this easier, I have written Tinycert, which you can install into your cluster (or a Kind cluster).

Tinycert is not a full production solution, but it's a good starting point for experimenting with Pod Certificates, as well as a base for creating your own signers.

Tinycert provides:

What next?

Happy hacking!

28 Aug 2026 6:30pm GMT