fidzu : k8s

Memory QoS has graduated to Beta in Kubernetes v1.37 and is now enabled by default. On Linux nodes running cgroup v2, the feature uses the memory controller to give the kernel better guidance on how to treat container memory. It was first introduced as Alpha in v1.22, and expanded in v1.36 with tiered memory reservation.

This post covers what changed in v1.37, what the Beta promotion means for cluster operators, and how to configure the feature.

What changed in v1.37

Memory QoS is Beta and enabled by default

The MemoryQoS feature gate is now Beta in v1.37. This means every v1.37 kubelet has the feature gate turned on without any configuration change. Turning on the feature by default is safe because the default kubelet configuration does not enable memory throttling or memory reservation. No memory.high, memory.min, or memory.low values are written to cgroups unless you explicitly configure them.

You can opt into specific behaviors through kubelet configuration fields:

1. Set memoryThrottlingFactor (for example, 0.9) to enable memory.high throttling on Burstable and BestEffort containers. The default is null, which means no throttling.
2. Set memoryReservationPolicy to TieredReservation to enable tiered memory protection via memory.min and memory.low. The default is None, which means no memory reservation.

Default memoryThrottlingFactor changed to null

In earlier Alpha releases, memoryThrottlingFactor defaulted to 0.9, which meant enabling the feature gate caused the kubelet to set memory.high on containers. In v1.37, the default is null, so the kubelet does not set memory.high unless you configure a value.

This change was made because, with the feature gate now on by default, an automatic memory.high could throttle workloads that were previously running without throttling. Making it null ensures that upgrading to v1.37 does not change runtime behavior for existing clusters.

If your kubelet configuration file already contains an explicit memoryThrottlingFactor value, that value is preserved during the upgrade and throttling continues to work as before. If your configuration file does not include memoryThrottlingFactor, the kubelet uses the new null default and stops setting memory.high. To keep throttling in that case, add memoryThrottlingFactor explicitly:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9

How to configure MemoryQoS in v1.37

For full details on configuring Memory QoS, see Memory QoS with cgroup v2, Configuring memory reservation, and System requirements

Enable memory throttling only

Set memoryThrottlingFactor to a value between 0 and 1. The kubelet uses this factor to calculate memory.high for Burstable and BestEffort containers. See Memory throttling for how memory.high is calculated for each QoS class.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9

Enable memory throttling and tiered reservation

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9
memoryReservationPolicy: TieredReservation

Enable tiered reservation without throttling

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryReservationPolicy: TieredReservation

Disable Memory QoS entirely

To disable the feature after upgrading, set the feature gate to false and ensure a compatible kubelet configuration. The kubelet rejects the configuration if memoryThrottlingFactor is set to anything other than the former default of 0.9, or if memoryReservationPolicy is TieredReservation, so remove or adjust those fields if you set them.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
 MemoryQoS: false

When the feature gate is off, or memoryReservationPolicy is not TieredReservation, the kubelet resets stale protection at startup on cgroup v2 nodes: memory.min=0 and memory.low=0 on the root kubepods cgroup, and memory.low=0 on the Burstable QoS cgroup. For containers, stale memory.high values are reset to max on reconciliation paths such as restart or resize.

Known limitation: memory reservation is node-wide

memoryReservationPolicy applies to every pod on the node. With TieredReservation, every Guaranteed pod gets memory.min and every Burstable pod gets memory.low; there is no way to opt individual pods in or out. A node that mixes workloads needing hard reservation with workloads that should stay reclaimable has to choose one policy for all of them.

Hard reservation also covers everything charged to the container's cgroup, including page cache, so a pod that reads large files can hold memory the kernel would otherwise reclaim to serve its neighbors.

SIG Node is tracking both in kubernetes/kubernetes#140246. If this affects you, that issue is the best place to describe your workload.

What to expect next

The next milestone for Memory QoS is graduation to GA. Feedback from Beta users will shape any remaining adjustments before that step. If you run into issues, please file bugs at kubernetes/kubernetes.

How can I learn more?

• KEP-2570: Memory QoS
• Pod Quality of Service Classes
• Memory QoS with cgroup v2
• Managing Resources for Containers
• Kubernetes cgroups v2 support
• Linux kernel cgroups v2 documentation

Getting involved

This feature is driven by SIG Node. If you are interested in contributing or have feedback, you can reach out through:

• Slack: #sig-node
• Mailing list
• SIG Node meetings

14 Sep 2026 6:30pm GMT

Changed Block Tracking (CBT) support for CSI drivers shipped as Alpha in September 2025. With the March 2026 v1.0.0 release of the external-snapshot-metadata project, the feature moved to Beta.

If you aren't yet familiar with changed block tracking for storage in Kubernetes, the Alpha announcement covers the motivation, the three primary components (the CSI SnapshotMetadata gRPC service, the SnapshotMetadataService CRD, and the external-snapshot-metadata sidecar), and a walkthrough of how to use the API. CBT currently applies to block volumes; file-volume and network file-share changed-list tracking is not covered by this feature. This post focuses on what is different in Beta.

What's new in Beta

The main change in that release was the promotion of the SnapshotMetadataService CRD from v1alpha1 to v1beta1. The CRD used to advertise a driver's metadata service now serves cbt.storage.k8s.io/v1beta1. The schema itself is unchanged, but this release removed v1alpha1 (rather than serving it alongside the new version). If you are upgrading from Alpha, you need to:

• Re-apply the CRD definition shipped with v1.0.0.
• Update SnapshotMetadataService manifests to use apiVersion: cbt.storage.k8s.io/v1beta1.
• Update any client or controller code that talks to the CRD.

This is a one-time change. There is no automatic conversion between the two versions.

Compatibility

• Minimum Kubernetes version: 1.33
• CSI spec: 1.10 or newer
• Container image: registry.k8s.io/sig-storage/csi-snapshot-metadata:v1.0.0

Trying it out

The Getting Started section in the Alpha blog still applies. In short:

1. Make sure your CSI driver supports volume snapshots and ships the external-snapshot-metadata sidecar.
2. Install the SnapshotMetadataService CRD (the v1beta1 definition from the v1.0.0 release).
3. Create a SnapshotMetadataService resource for your driver.
4. Use a client - snapshot-metadata-lister, or your own implementation - to call GetMetadataAllocated and GetMetadataDelta.

If you want to see the full flow end-to-end, the hostpath driver example is a good starting point.

What's next?

The focus for the rest of the Beta cycle is wider CSI driver adoption and operational feedback before the feature moves towards GA. If you maintain a CSI driver, this is a good time to evaluate adding support. If you are building a backup application on top of the API, feedback on the streaming clients and the iterator package is very welcome.

Where can I learn more?

• The CSI developer documentation for snapshot metadata.
• KEP-3314.
• The external-snapshot-metadata repository.
• The gRPC schema.
• The snapshot-metadata-lister example client.

How do I get involved?

This work is the result of contributions from many people across SIG Storage. A big thank you to everyone who helped review, code, and test the feature through Alpha and into Beta:

• Ben Swartzlander (bswartz)
• Carl Braganza (carlbraganza)
• Daniil Fedotov (hairyhum)
• Ivan Sim (ihcsim)
• Nikhil Ladha (Nikhil-Ladha)
• Praveen M (iPraveenParihar)
• Rakshith R (Rakshith-R)
• Xing Yang (xing-yang)

If you would like to get involved with CSI or storage in Kubernetes, SIG Storage is the place to start. The Data Protection Working Group also holds regular meetings, and new attendees are always welcome.

14 Sep 2026 6:30pm GMT

I'm excited to announce that native histogram support for Kubernetes metrics is graduating to Beta and is enabled by default in Kubernetes v1.37!

Native histograms (previously introduced as Alpha in Kubernetes v1.36 under KEP-5808) bring high-resolution, low-cardinality observability to Kubernetes metrics. By adopting Prometheus Native Histograms, Kubernetes components now expose latency and duration metrics with far greater accuracy while significantly reducing telemetry storage and scraping overhead.

Why move beyond classic histograms?

Since the early days of Kubernetes observability, duration and latency metrics (such as API server request latencies or scheduling durations) have relied on classic Prometheus histograms.

Classic histograms require metric authors to define a static list of cumulative bucket boundaries (le labels), such as 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10. While familiar, this approach introduces three major challenges:

1. The Bucket Guessing Game: If a workload's latency profile changes, for example, shifting into microsecond ranges or experiencing long-tail tail latencies beyond the highest bucket, the histogram loses visibility. Specifying bucket boundaries upfront requires knowing the distribution before observing it
2. High Cardinality & Storage Cost: With classic histograms, each bucket boundary is exported as a separate time series (_bucket{le="..."}). A histogram with 10 buckets across multiple labels multiplies the number of time series by 10, increasing memory consumption in Prometheus and inflating time series database (TSDB) storage costs
3. Interpolation Error in Quantiles: Calculating percentiles using histogram_quantile() relies on linear interpolation between static bucket boundaries. When bucket spans are coarse, quantile calculations can suffer from significant estimation error

What are Prometheus native histograms?

Prometheus Native Histograms replace static user-defined buckets with dynamic, exponential buckets.

Instead of emitting a separate time series for every single bucket boundary, a native histogram is stored as a single time series containing a rich schema of positive and negative spans, zero thresholds, and exponential scaling factors.

• High Resolution Automatically: Exponential buckets dynamically adjust to any value range - from nanoseconds to hours - without requiring pre-configured bucket boundaries
• Up to 90% Fewer Time Series: By consolidating buckets into structured spans within a single time series, scraping and storage overhead are dramatically reduced
• Accurate Quantile Calculation: Quantiles can be calculated with mathematical bounds on error (≃5% worst-case relative error under default settings) across the entire spectrum of observations

How native histograms work in Kubernetes

In Kubernetes, native histogram support is implemented directly inside the shared metrics subsystem (k8s.io/component-base/metrics).

Figure 1 illustrates how native histogram metrics are processed and exposed across Kubernetes components.

1. Dual exposition for zero breaking changes

A primary design requirement for KEP-5808 was zero disruption for existing observability stacks. When the NativeHistograms feature gate is enabled, Kubernetes components use dual exposition:

• Classic buckets (h.Bucket) are still emitted alongside native spans. Existing Prometheus servers, dashboards, and alerting rules that rely on traditional text scraping or classic bucket labels continue to work unmodified
• Native spans (h.Schema, h.PositiveSpan) are included in the same Protobuf payload for collectors that understand native histograms

2. Tuned default exponential configuration

When NativeHistograms is enabled, the k8s.io/component-base/metrics package automatically applies standardized exponential options to all histogram metrics:

• BucketFactor: 1.1: Configures exponential buckets where each bucket is at most 10% wider than the preceding one. This guarantees a mathematically bounded worst-case relative error of at most ~5% for quantile calculations regardless of whether an operation takes 1 millisecond or 10 seconds.
• MaxBucketNumber: 160: Caps the maximum number of buckets per histogram to 160. Following OpenTelemetry SDK recommendations for base-2 exponential histogram aggregation, this limit protects component memory usage even under extreme outlier distributions.

3. Broad component support

Because native histograms are integrated into component-base/metrics, all major Kubernetes control plane and node components inherit support automatically, including:

• kube-apiserver (e.g., apiserver_request_duration_seconds, authentication/authorization metrics, validation latencies)
• kube-scheduler (e.g., scheduler_plugin_execution_duration_seconds, scheduler_scheduling_algorithm_duration_seconds)
• kubelet (node-level container runtime and pod lifecycle metrics)
• kube-controller-manager and kube-proxy

How to scrape native histograms

The simple answer: upgrade to Kubernetes v1.37, and it works.

Because Kubernetes v1.37 enables NativeHistograms by default, your cluster is already emitting dual-exposition metrics. How you configure Prometheus to scrape native histograms depends on your Prometheus version:

1. Prometheus scrape configuration by version

• Prometheus 3.0+ (Recommended): Use explicit per-job configuration in your scrape_configs rather than global flags (the global --enable-feature=native-histograms flag is deprecated in Prometheus 3.9+):

  scrape_configs:
   - job_name: 'kubernetes-apiservers'
   scrape_native_histograms: true
   always_scrape_classic_histograms: true # Recommended during transition
  

  You must read the caution in Migrating dashboards and alerts in the Native Histograms documentation. In summary: always set always_scrape_classic_histograms: true during your transition period. Without this setting, Prometheus will only ingest the native format and stop ingesting classic _bucket, _count, and _sum series. Setting always_scrape_classic_histograms: true ensures existing dashboards (histogram_quantile(..._bucket...)) and alerts continue to work while you migrate them to native histograms.

• Prometheus 2.40 - 2.x: Enable Native Histograms globally by starting Prometheus with the feature flag:

  prometheus --enable-feature=native-histograms
  

  Note that in Prometheus 2.x, this is an all-or-nothing setting for all scrape targets.

2. Verify Protobuf dual exposition

Standard Prometheus text scraping (application/openmetrics-text or plain text format) only transfers classic buckets. When scrape_native_histograms is enabled, Prometheus automatically negotiates Protobuf format with Kubernetes endpoints.

You can verify that a Kubernetes component is exporting native histograms using curl with an Accept header specifying Protobuf. For example:

## THIS IS NOT SECURE. ONLY DO THIS IN A TEST CONTEXT.
curl --insecure \
 -H "Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited" \
 --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
 https://localhost:6443/metrics

When decoded, the returned MetricFamily for histogram metrics (like apiserver_request_duration_seconds) will contain both traditional bucket entries and populated schema / positive_span fields.

Querying native histograms in PromQL

Once native histograms are ingested into Prometheus, you can query them using standard PromQL histogram functions without needing static le bucket labels or _bucket suffixes:

# 1. Calculating P99 latency for a single target:
# Classic histogram (requires _bucket suffix):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m]))

# Native histogram (operates directly on the metric name):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds[5m]))

# 2. Aggregating across multiple instances (e.g., all API servers):
# Classic histogram (requires sum by (le) to preserve bucket boundaries):
histogram_quantile(0.99, sum by (le) (rate(apiserver_request_duration_seconds_bucket[5m])))

# Native histogram (no grouping by le required!):
histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds[5m])))

With native histograms, functions like histogram_quantile() operate directly on the dynamic exponential spans inside the time series, producing highly accurate quantiles without static bucket interpolation error.

For official documentation on querying Native Histograms in PromQL, see:

• PromQL histogram_quantile documentation
• PromQL histogram_fraction documentation
• PromQL histogram_sum documentation
• PromQL histogram_count documentation
• PromQL histogram_count documentation
• PromQL histogram_stddev and histogram_stdvar documentation

Dashboard migration & rollback strategy

Recommended migration workflow

To safely transition your monitoring infrastructure to Native Histograms without breaking existing alerts or dashboards, I recommend a four-step migration workflow:

1. Enable Both Formats: In your Prometheus 3.x scrape config, set scrape_native_histograms: true AND always_scrape_classic_histograms: true so both formats are collected safely during transition
2. Migrate Queries: Update your Grafana dashboards and Prometheus alerting rules from classic quantile queries (histogram_quantile(..._bucket...)) to native histogram queries (histogram_quantile(...)), and replace references to classic _count and _sum series with histogram_count(...) and histogram_sum(...)
3. Verify in Staging/Production: Validate that all dashboards and SLO alerts fire and graph correctly using the new native histogram queries
4. Unlock ~10x Storage Savings: Once migration is complete, set always_scrape_classic_histograms: false. Prometheus will stop ingesting the static _bucket, _count, and _sum time series, reducing your histogram time series count by up to 90%!

Opt-out and rollback flexibility

Because native histograms are dual-exposed, using them is entirely opt-in from a collector perspective:

• Instant Collector Rollback: If you need to stop ingesting native histograms, simply set scrape_native_histograms: false in your Prometheus job configuration. No Kubernetes restart is required, and Prometheus will immediately resume scraping only the classic format without data loss
• Component Feature Gate Rollback: Administrators can also disable the feature gate on Kubernetes components using --feature-gates=NativeHistograms=false (requires component restart)

What's next & how to get involved

As native histograms progress toward General Availability (GA) in future Kubernetes releases, SIG Instrumentation will continue evaluating ecosystem readiness, performance characteristics, and long-term plans for eventually deprecating static classic buckets once native histogram adoption becomes ubiquitous across the monitoring community.

• Read the KEP-5808 page or the KEP GitHub issue to learn more.
• Read the Prometheus Native Histograms specification and PromQL querying functions documentation
• Get involved with SIG Instrumentation on Slack in #sig-instrumentation or join the weekly SIG meetings

Acknowledgements

A huge thank you to contributors across SIG Instrumentation and component owners who collaborated on the design, implementation, testing, and review of native histograms in Kubernetes!

11 Sep 2026 6:30pm GMT