16 Nov 2026

feedKubernetes Blog

Scheduling the removal of the kube-proxy IPVS backend

The ipvs backend for kube-proxy was an experiment in providing a service proxy implementation with better performance than the iptables backend. While it succeeded in that goal, the kernel IPVS API has turned out to be a bad match for the Kubernetes Services API, and the ipvs backend was never able to implement all of the edge cases of Kubernetes Service functionality correctly.

In Kubernetes 1.29, the project introduced the nftables backend for kube-proxy, which became GA in 1.33. Part of the plan behind adding the nftables backend was the idea that it would eventually replace both the iptables and ipvs backends. Because the ipvs backend in particular is complex, not 100% feature complete, and architecturally quite different from the other two backends (and thus more work to maintain), Kubernetes SIG Network decided to officially deprecate it as of Kubernetes 1.35. Although it currently remains available in kube-proxy (with a warning about its deprecation), we (Kubernetes SIG Network) now plan to remove it in Kubernetes 1.40. All ipvs users should therefore be planning to migrate to either the nftables or iptables backends.

Why was the ipvs backend written?

The ipvs backend was originally designed to fix two different kinds of performance problems in the iptables backend:

The ipvs backend solved both of these scaling problems. The data plane problem was solved mostly automatically, because the kernel IPVS subsystem uses hash tables rather than linear lookup. The control-plane problem was solved by not rewriting the entire ruleset on every update. (The kernel IPVS API doesn't have a way of doing batch updates anyway, so this was essentially forced.)

Why is the ipvs backend now being deprecated?

Almost from the start, the ipvs backend also had problems.

The first version didn't have 100% compatibility with the iptables backend, but it was assumed that this would be fixed over time. Unfortunately, the original authors moved on to other things outside the Kubernetes ecosystem (as inevitably sometimes happens in open source), and it took a while to find new people interested in maintaining the code.

Meanwhile, as people began using the ipvs backend in larger and larger clusters, its own scaling problems started to emerge. In particular, the lack of batch update operations in the IPVS API meant that while the ipvs backend was faster than iptables when making a few small changes, it was potentially slower when making a large number of changes at once. And because the IPVS API is fairly limited in what it can do, kube-proxy's ipvs backend needed to fall back to using iptables to implement various features of Kubernetes Services beyond basic "load balancing", making it run into some of the same scaling problems the iptables backend has. While the ipvs backend tried to mitigate this by using ipset for some of its rules, this again ran into the problem of not having a batch update API. Additionally, the fact that the ipvs backend uses iptables means that it has the same long-term support problem as the iptables backend: Red Hat's plan to remove the iptables API entirely from Red Hat Enterprise Linux means the ipvs kube-proxy backend will also no longer be available to Kubernetes users on RHEL.

What about IPVS schedulers?

Beyond performance, the other reason some users were excited about the ipvs backend was the ability to pick from among various IPVS "schedulers" for distributing traffic to endpoints, rather than relying on the simple random selection used by iptables (and nftables).

Unfortunately, the way IPVS is used by kube-proxy prevents it from being able to use most of the IPVS schedulers in a useful way. The problem is that kube-proxy is a distributed service that runs on all nodes in the cluster, but the kube-proxy processes on different nodes do not share their IPVS scheduler state, so each node picks endpoints without knowing what is happening with clients on any other node. Even the schedulers that are designed to work in a distributed way generally don't work better than random selection in the context of kube-proxy. For example, the mh scheduler uses Maglev Hashing to select endpoints, which theoretically allows for extremely efficient distributed load balancing. But that efficiency is only in the context of a specific sort of two-layer load-balancer design, and kube-proxy does not have the right architecture to be able to act as either of the two layers in that design.

Switching away from ipvs

Switching to nftables

If you are running Kubernetes 1.36 or later, and your nodes are running a Linux distribution with kernel 5.13 or later, and you use the ipvs proxy mode, then Kubernetes recommends switching to the nftables mode of kube-proxy. (Although the nftables backend is GA -and fully usable-as of Kubernetes 1.33, there have been some performance- and scaling-related fixes since then that that weren't backported to releases earlier than 1.36.)

The nftables backend is fully supported, feature-complete, and even has very slightly better performance than ipvs:

kube-proxy ipvs-vs-nftables first packet latency, at various percentiles, in clusters of various sizes

(Taller bars represent longer latency. The "p50" group is the average connection establishment latency. Note that even the largest numbers in this graph are quite tiny; the point is not to claim that nftables is (noticeably) faster than ipvs, just to show that, unlike iptables, it's not slower.)

Switching is as easy as changing the kube-proxy configuration and restarting kube-proxy. For example, in clusters deployed via kubeadm, change mode: ipvs to mode: nftables in the kube-proxy ConfigMap and then restart the kube-proxy pods, as described in the documentation on reconfiguring a kubeadm cluster. (If you are overriding other ipvs-specific kube-proxy configuration, you will also need to change those options to the corresponding nftables ones.)

Switching to iptables: "It's Not As Slow As You Think!"

If you can't switch to nftables, either because you are on an older version of Kubernetes, or because your cluster runs on an older Linux distribution that isn't new enough to support the nftables backend, you may be able to just switch to the iptables backend, which has improved greatly since the original creation of the ipvs backend in 2017:

The net result of this is that the iptables backend of 2026, while still not as good as the ipvs backend, is at least much better than the iptables backend of 2017 was, and many people who found iptables unusable in the past would not have problems with it now.

As with switching to nftables, switching to iptables is mostly just a matter of updating your kube-proxy configuration, although for technical reasons, in this case we recommend that you reboot each node after changing the kube-proxy configuration, rather than merely restarting kube-proxy.

Learn more

16 Nov 2026 6:35pm GMT

29 Oct 2026

feedKubernetes Blog

From Working Group to SIG Architecture: spotlight on AI Conformance

In this SIG Architecture spotlight we talked with Janet Kuo (Google) and Yuan Tang (TensorScale AI) about the journey of AI Conformance in Kubernetes, from the initial Working Group to the integration SIG Architecture as a subproject.

Introductions

Frederico Muñoz and Kirti Goyal (FK): Hi Janet, Yuan, and thank you for the opportunity to learn a bit more about AI Conformance. Can you tell us a bit about yourselves and how you got involved in Kubernetes?

Janet Kuo: I'm a Senior Staff Software Engineer at Google and have been a Kubernetes maintainer since 2015, joining the community just as we were racing toward the 1.0 launch. I started my journey writing the core workload controllers that standardized how applications are orchestrated on Kubernetes. Today, I lead SIG Apps as a Co-chair & TL, and also lead the AI Conformance program. My current focus is to ensure Kubernetes remains the best platform for the next generation of workloads, such as AI.

Yuan Tang: I'm a co-founder of TensorScale AI where we build AI infrastructure for world models. Prior to that, I was a Senior Principal Software Engineer at Red Hat, where I led the development of Red Hat AI platform and upstream open source strategy. I became involved in Kubernetes early on by fostering collaborations between the Kubernetes community and projects I was leading, including Kubeflow, Argo, and KServe. I later helped found and lead WG Serving and the AI Conformance program, advancing innovation and strengthening community collaboration around running AI workloads on Kubernetes.

Program, Working Group, Subproject: the many shapes of AI Conformance

FK: AI Conformance started as a CNCF Program, was it then you got involved?

Janet Kuo: I was actually involved from day one since I pitched the original idea. It became obvious then that everyone was starting to run AI on Kubernetes, but vendors were building their infrastructure stacks in different ways. I got worried about ecosystem fragmentation. So I wrote up the initial Kubernetes AI Conformance Proposal, took it to the CNCF leadership to get their buy-in, and started the AI Conformance Program. Once we had the green light, I reached out to my fellow co-chairs Yuan Tang, Rita Zhang, and Mario Fahlandt to get the community rallied around it.

Yuan Tang: Yes. I was a co-chair of WG Serving at that time and some of the initiatives such as the Gateway AI Inference Extension quickly gained traction. There was a strong interest among the community to include inference-related requirements in the Kubernetes AI Conformance program.

FK: How would you describe the motivation behind the creation of the AI Conformance Working Group?

Janet and Yuan: The motivation behind creating the Kubernetes AI Conformance Working Group (AI WG) was to address the rapid convergence of AI workloads onto Kubernetes while preventing ecosystem fragmentation.

As AI adoption accelerated, organizations began running increasingly complex workloads, including distributed training, large-scale inference, GPU-intensive pipelines, and emerging agentic AI systems across many different Kubernetes platforms. However, vendors were implementing AI infrastructure capabilities differently, especially around GPUs, networking, scheduling, and inference orchestration. That created portability challenges, operational inconsistency, and growing vendor lock-in concerns.

The AI WG was formed to bring the community together around:

FK: WGs are temporary by nature, based on outcomes: would you say the main goals of the WG were achieved?

Janet and Yuan: Yes, the WG achieved its primary initial goals.

The WG has succeeded in moving the ecosystem from: fragmented, vendor-specific AI infrastructure approaches to a shared, community-defined baseline for AI workloads on Kubernetes.

Some concrete indicators of success include:

Most importantly, the WG created consensus that AI workloads are now a first-class Kubernetes use case.

KubeCon stage with 4 persons in it, one of them talking.

FK: AI Conformance was specifically highlighted at the latest KubeCon: is this a reflection of the growing importance of AI throughout the project?

Janet and Yuan: Yes, the prominence of AI Conformance at the latest KubeCon + CloudNativeCon is a strong reflection of how central AI has become to the Kubernetes ecosystem overall.

A few years ago, AI workloads on Kubernetes were still viewed as somewhat specialized, mostly tied to ML platforms or GPU-heavy research environments. What's changed is that AI is now becoming a mainstream infrastructure concern for nearly every large-scale platform operator.

That shift naturally elevates AI from an application running on Kubernetes to a core workload category Kubernetes must optimize for.

The AI Conformance initiative reflects that evolution. It signals that the community now sees AI infrastructure requirements, things like accelerator scheduling, high-performance networking, workload-aware orchestration, and low-latency inference, as important enough to standardize at the ecosystem level.

Its visibility at KubeCon also shows a broader strategic realization: if Kubernetes is going to remain the universal control plane for modern infrastructure, it needs to provide a consistent foundation for AI workloads across clouds, on-prem, and edge environments.

So the emphasis on AI Conformance is not just about chasing AI hype; it's about Kubernetes adapting to a major shift in workload patterns and ensuring the ecosystem evolves cohesively rather than fragmenting into vendor-specific AI stacks.

(slide) Initial adopters and major contributors: Akamai (logo), Alibaba Cloud (logo), AWS (logo), Broadcom (logo), CoreWeave (logo), DaoCloud (logo), Google Cloud (logo), Kubermatic (logo), Microsoft Azure (logo), NVIDIA (logo), Oracle (logo), SUSE (logo), Red Hat (logo), SAP (logo)

From Working Group to SIG Architecture subproject

FK: How would you describe the change from WG to SIG? Is it a reflection of AI as something that requires ongoing architectural governance?

Janet and Yuan: In Kubernetes, Working Groups are meant to be temporary. Our WG had a specific goal to get the initial Kubernetes AI Conformance requirements published and get the certification process off the ground. Once we hit those milestones, we archived the WG and moved AI Conformance to be a permanent subproject under SIG Architecture.

Moving to SIG Architecture definitely proves that AI needs ongoing governance. AI isn't just a passing trend or a one-off workload. It has become a core part of what Kubernetes runs every day. Because SIG Architecture already owns overall Kubernetes Conformance and API design principles, housing the AI Conformance subproject there makes perfect sense. It makes sure our AI Conformance requirements stay aligned with the rest of the Kubernetes ecosystem for the long haul.

FK: And with that change, how would you describe the main goals and areas of intervention of the subproject?

Janet and Yuan: Now that we're a subproject, our main goal is to keep the AI Conformance standards current, vendor-neutral, and standard-setting. We want to make sure Kubernetes captures the latest advancements in AI infrastructure while keeping the ecosystem fair for everyone. When we find gaps where no clear community pattern exists, we help foster and form new standards to solve them.

AI Conformance and Kubernetes

FK: AI Infrastructure is evolving very quickly, while Kubernetes APIs are designed to remain stable for years. Is this something you consider a challenge?

Janet and Yuan: It's definitely a tricky balance. AI moves very fast, while Kubernetes is designed to be stable and boring. But we actually structured the program around this exact tension.

First, we use a tiered graduation model. When new AI infrastructure concepts emerge, we introduce them as recommended SHOULD requirements. That gives the ecosystem room to experiment and agree on best practices without locking Kubernetes into rigid APIs too early. Once those APIs stabilize and with ecosystem adoption, we graduate them to mandatory MUST requirements.

Second, we intentionally keep our focus on the infrastructure layer. We test for core capabilities like accelerator support through Dynamic Resource Allocation (DRA), rather than getting bogged down in specific AI frameworks or model servers. That keeps the base platform stable while letting the AI tooling evolve as fast as it needs to.

FK: DRA has evolved, also due to the work of WG Device Management, as an important part of Kubernetes (and especially so for AI workloads), with SIG Architecture having an important role in how it's integrated in Kubernetes: do you find any parallel in this? Specifically, do you think there will be a heavier focus in the integration with the evolving Kubernetes architecture as part of SIG Architecture?

Janet and Yuan: There is a huge parallel there. Whenever we face a massive, complicated challenge that touches multiple parts of the codebase and involves multiple SIGs, a Working Group is the best place to start. It lets experts from different SIGs get together to figure out the problem, and build consensus.

Being part of SIG Architecture means there will be a much heavier focus on how AI capabilities weave into the evolving Kubernetes architecture. Instead of treating AI as an add-on layer, we're looking at how core Kubernetes primitives, like scheduling and resource allocation, need to adapt at a foundational level to support the next decade of AI workloads.

Final remarks

FK: For interested contributors, what's the best way to get involved with the AI Conformance subproject?

Janet and Yuan: We're always looking for new contributors. The easiest way to get started is by checking out our main repo at kubernetes-sigs/ai-conformance and join our meetings. If you're interested in defining new AI Conformance requirements, you can jump into PR reviews or propose your own using our KAR template. If you want to write code, we always need help expanding our automated Go test suites in the test/ directory.

For vendors looking to certify their platforms, all the details are at cncf/k8s-ai-conformance.

FK: Any final comments you would like to share?

Janet and Yuan: We've been blown away by how fast the community has rallied around this. When we first announced the AI Conformance program during the keynote at KubeCon NA 2025, we weren't sure what the response would be. But by the time we gave our update keynote at KubeCon EU 2026, we had grown from 18 certified platforms to 31 in just a few months.

That kind of growth shows just how much the industry wanted a unified, vendor-neutral standard. It gives organizations the confidence that they can run their AI workloads on any certification Kubernetes platform without getting locked into a proprietary stack.

If you're a platform vendor, we'd love to see you get certified. If you're a developer interested in the intersection of AI and cloud-native infrastructure, come join us in SIG Architecture!

FK: Excellent, thank you both for guiding us in this journey on AI and AI Conformance in Kubernetes.

29 Oct 2026 5:00am GMT

27 Oct 2026

feedKubernetes Blog

Announcing the Workload Conformance Working Group

We are excited to announce that the Kubernetes community now has a Workload Conformance Working Group. The working group is writing a shared standard for how workloads behave at runtime on Kubernetes. To get it right, we need input from the people who build those workloads and from the people who run the clusters they're deployed on.

Why workloads need a shared standard

Certified Kubernetes tests the cluster, and Kubernetes AI Conformance tests whether a cluster can run AI and ML workloads. Nothing comparable exists for the workloads themselves, so cluster operators still check each application by hand before routine maintenance or a Kubernetes upgrade.

The problems they find are familiar. A routine node drain is often how you discover that an application can't tolerate disruption. An upgrade can break an application that still depends on a removed API version. Others show up in day-to-day operation: workloads that ask for privileges they don't need, or reserve resources they never use.

Kubernetes already documents much of this, including security and handling disruptions. Even so, every cluster operator ends up repeating the same assessment, and application maintainers get the same questions from each environment they support. A common specification with repeatable tests would give both groups the same baseline to start from.

What the working group will deliver

The charter sets out three deliverables:

The first step is deciding which workloads and behaviors the specification covers. The scope includes security, resilience during disruptions, resource use, networking, storage, API usage, and observability, but the group hasn't yet agreed on specific requirements or on how to test them. However, one rule is already set: every requirement needs an objective test that a test binary can run, so nothing subjective goes into the specification.

The specification covers how a workload runs, and importantly not what it does. Consequently, application logic is out of scope of workload conformance. The binary won't retest anything Certified Kubernetes already covers, and it won't require a particular workload packaging format, or vendor tool like policy engine, security scanner, etc.

How the work is organized

Two Special Interest Groups (SIGs) are sponsors for the working group.

Whenever a requirement touches another Kubernetes SIG's area, such as node behavior or networking, the working group will initiate conversations with those SIGs as well.

The working group is a forum where SIGs can get involved and shape the direction of Workload Conformance while the specification is being written. We're aiming for a stable first specification and working tests within two to three Kubernetes release cycles.

Get involved

The specification will and needs to be built from real examples, so we want to hear your experiences. If you run clusters, tell us how you assess workloads today and what broke during your last upgrade or maintenance window. If you maintain an application, tell us what it needs from the cluster and where the existing guidance left you guessing.

You can find us in #wg-workload-conformance on Kubernetes Slack (visit slack.k8s.io for an invitation) and on the mailing list. The working group meets every other Wednesday at 15:00 UTC. The Zoom link and the agenda are in the meeting notes. If you want to raise a topic, add it to the agenda. Joining the mailing list usually adds the meeting invite to your calendar.

Thanks to everyone who reviewed and supported the working group proposal.

27 Oct 2026 4:30am GMT