Listen to this Post
Introduction
In a highly available Kubernetes control plane undergoing a rolling upgrade, API servers of different versions coexist for a period of time. This mixed-version state creates a dangerous pitfall: a client request might land on an older API server that doesn’t recognize a newly introduced resource, causing it to return a 404 Not Found error — even though the resource actually exists elsewhere in the cluster. The Mixed Version Proxy (MVP), first introduced as an Alpha feature in Kubernetes 1.28, solves this by intelligently proxying such requests to a peer API server that can handle them. Now, with its graduation to Beta in Kubernetes 1.36 and enabling by default, this feature has matured into an essential safety net for cluster upgrades.
Learning Objectives
- Understand the problem MVP solves: eliminating false 404 errors during mixed-version control plane upgrades
- Learn how MVP evolved from Alpha to Beta, including the shift from StorageVersion API to Aggregated Discovery
- Configure and verify MVP in your Kubernetes 1.36+ clusters using the required flags and feature gates
- Master Peer-Aggregated Discovery and how to inspect both aggregated and local API views
- Apply best practices for zero-downtime upgrades using MVP in production environments
1. Understanding the Mixed Version Proxy Architecture
What Problem Does MVP Solve?
During a rolling upgrade of a high-availability control plane, you will have multiple API servers running different Kubernetes versions simultaneously. These servers serve different sets of APIs (Groups, Versions, Resources). Without MVP, if a client request lands on an API server that does not serve the requested resource — for example, a new API version introduced in the upgrade — that server returns a 404 Not Found.
This is technically incorrect because the resource is available in the cluster, just not on that specific server. The consequences can be severe:
- Mistaken garbage collection — controllers may delete resources they believe no longer exist
- Blocked namespace deletions — the API server fails to finalize deletion
- Failed or stuck upgrade processes — operators waste time debugging phantom issues
How MVP Works
When an API server receives a request for a resource it cannot serve locally, MVP performs the following steps:
- The API server determines it cannot serve the requested resource locally
- It looks up a capable peer in its discovery cache
- It proxies the request to the peer API server, adding the `x-kubernetes-peer-proxied` header for internal tracking
4. The peer processes the request locally
- The peer returns the response to the original API server
- The original API server forwards the response back to the client
This entire process is transparent to the client — the client receives the correct response without ever knowing a proxy occurred.
2. Evolution from Alpha to Beta: Key Improvements
The Beta release in Kubernetes 1.36 brings two major architectural improvements over the Alpha version introduced in 1.28.
From StorageVersion API to Aggregated Discovery
In the Alpha version, API servers relied on the StorageVersion API to determine which peers served which resources. While functional, this approach had a critical limitation: the StorageVersion API does not support CRDs and aggregated APIs.
For Beta, the implementation has been completely overhauled to use Aggregated Discovery. API servers now use aggregated discovery data to dynamically understand the capabilities of their peers, providing full support for:
- Custom Resource Definitions (CRDs)
- Aggregated API servers
- All API groups and versions
Peer-Aggregated Discovery: The Missing Piece
The 1.28 blog post noted a significant gap: while we could proxy resource requests, discovery requests still only showed what the local API server knew about. In 1.36, Peer-Aggregated Discovery has been added.
Now, when a client performs discovery (e.g., listing available APIs using kubectl api-resources), the API server merges its local view with discovery data from all active peers. This provides clients with a complete, unified view of all APIs available across the entire cluster, regardless of which API server they connected to.
Note: Peer-aggregated discovery is enabled if the `–peer-ca-file` flag is set. Otherwise, the server falls back to showing only its local APIs.
3. Configuration and Verification
Enabling MVP in Kubernetes 1.36
In Kubernetes 1.36, the feature gate `UnknownVersionInteroperabilityProxy` is enabled by default. However, for MVP to function, you must configure secure communication between peer API servers.
Required Flags for kube-apiserver
On each API server, set these flags: --peer-ca-file=/path/to/ca.crt --peer-cert-file=/path/to/peer.crt --peer-key-file=/path/to/peer.key
Using kubeadm
For kubeadm-managed clusters, add the following to your `KubeAPIServer` configuration in kubeadm-config.yaml:
apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration apiServer: extraArgs: peer-ca-file: /etc/kubernetes/pki/ca.crt peer-cert-file: /etc/kubernetes/pki/apiserver.crt peer-key-file: /etc/kubernetes/pki/apiserver.key
Verifying MVP is Working
Check the Feature Gate
Check if the feature gate is enabled kubectl get --raw /api/v1/ | jq '.' Or check API server logs for MVP-related entries kubectl logs -1 kube-system kube-apiserver-<pod-1ame> | grep -i "proxy"
Test Cross-Version Resource Access
During a mixed-version upgrade, you can verify MVP is functioning by:
- Creating a resource that only exists in the newer API version
- Directing a request to the older API server (e.g., using
kubectl --server=<old-api-server>) - Observing that the request succeeds despite the older server not serving the resource locally
Inspect the Proxied Request Header
When MVP proxies a request, it adds the `x-kubernetes-peer-proxied` header. You can verify this by enabling verbose logging or inspecting audit logs:
Enable audit logging and look for the header In API server audit policy, add: - level: Metadata users: ["system:apiserver"] verbs: ["get", "list", "watch"] resources: - group: "" resources: [""]
4. Peer-Aggregated Discovery in Action
The Unified API View
With Peer-Aggregated Discovery, clients get a complete view of all APIs in the cluster. This is particularly important for:
- kubectl — command-line autocompletion and resource discovery work correctly regardless of which API server you connect to
- Controllers — they can discover and watch resources that may only exist on newer API servers
- CI/CD pipelines — automated tools can reliably detect available APIs
Viewing Local vs. Aggregated Discovery
There may be cases where you need to inspect only the resources served by the specific API server you are connected to (e.g., for debugging). You can request this non-aggregated view by including the `profile=nopeer` parameter in your request’s Accept header:
Aggregated view (default when --peer-ca-file is set) curl -k https://<api-server>:6443/apis Local-only view (non-aggregated) curl -k -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList;profile=nopeer" \ https://<api-server>:6443/apis
Discovery Cache Management
API servers maintain a discovery cache of peer capabilities. The cache is updated periodically and on-demand. To force a cache refresh:
Restart the API server (cache will rebuild on startup) kubectl delete pod -1 kube-system kube-apiserver-<pod-1ame> Or use the discovery API directly kubectl get --raw /apis
5. Security Considerations and Best Practices
mTLS for Peer Communication
MVP requires mutual TLS (mTLS) between peer API servers for secure communication. The certificates must be valid and trusted by all peers.
Generating Peer Certificates
Generate a CA if you don't have one openssl genrsa -out ca.key 2048 openssl req -1ew -x509 -days 365 -key ca.key -out ca.crt -subj "/CN=kubernetes-ca" Generate peer certificate openssl genrsa -out peer.key 2048 openssl req -1ew -key peer.key -out peer.csr -subj "/CN=kube-apiserver-peer" openssl x509 -req -days 365 -in peer.csr -CA ca.crt -CAkey ca.key -out peer.crt
Firewall and Network Policies
Ensure that API servers can communicate with each other on the appropriate ports (typically 6443 for secure API server communication). If you use network policies:
apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-api-server-peering namespace: kube-system spec: podSelector: matchLabels: component: kube-apiserver policyTypes: - Ingress - Egress ingress: - from: - podSelector: matchLabels: component: kube-apiserver ports: - protocol: TCP port: 6443 egress: - to: - podSelector: matchLabels: component: kube-apiserver ports: - protocol: TCP port: 6443
Audit Logging for Proxied Requests
To track proxied requests for security and compliance:
Audit policy snippet apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata users: ["system:apiserver"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] resources: - group: "" resources: [""] omitStages: - RequestReceived The x-kubernetes-peer-proxied header will appear in audit logs
6. Upgrade Strategies with MVP
Rolling Upgrade Workflow
With MVP enabled by default in Kubernetes 1.36+, the rolling upgrade process becomes significantly safer:
- Upgrade the first API server to the new version
2. Mixed-version state —新旧 API servers coexist
- Client requests to the old API server for new resources are transparently proxied to the new one
- Discovery shows a unified API view across all peers
5. Upgrade remaining API servers one by one
- Full upgrade complete — all API servers now run the new version
Zero-Downtime Upgrade Checklist
- [ ] Verify MVP feature gate is enabled (
UnknownVersionInteroperabilityProxy=true) - [ ] Configure
--peer-ca-file,--peer-cert-file, `–peer-key-file` on all API servers - [ ] Ensure mTLS certificates are valid and trusted
- [ ] Test discovery aggregation with `kubectl api-resources` during the upgrade
- [ ] Monitor API server logs for proxy-related errors
- [ ] Have a rollback plan in case issues arise
Monitoring MVP During Upgrades
Use these commands to monitor MVP activity:
Check API server logs for proxy events kubectl logs -1 kube-system kube-apiserver-<pod-1ame> --tail=100 | grep -E "proxy|404|peer" Monitor API server metrics (if exposed) curl -s http://<api-server>:10251/metrics | grep -E "apiserver_proxy|apiserver_peer" Verify discovery aggregation kubectl get --raw /apis | jq '.groups[].name' | sort
What Undercode Say
- MVP is a game-changer for cluster upgrades — eliminating false 404 errors removes a major source of operational headaches and prevents dangerous side effects like mistaken garbage collection.
- The shift to Aggregated Discovery is critical — it finally brings full CRD and aggregated API support, making MVP viable for real-world production clusters that heavily rely on custom resources.
- Peer-Aggregated Discovery completes the picture — clients now see a unified API view, which is essential for tools like `kubectl` and controllers to function correctly during upgrades.
- Beta status with default enablement means wider adoption — operators no longer need to opt in; the safety net is now automatically available.
- mTLS configuration is non-1egotiable — securing peer communication is essential for production deployments; don’t skip this step.
- MVP is not just for upgrades — it also benefits clusters with heterogeneous control planes, such as those running different Kubernetes versions across availability zones.
- The `profile=nopeer` header is a useful debugging tool — it allows operators to isolate issues by viewing only local APIs when troubleshooting.
- Adoption of MVP encourages better upgrade practices — teams can now adopt more aggressive upgrade schedules with reduced risk, accelerating time-to-market for new features.
Prediction
- +1 MVP will become generally available (GA) in Kubernetes 1.38 or 1.39, with further optimizations to discovery caching and reduced latency for proxied requests.
- +1 Cloud providers (GKE, EKS, AKS) will enable MVP by default in their managed control planes, making it the standard for all cluster upgrades.
- +1 The success of MVP will inspire similar interoperability features for other Kubernetes components, such as the kubelet and controller-manager, during rolling upgrades.
- +1 MVP will reduce upgrade-related incidents by an estimated 60-70%, as the primary cause of upgrade failures (false 404s) is eliminated.
- +1 The feature will become a cornerstone of Kubernetes’ reliability engineering, alongside etcd backups and cluster recovery procedures.
- -1 Organizations that fail to configure mTLS for peer communication will not benefit from MVP, as the feature falls back to local-only discovery without the `–peer-ca-file` flag.
- -1 Clusters with network policies that block API server-to-API server communication will experience broken discovery and proxy functionality, requiring careful network configuration.
- -1 Operators who skip testing MVP in staging environments may encounter unexpected behavior during production upgrades, particularly around discovery cache staleness.
- -1 The added complexity of managing peer certificates may be a barrier for smaller teams, though this is mitigated by tools like kubeadm that automate certificate generation.
- -1 Legacy clusters running versions prior to 1.36 will need to upgrade to benefit, leaving some organizations behind if they delay Kubernetes version upgrades.
▶️ Related Video (72% Match):
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
IT/Security Reporter URL:
Reported By: Alina Babayeva – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
