Listen to this Post
Introduction:
The AI inference landscape is fracturing. Organizations are no longer content with black-box, managed API providers; they are running open-weight models on their own GPU fleets spread across clouds, neoclouds, and on-premise data centers. This fragmentation has forced platform teams to build custom coordination layers by hand—one operator at a time. Enter Modelplane: an open-source control plane built on Crossplane that brings vendor-1eutral, declarative orchestration to AI inference fleets, doing for AI what Kubernetes did for compute and Crossplane did for cloud infrastructure.
Learning Objectives:
- Understand the architectural foundations of Modelplane and its relationship to the Crossplane and Kubernetes ecosystems.
- Learn how to deploy and configure Modelplane to unify heterogeneous GPU infrastructure under a single control plane.
- Master the declarative management of AI inference workloads, including model deployment, auto-scaling, and traffic routing.
- Implement security and policy controls using the inference gateway for cost, compliance, and sovereignty enforcement.
1. Deploying Modelplane on a Local Kind Cluster
Modelplane is software you install and run in your own environment. The quickest way to get started is to deploy it on a local kind (Kubernetes in Docker) cluster. This allows you to experiment with the control plane without provisioning expensive cloud GPUs.
Step‑by‑step guide:
- Install Prerequisites: Ensure you have
kubectl,kind, and `helm` installed on your local machine. - Create a Kind Cluster: Create a cluster with sufficient resources. A simple config file mapping ports for the inference gateway is recommended.
kind-config.yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes:</li> </ol> - role: control-plane extraPortMappings: - containerPort: 30443 hostPort: 8080 protocol: TCP
Run `kind create cluster –config kind-config.yaml`.
- Install Modelplane: Follow the official getting started guide. Typically, this involves applying the Crossplane provider and the Modelplane configuration to the cluster.
Add the Modelplane Helm repository (example) helm repo add modelplane https://charts.modelplane.ai helm repo update helm install modelplane modelplane/modelplane --1amespace modelplane-system --create-1amespace
- Verify Installation: Check that the Modelplane pods are running.
kubectl get pods -1 modelplane-system
Linux/Windows Commands:
- Linux/macOS: `kubectl get crds | grep modelplane` to verify the custom resource definitions are installed.
- Windows (PowerShell): `kubectl get crds | Select-String “modelplane”`
- Declaring Your Inference Fleet with InferenceClusters and InferenceClasses
Once Modelplane is running, platform teams must describe the available GPU fleet. This is done using two custom resources: `InferenceCluster` and
InferenceClass. An `InferenceClass` defines a hardware recipe—the devices a node pool offers and how to provision it. An `InferenceCluster` represents the actual GPU fleet, whether it’s a managed cloud node pool or bare-metal servers.Step‑by‑step guide:
- Define an InferenceClass: Create a YAML manifest that specifies the GPU type and memory requirements.
inference-class.yaml apiVersion: modelplane.ai/v1alpha1 kind: InferenceClass metadata: name: nvidia-a100-40gb spec: hardware: gpu: vendor: nvidia.com model: A100 memory: 40Gi provisioning: nodeSelector: cloud.provider: aws instance-type: p4d.24xlarge
- Define an InferenceCluster: Reference the class and provide the connection details for the cluster.
inference-cluster.yaml apiVersion: modelplane.ai/v1alpha1 kind: InferenceCluster metadata: name: aws-us-east-1 spec: classRef: name: nvidia-a100-40gb connection: endpoint: https://<your-eks-cluster-endpoint> credentials: secretRef: name: aws-credentials key: kubeconfig
3. Apply the Manifests:
kubectl apply -f inference-class.yaml kubectl apply -f inference-cluster.yaml
4. Verify: Check the status of the cluster.
kubectl get inferenceclusters
3. Deploying a Model with a Declarative Manifest
With the fleet defined, developers can deploy models without worrying about the underlying infrastructure. They create a `ModelDeployment` and a
ModelService. Modelplane schedules the replica onto a cluster with free, compatible GPUs and exposes it behind a unified, OpenAI-compatible endpoint.Step‑by‑step guide:
- Create a ModelDeployment: Specify the model, engine image, and replica count. The example below deploys a Qwen model using the vLLM engine.
model-deployment.yaml apiVersion: modelplane.ai/v1alpha1 kind: ModelDeployment metadata: name: qwen-demo namespace: ml-team spec: replicas: 1 engines:</li> </ol> - name: qwen members: - role: Standalone nodeSelector: devices: - name: gpu count: 1 selectors: - cel: device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0 template: spec: containers: - name: engine image: vllm/vllm-openai:v0.23.0 args: ["--model=Qwen/Qwen2.5-0.5B-Instruct"]2. Expose the Model via a ModelService: Create an endpoint that routes traffic to the deployment replicas.
model-service.yaml apiVersion: modelplane.ai/v1alpha1 kind: ModelService metadata: name: qwen namespace: ml-team spec: endpoints: - selector: matchLabels: modelplane.ai/deployment: qwen-demo
3. Apply the Manifests:
kubectl apply -f model-deployment.yaml kubectl apply -f model-service.yaml
4. Test the Endpoint: Once the pod is running, you can send a request to the OpenAI-compatible endpoint.
curl http://localhost:8080/v1/completions \ -H "Content-Type: application/json" \ -d '{"model": "qwen", "prompt": "What is Modelplane?", "max_tokens": 50}'4. Implementing Auto-Scaling and Weight Caching
Modelplane continuously reconciles the fleet toward the declared state. It supports auto-scaling replicas based on load and caches model weights locally to reduce latency.
Step‑by‑step guide for auto-scaling:
- Configure Replica Scaling: While the current v0.1 release uses the standard Kubernetes scale subresource, you can integrate with KEDA (Kubernetes Event-driven Autoscaling) to scale based on custom metrics like request queue length.
- Enable Weight Caching: Modelplane can cache model weights on local storage. This is configured in the `ModelDeployment` spec.
spec: cache: enabled: true storage: 50Gi
- Monitor Caching: Check the status of cached weights.
kubectl get modelcaches -1 ml-team
Linux/Windows Commands:
- Linux: `watch -1 2 kubectl get pods -1 ml-team` to monitor pod scaling in real-time.
- Windows: Use a loop in PowerShell:
while ($true) { kubectl get pods -1 ml-team; Start-Sleep -Seconds 2 }.
5. Securing the Inference Gateway with Policy Controls
The inference gateway is a critical security control point. It routes inference requests and applies cost, compliance, and sovereignty policies. For regulated enterprises, this ensures inference runs inside infrastructure they govern directly.
Step‑by‑step guide:
- Define a Policy: Policies can be defined using OPA (Open Policy Agent) or similar frameworks. The gateway can enforce rate limiting, IP whitelisting, and content filtering.
- Configure the InferenceGateway Resource: Reference the policy in the gateway configuration.
inference-gateway.yaml apiVersion: modelplane.ai/v1alpha1 kind: InferenceGateway metadata: name: secure-gateway spec: policies:</li> </ol> - type: RateLimit value: 1000/minute - type: IPWhitelist value: "192.168.1.0/24" fallback: enabled: true provider: external-ai-provider
3. Apply the Gateway:
kubectl apply -f inference-gateway.yaml
4. Test Security Controls: Attempt to send a request from a non-whitelisted IP and verify it is blocked.
API Security Hardening:
- Authentication: Integrate with OIDC providers to authenticate requests.
- Encryption: Ensure all traffic to the gateway is over TLS.
- Audit Logging: Enable detailed logging for all inference requests for compliance and forensic analysis.
6. Multi-Cloud and Hybrid Deployment Strategies
Modelplane is designed to unify fragmented environments. It can provision clusters across AWS, GCP, Azure, neoclouds, and on-premise data centers.
Step‑by‑step guide:
- Define Multiple InferenceClusters: Create separate cluster resources for each cloud provider.
- Use Node Selectors and Taints: In the
ModelDeployment, use node selectors to pin workloads to specific hardware or locations. - Implement Cross-Cloud Routing: The `ModelService` can route traffic to replicas in different clusters based on latency or cost.
- Disaster Recovery: Configure the gateway to redirect requests to an external inference environment during outages.
Linux/Windows Commands:
- Linux: Use `kubectl config use-context` to switch between different cluster contexts.
- Windows: Use `kubectl config get-contexts` to list available contexts.
7. Troubleshooting and Monitoring Modelplane
Modelplane provides detailed status conditions on its custom resources. Monitoring the control plane is essential for maintaining a healthy inference fleet.
Step‑by‑step guide:
1. Check Resource Status:
kubectl describe modeldeployment qwen-demo -1 ml-team
2. View Logs: Inspect the Modelplane controller logs.
kubectl logs -1 modelplane-system -l app=modelplane
3. Monitor with Prometheus: Modelplane exposes metrics that can be scraped by Prometheus for dashboards in Grafana.
4. Common Issues:
- Out of GPU Memory: The scheduler will fail to place a replica. Check the `InferenceClass` and node capacity.
- Image Pull Errors: Ensure the serving engine image (e.g.,
vllm/vllm-openai) is accessible from the target cluster.
What Undercode Say:
- Key Takeaway 1: Modelplane is not just another orchestration tool; it is a paradigm shift that brings the declarative, GitOps-driven approach of Crossplane to the chaotic world of AI inference. It empowers platform teams to treat GPUs as a schedulable resource, abstracting away the underlying complexity from data scientists.
- Key Takeaway 2: The security and policy controls embedded in the inference gateway are a game-changer for regulated industries. By enforcing cost, compliance, and sovereignty policies at the gateway level, organizations can maintain strict governance over their AI workloads without stifling innovation.
Analysis:
The release of Modelplane addresses a critical pain point in the AI industry: the lack of a standardized, open-source control plane for inference. As open-weight models proliferate, organizations are moving away from vendor lock-in and toward self-managed infrastructure. Modelplane fills this gap by providing a single pane of glass for managing heterogeneous GPU fleets. Its reliance on Crossplane—a CNCF graduated project—ensures enterprise-grade stability and a vibrant ecosystem. However, as a v0.1 release, it is still early days. The community will need to contribute integrations, performance benchmarks, and production hardening. Nevertheless, the trajectory is clear: Modelplane is poised to become the de facto standard for AI inference orchestration.
Prediction:
- +1 Modelplane will accelerate the adoption of open-weight models by making it economically and operationally viable for enterprises to run their own inference fleets, reducing reliance on expensive managed APIs.
- +1 The project will see rapid community growth, similar to Crossplane, leading to a rich ecosystem of providers and composition functions that support a wide array of serving engines and hardware accelerators.
- -1 The complexity of managing multi-cloud GPU infrastructure will remain a barrier for smaller organizations, potentially creating a divide between AI-1ative companies that can afford dedicated platform teams and those that cannot.
- +1 The policy gateway will evolve into a comprehensive AI firewall, capable of detecting and mitigating prompt injection attacks, model theft, and data exfiltration, becoming an essential component of any enterprise AI security strategy.
- -1 Without proper governance, the ease of deploying models via Modelplane could lead to “shadow AI” proliferation, where teams spin up inference workloads without proper cost or security oversight.
▶️ Related Video (76% Match):
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by ThousandsIT/Security Reporter URL:
Reported By: Dlross Modelplane – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅🔐JOIN OUR CYBER WORLD [ CVE News • HackMonitor • UndercodeNews ]
📢 Follow UndercodeTesting & Stay Tuned:
- Install Modelplane: Follow the official getting started guide. Typically, this involves applying the Crossplane provider and the Modelplane configuration to the cluster.
