Listen to this Post
Introduction:
The virtualization landscape is witnessing a paradigm shift with the emergence of Incus OS, an immutable operating system announced in late 2025 by Stéphane Graber that strips away the traditional shell access in favor of an API-only management model. This radical approach promises bit-for-bit identical nodes, enhanced security through the elimination of SSH and sudo, and streamlined operations via authenticated REST API calls. However, the transition from conventional hypervisors like Proxmox to this shell-less environment presents unique challenges that demand a complete rethinking of infrastructure management workflows.
Learning Objectives:
- Understand the architecture and security model of Incus OS, including its immutable base, A/B atomic updates, and reliance on UEFI Secure Boot with TPM 2.0.
- Learn how to deploy a multi-1ode Incus OS cluster entirely through seed configuration files and API calls, without ever accessing a graphical interface or shell.
- Identify common pitfalls in cluster formation, including storage pool misconfigurations and certificate management issues, and apply verified remediation steps.
You Should Know:
1. The Immutable Foundation: Understanding Incus OS Architecture
Incus OS is not merely a distribution where Incus is installed; it is a purpose-built system image where the kernel, ZFS, and Incus are integrated and cryptographically signed, ensuring identical binaries across all machines. Built upon a Debian 13 base, the system partitions are mounted as read-only, with updates applied atomically via an A/B partitioning scheme that allows for automatic rollback on failure. The boot process is secured by UEFI Secure Boot and TPM 2.0, providing full-disk encryption and a verified trust chain.
The core differentiator of Incus OS compared to other immutable operating systems like Talos Linux or Fedora CoreOS is its singular focus on Incus, positioning it as a complete hypervisor platform capable of managing both containers and virtual machines. This focus is part of a broader strategy, the FuturFusion Cloud stack, which aims to provide an open-source exit path from VMware. For administrators, this means a system where configuration drift is impossible, and every node in a cluster is a perfect replica of the others.
- Verify System State (API Call): To confirm the system’s immutability and trust status, you can query the Incus API. Replace `
` with your node’s IP and ensure your client certificate is properly configured. curl -k --cert client.crt --key client.key https://<node-ip>:8443/1.0
This returns a JSON payload. Look for the `”auth”: “trusted”` field to confirm your client is authenticated.
2. Seed-Based Installation: Automating Node Deployment
The installation process for Incus OS deviates from traditional interactive setups. It relies on an ISO image accompanied by small configuration files, known as “seeds,” which are placed on a secondary storage medium labeled SEED_DATA. These seed files drive the entire installation and initial configuration of the node, covering networking, storage, and crucially, the establishment of trust.
The seed is a YAML or similar structured file that pre-configures the system. To deploy a node, you must:
- Prepare the Seed Media: Format a USB drive or virtual disk with the label
SEED_DATA. - Create the Seed Configuration: This file must define the network interface configuration (IP, netmask, gateway), storage pool details (type, source), and the client certificate for API authentication.
- Boot from ISO: Insert both the Incus OS ISO and the seed media, then boot the server. The installer will automatically detect the seed and proceed with an unattended installation.
- Sample Seed Configuration Snippet (Conceptual):
network-config version: 1 config:</li> <li>type: physical name: eth0 subnets:</li> <li>type: static address: 192.168.1.10/24 gateway: 192.168.1.1 dns_nameservers:</li> <li>8.8.8.8
incus-config server: https_address: 192.168.1.10:8443 cluster: cluster_join_address: 192.168.1.1:8443 cluster_certificate: "--BEGIN CERTIFICATE--..." storage_pools:</li> <li>name: default driver: zfs source: rpool/incus
This configuration sets a static IP, prepares the node to join a cluster, and defines a ZFS storage pool.
3. Cluster Formation: API-Driven Node Joining
Forming a cluster in Incus OS is an API-centric process that requires careful preparation. The first node in the cluster is initialized with its own certificate. For subsequent nodes, the seed must include the cluster’s certificate to establish trust without manual intervention. If the certificate is omitted, the node fails to join with the error No target cluster member certificate provided. The client interactive mode resolves this automatically, but the seed application does not.
Once the first node is online, you can add members using the Incus command-line tool from a trusted client:
- Initialize the First Node:
incus admin init
During this process, you will set the cluster name, network address, and generate the cluster certificate.
-
Join a New Node to the Cluster:
On the first node, retrieve the join token:
incus cluster add <new-1ode-1ame>
This command outputs a token or command that must be included in the new node’s seed configuration or passed via the API. For API-only joining, the new node’s seed must contain the cluster certificate and the join address.
- Verify Cluster Status:
incus cluster list
This command should display all nodes with their status. A successful cluster shows all nodes as `ONLINE` and the quorum achieved.
4. Navigating Storage and Disk Constraints
One of the most significant hurdles encountered during deployment is storage configuration. Incus OS imposes specific requirements that differ from standard Incus installations. The target disk for installation must be of type virtio-scsi; using `virtio-blk` causes the installer to fail with the error no potential install devices found. Additionally, the storage pool must be properly defined in the seed. If the source of the storage pool is not explicitly indicated, Incus attempts to create a loop-backed pool, which Incus OS explicitly forbids with the error Loop backed pools aren't supported on IncusOS.
To correctly configure storage:
- Identify the Correct Disk: Ensure your VM or hardware presents the disk as a SCSI device.
- Define the Storage Pool in the Seed: The seed must specify the driver (ZFS is recommended and default) and the source device or dataset.
- Configuring a ZFS Storage Pool Post-Installation (via API):
If you need to add a storage pool after installation, you can use the Incus CLI:incus storage create default zfs source=/dev/sdb
This command creates a ZFS pool named “default” using the `/dev/sdb` device.
5. Security Implications and Operational Realities
The “no shell” policy of Incus OS is a deliberate security design choice, not a deficiency. By eliminating SSH and sudo, the operating system closes a common attack vector and ensures that all management actions are auditable through the API. However, this imposes a significant operational shift. Administrators accustomed to ad-hoc troubleshooting and quick fixes via the shell must now plan all changes in advance, codifying them into seed files or automation scripts.
For production environments, this model offers immense benefits in terms of reproducibility and security. For home labs and learning, however, the official recommendation is to start with a standard Incus installation to gain familiarity before transitioning to the more rigid Incus OS. The prerequisites are also non-1egotiable: Incus OS requires UEFI with Secure Boot and a TPM 2.0 module, which may not be available on all hardware.
- Managing the System Without a Shell:
- Logs: Access logs via the API or by configuring remote logging.
- Monitoring: Integrate with Prometheus or similar tools that can scrape metrics from the Incus API endpoint.
- Updates: System updates are applied atomically. To trigger an update, you typically reboot the node, which will boot into the new A/B partition. If the new partition fails to boot, the system automatically reverts to the previous one.
What Undercode Say:
- Immutability is a double-edged sword: The promise of bit-for-bit identical nodes is revolutionary for consistency and security, but it demands a level of foresight and automation that many teams are not yet prepared for.
- The API is the new shell: The success of Incus OS hinges on the completeness and usability of its REST API. The shift from interactive shells to programmatic API calls is the future of infrastructure management.
Analysis:
The Incus OS experiment reveals a clear trajectory towards fully automated, API-driven infrastructure. The pain points encountered—certificate management, storage misconfiguration, and the absence of a safety net like SSH—are not flaws but features that expose the brittleness of manual operations. The project’s alignment with the FuturFusion stack positions it as a serious contender in the post-VMware landscape, particularly for organizations seeking vendor independence. However, its adoption will be gated by the maturity of its tooling and the willingness of operations teams to abandon decades of shell-based habits. The “no shell” model is a bold statement, but its success will ultimately depend on how well the ecosystem around Incus OS can provide the visibility and control that administrators have come to expect from their hypervisors.
Prediction:
- +1 Incus OS will gain significant traction in greenfield deployments and organizations with mature Infrastructure-as-Code practices, offering a secure and reproducible alternative to traditional hypervisors.
- +1 The FuturFusion stack, including the Operations Center and Migration Manager, will accelerate enterprise adoption by providing the management and migration capabilities required for large-scale production environments.
- -1 The steep learning curve and rigid hardware requirements (UEFI, Secure Boot, TPM 2.0) will limit its immediate appeal to the home lab and small-to-medium business sectors, keeping Proxmox as the dominant choice for those environments in the short term.
- -1 Without robust, user-friendly debugging and observability tools designed for a shell-less environment, troubleshooting complex issues will remain a significant barrier, potentially leading to “cattle not pets” operational models where nodes are simply replaced rather than repaired.
▶️ Related Video (78% Match):
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
IT/Security Reporter URL:
Reported By: Stephanerobert1 Opensource – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
