Listen to this Post
Introduction:
In the world of virtualization, relying on High Availability (HA) within a single datacenter is a gamble; if the entire site goes dark due to a power outage or network failure, your services go with it. True business continuity requires a multi-site Disaster Recovery (DR) strategy. By leveraging Proxmox VE combined with Ceph’s RBD snapshot-based mirroring, administrators can create a resilient, active-passive infrastructure that allows for rapid failover between geographically separated datacenters without relying on proprietary vendor lock-in.
Learning Objectives:
- Understand the architecture of Ceph RBD Mirroring between two independent Proxmox clusters.
- Learn how to configure snapshot-based replication for Virtual Machine disk images.
- Master the steps for performing a manual failover and failback to ensure service continuity.
You Should Know:
- Understanding the Architecture: Proxmox and Ceph RBD Mirroring
The foundation of this DR setup relies on treating your primary datacenter (Site A) as the production environment and the secondary site (Site B) as the DR standby. In a standard Proxmox+Ceph integration, VM disks are stored as RBD (RADOS Block Device) images within a Ceph pool. To replicate this data off-site, we utilize Ceph RBD Mirroring in snapshot-based mode.
Unlike synchronous replication (which requires low-latency links and mirrors every write), snapshot-based mirroring periodically replicates only the changes between snapshots. This reduces the performance overhead on the production cluster and is far more forgiving of higher latency between sites. On the DR site, the images are kept in a `non-primary` state, meaning they cannot be modified by the secondary Proxmox cluster, preventing data corruption or split-brain scenarios.
2. Prerequisites and Initial Configuration
Before configuring mirroring, you must ensure both Proxmox clusters are healthy and can communicate with each other.
Step-by-step guide:
- Network Connectivity: Ensure network connectivity between the Ceph public networks of both sites. Firewalls must allow traffic on Ceph ports (typically 6789 and 6800-7300).
- Ceph Client Installation: Install the Ceph common packages on the Proxmox nodes if not already present.
On all Proxmox nodes (both sites) apt update && apt install ceph-common -y
- Bootstrap Peering: You need to establish a trust relationship between the two Ceph clusters. This involves exchanging keys.
– On Site A (Primary), extract the client name and key:
ceph auth get client.rbd-mirror.site-a
– On Site B (Secondary), create a similar user and import the key from Site A to establish the peer.
– Note: Modern Ceph versions allow bootstrapping via `rbd mirror pool peer bootstrap create` commands to simplify key exchange.
3. Configuring Snapshot-Based Mirroring on Ceph Pools
Once the clusters are peered, you must enable mirroring on the specific pool(s) containing your VM disks (e.g., vm-storage-pool).
Step-by-step guide:
- Enable Mirroring on the Pool (Site A – Primary):
rbd mirror pool enable vm-storage-pool snapshot
- Add the Secondary Cluster as a Peer (Site A):
rbd mirror pool peer add vm-storage-pool client.rbd-mirror-peer@ceph-dr-site-b
- Schedule Snapshots (Site A): To ensure consistency, schedule snapshots for the images you wish to replicate. You can set this per image or for the entire pool.
Schedule snapshots every 5 minutes for the pool rbd mirror snapshot schedule add --pool vm-storage-pool --interval 5m
4. Verify Status:
rbd mirror pool info vm-storage-pool rbd mirror pool status vm-storage-pool
This will show the health of the mirroring and the daemon status.
4. Managing VM Configuration Files
Ceph replicates the disk data, but it does not automatically replicate the Proxmox VM configuration files (stored in /etc/pve/qemu-server/). To boot a VM on the DR site, you need these configuration files available.
Step-by-step guide:
- Manual Sync (Basic): Use `rsync` to copy configuration files from Site A to Site B regularly.
On Site B DR node (pulling from Site A) rsync -avz root@proxmox-site-a:/etc/pve/qemu-server/ /etc/pve/qemu-server/
- Automation: This sync should be automated via a cron job or a simple script. However, you must ensure the VMIDs do not conflict with existing VMs on the DR site. It is best practice to reserve the same VMID ranges on both clusters for DR purposes.
5. Performing a Failover (Promote DR Site)
When Site A fails, you must promote the `non-primary` images on Site B to `primary` to allow writes and start the VMs.
Step-by-step guide:
- Stop Services (if possible): If Site A is down, this is moot. If it’s a planned failover, shut down the VMs gracefully on Site A.
- Promote Images on Site B: For each VM disk, you must promote the image.
On Site B (DR Cluster) rbd mirror image promote vm-storage-pool/vm-100-disk-0
To force promotion in a disaster scenario (where the primary is unreachable), use the `–force` flag:
rbd mirror image promote --force vm-storage-pool/vm-100-disk-0
- Start VMs: On the Proxmox DR cluster, navigate to the VMs (whose configs were synced) and start them. They will now write data to the DR site’s Ceph cluster.
6. Performing a Failback (Restore Primary)
Once Site A is restored, you need to reverse the replication direction and move services back.
Step-by-step guide:
- Synchronize Changes: On Site A (now the secondary), you must ensure the images are up to date. You may need to re-establish mirroring in the opposite direction.
- Demote DR and Promote Original: On Site B, demote the images to allow them to become read-only replicas again.
On Site B rbd mirror image demote vm-storage-pool/vm-100-disk-0
- Promote Original: On Site A, promote the images back to primary.
On Site A rbd mirror image promote vm-storage-pool/vm-100-disk-0
- Resume Snapshots: Ensure the snapshot schedules are active on Site A. Shut down VMs on Site B and restart them on Site A once the data is fully synced.
What Undercode Say:
- Simplicity is Security: Using Ceph’s native snapshot mirroring within Proxmox eliminates the complexity of third-party replication tools, reducing the attack surface and administrative overhead.
- The Configuration Gap is the Weakest Link: Replicating disk data is useless if you forget to sync the VM configuration files and network mappings. A DR runbook must include configuration management.
- Test or Fail: As highlighted in the comments, tools exist to simulate node or site loss for compliance (like ISO 27001). Regular, documented failover tests are not optional; they are the only way to ensure the DR process works when a real incident occurs.
Prediction:
As hybrid-cloud and edge computing expand, we will see a shift towards “Active-Active” stretched clusters rather than simple “Active-Passive” DR. However, until networking latency and split-brain consensus algorithms (like those being developed in Ceph Quincy and later releases) become universally reliable and affordable, snapshot-based mirroring between Proxmox datacenters will remain the gold standard for SME resilience. The integration of AI-driven log analysis will soon automate the detection of site failure and trigger these failover runbooks without human intervention.
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Morteza Vasaraki – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
