Listen to this Post
Introduction:
Keeping a Proxmox VE cluster patched with the latest security updates while avoiding VM downtime is a persistent challenge for sysadmins and DevOps engineers. Native Proxmox tools lack a built-in rolling patch orchestration feature, forcing administrators to manually drain nodes, migrate VMs, apply updates, and reboot – a process prone to error and extended maintenance windows. ProxPatch, a lightweight open-source automation tool, fills this gap by sequentially patching cluster nodes, automatically migrating workloads, and conditionally rebooting only when necessary, ensuring continuous cluster availability.
Learning Objectives:
- Automate rolling patch management across multi-node Proxmox clusters using ProxPatch without external dependencies.
- Execute live VM migration, node update sequencing, and post-reboot validation through native Proxmox tools (
pvesh,qm,ssh). - Integrate ProxPatch with ProxLB for post-patch load balancing and audit the patch workflow for compliance and security hardening.
You Should Know:
1. Understanding ProxPatch Architecture and Pre-Flight Checks
ProxPatch is a pure-shell orchestration tool that leverages Proxmox’s native CLI and API endpoints. It requires passwordless SSH access between nodes and a working `pvesh` (Proxmox VE shell) context. Before deployment, verify cluster health and quorum.
Step-by-step guide:
- Ensure all nodes can resolve each other’s hostnames and have SSH key‑based authentication.
On each Proxmox node ssh-keygen -t rsa -b 4096 -N "" -f ~/.ssh/id_rsa ssh-copy-id root@<other-node-ip>
- Test quorum and node status:
pvecm status Look for "Quorate: Yes" and list nodes
- Download ProxPatch from its GitHub repository (linked below). The script requires no extra frameworks or API tokens.
git clone https://github.com/<repository> replace with actual repo after expansion cd proxpatch chmod +x proxpatch.sh
- Review the configuration block inside the script (e.g., `NODES` array, migration timeouts, reboot detection).
2. Rolling Patch Execution – Live Migration Sequence
ProxPatch processes nodes one by one. For each node, it drains VMs and containers, applies system updates (apt-get), checks if a reboot is needed, and returns the node to service before moving to the next.
Step-by-step guide:
- Run the patcher dry‑run to see what would happen:
./proxpatch.sh --dry-run
- For live execution:
./proxpatch.sh --update
- Under the hood, ProxPatch executes commands like:
Migrate all running VMs from current node to another available node for vmid in $(pvesh get /nodes/$NODE/qemu --output-format json | jq -r '.[].vmid'); do qm migrate $vmid <target_node> --online done
- After migration, the node is updated:
apt update && apt upgrade -y
- Reboot is triggered only if `/var/run/reboot-required` exists:
if [ -f /var/run/reboot-required ]; then reboot fi
- Wait for the node to come back online with:
while ! ping -c1 $NODE; do sleep 5; done
- Proceed to next node automatically.
- Integrating ProxPatch with ProxLB for Post‑Patch Load Balancing
After the cluster is fully patched, VM distribution may be unbalanced because migrations leave VMs concentrated on the first few nodes. ProxLB (another open‑source tool) re‑balances VMs based on CPU/memory usage.
Step‑by‑step guide:
- Install ProxLB on a management machine or one of the nodes:
git clone https://github.com/gh2o/proxlb cd proxlb pip install -r requirements.txt
- Configure ProxLB to connect to your Proxmox cluster (API token or user/pass).
- Run ProxLB after finishing ProxPatch:
python3 proxlb.py --mode loadbalance --strategy most_utilized
- For full automation, chain the commands:
./proxpatch.sh --update && python3 proxlb.py --mode loadbalance
- This ensures optimal resource usage and reduces cross‑node network overhead.
4. Security Hardening for ProxPatch Operations
Since ProxPatch requires root‑level SSH access to migrate VMs and update packages, enforce strict security controls to prevent lateral movement.
Step‑by‑step guide:
- Restrict SSH access using `AllowUsers` in `/etc/ssh/sshd_config` and use certificate‑based authentication.
- Run ProxPatch from a dedicated jump host with audit logging:
Log every action ./proxpatch.sh --update 2>&1 | tee /var/log/proxpatch-$(date +%F).log
- Implement `sudo` granularity instead of full root SSH: allow specific commands (
qm migrate,pvesh,apt) for the patching user. - Monitor ProxPatch activity with auditd:
auditctl -w /usr/local/bin/proxpatch.sh -p wa -k proxpatch_activity
- Schedule patching during low‑traffic windows and require manual approval for critical production clusters.
5. Troubleshooting Common Failures (Migration Stuck, No Quorum)
ProxPatch may fail if a VM cannot migrate (e.g., local CDROM mounted, incompatible CPU flags). The tool’s error handling can be extended.
Step‑by‑step guide:
- Before running ProxPatch, check for migration blockers:
Find VMs with local resources for vmid in $(qm list | awk 'NR>1 {print $1}'); do qm config $vmid | grep -E "local|cifs|nfs" done - If a VM is stuck during live migration, fallback to offline migration (stop VM, move disk, start):
qm stop $vmid qm move-disk $vmid <target_storage> qm start $vmid --migrate
- For quorum loss after a node reboot, manually re-establish:
pvecm expected 1 then add node back pvecm add <node_ip>
- ProxPatch can be extended with a retry loop: wrap migration commands in a `while` loop with exponential backoff.
- ProxPatch for Homelab vs Production – Configuration Differences
In homelabs, you may want aggressive parallel patching; in production, strict serialization with health checks is mandatory.
Step‑by‑step guide – Production settings:
- Set `ROLLBACK_ENABLED=true` in ProxPatch config – after each node patch, run smoke tests (e.g., ping a test VM).
- Define a custom maintenance window using cron:
Every Sunday at 2 AM 0 2 0 /root/proxpatch/proxpatch.sh --update --safe-mode
- For production, enable `BLOCK_MIGRATE_FAIL=yes` to halt if any VM cannot migrate.
- Homelab example – patch all nodes simultaneously (not recommended for prod):
for node in node1 node2 node3; do ssh $node "apt update && apt upgrade -y" & done wait
What Undercode Say:
- Key Takeaway 1: ProxPatch eliminates manual node draining and VM migration errors by automating rolling patches through native Proxmox tooling, making it ideal for both homelab enthusiasts and production sysadmins.
- Key Takeaway 2: Combining ProxPatch with ProxLB ensures not only security updates but also post‑patch load balancing, closing the automation loop for cluster health.
Analysis: The absence of a native rolling update mechanism in Proxmox VE forces administrators to either accept downtime or develop fragile scripts. ProxPatch’s lightweight, dependency‑free approach respects the Unix philosophy – do one thing well. However, organizations must still harden SSH access and implement audit trails to prevent privilege abuse. The tool’s focus on predictability over speed aligns with security‑first patching strategies, yet it could benefit from native API token support to reduce reliance on root SSH. For now, it fills a critical gap and demonstrates how community tooling often outpaces vendor solutions in niche automation.
Prediction:
As Proxmox gains traction in enterprise edge computing and HCI environments, demand for certified, out‑of‑the‑box rolling patch orchestration will grow. Either the Proxmox team will integrate a native “rolling update” mode (similar to Kubernetes’ rollout strategy), or tools like ProxPatch will evolve into full‑fledged cluster lifecycle managers with integrated vulnerability scanning and compliance reporting. Expect to see ProxPatch’s approach influence future Proxmox APIs – especially endpoints for coordinated node reboots and maintenance mode flags. For now, adoption will rise among cost‑conscious DevOps teams running Proxmox in production, making this script a hidden gem in the infrastructure automation landscape.
GitHub Repository (original post): https://lnkd.in/dfeJEmdB (expand via LinkedIn)
▶️ Related Video (82% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Nusretonen Opensource – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
