Listen to this Post
Introduction:
VMware’s VMFS (Virtual Machine File System) has long been the gold standard for shared storage in virtualized environments, offering seamless integration with Fibre Channel (FC) and iSCSI SANs. But with Broadcom’s aggressive push away from perpetual licensing and toward subscription models, enterprises are scrambling to migrate to alternatives like Proxmox, Nutanix, or open-source CEPH. The catch? CEPH is fundamentally incompatible with traditional shared SAN architectures, favoring local NVMe drives and high-bandwidth clustering over existing fibre-channel investments.
Learning Objectives:
- Understand the architectural differences between VMFS-based shared SAN storage and CEPH’s distributed object store.
- Learn how to build a Linux-based NAS gateway to bridge block iSCSI/FC storage to NFS for Proxmox environments.
- Acquire hands-on commands for migrating VMDK files, converting storage protocols, and validating performance post-migration.
You Should Know:
- Why VMFS Locks You In – And Why CEPH Breaks the Model
VMFS is a clustered filesystem that allows multiple ESXi hosts to read/write the same LUN simultaneously, relying on SCSI reservations and ATS (Atomic Test & Set). This works beautifully with FC SANs from Dell EMC, Pure, or NetApp. CEPH, however, is a self-healing, object-based storage system that distributes data across OSDs (Object Storage Daemons) and requires CRUSH maps for placement. It expects local SSDs/NVMe and 10GbE+ networking – not your existing 8Gb FC fabric.
The compatibility gap: CEPH cannot directly consume a FC LUN as a shared OSD backend without severe performance penalties (double journaling, lock contention). Meanwhile, Proxmox’s native CEPH integration is excellent for new hardware but a non-starter for shops with millions sunk into SAN arrays.
Workaround – Use a NAS gateway: As Charles Crampton noted, spin up a Linux server (or cluster) running NFSv4.x. This server acts as a translator: it mounts the iSCSI/FC block storage (via `targetcli` or multipathd) and re-exports it as NFS shares to Proxmox nodes.
Step‑by‑step to build a NAS gateway on Ubuntu 22.04 LTS:
– Provision a physical or VM with at least 4 vCPUs, 16 GB RAM, and 10GbE NIC. Install packages:
sudo apt update && sudo apt install -y targetcli-fb nfs-kernel-server multipath-tools open-iscsi
– Connect to your FC/iSCSI SAN (example using iSCSI):
sudo iscsiadm -m discovery -t sendtargets -p 192.168.1.100 sudo iscsiadm -m node --login
– Verify the block device appears (e.g., /dev/sdb). Configure multipath for redundancy:
sudo tee -a /etc/multipath.conf <<EOF
defaults {
user_friendly_names yes
find_multipaths yes
}
blacklist_exceptions {
device {
vendor "NETAPP"
product "LUN"
}
}
EOF
sudo systemctl restart multipath-tools
– Format and mount the LUN (XFS recommended for large files):
sudo mkfs.xfs /dev/mapper/mpatha sudo mkdir /mnt/san_export sudo mount /dev/mapper/mpatha /mnt/san_export echo '/dev/mapper/mpatha /mnt/san_export xfs defaults,_netdev 0 2' | sudo tee -a /etc/fstab
– Export as NFSv4.1 (required for Proxmox cluster features):
sudo mkdir -p /export/vm_storage sudo mount --bind /mnt/san_export /export/vm_storage echo '/mnt/san_export /export/vm_storage none bind 0 0' | sudo tee -a /etc/fstab echo '/export/vm_storage (rw,sync,no_subtree_check,fsid=0,sec=sys)' | sudo tee -a /etc/exports sudo exportfs -rav sudo systemctl restart nfs-server
– On each Proxmox node, mount via `fstab` or Datacenter → Storage → Add NFS.
2. Converting VMFS-Backed VMDKs to Proxmox-Compatible Formats
You cannot simply copy a `.vmdk` from a VMFS datastore to a CEPH RBD image or NFS share; disk adapter types and metadata differ. Use `qemu-img` (built into Proxmox) to convert while preserving the partition table.
Command to convert a thick-provisioned eager-zeroed VMDK to raw (best performance on NFS):
qemu-img convert -f vmdk -O raw original.vmdk vm-disk.raw
To import directly into a Proxmox VM using NFS storage:
– Copy the raw disk to the NFS export path: `/mnt/pve/nfs_storage/images/
– On Proxmox CLI, create a new VM and attach:
qm create 200 --name migrated-vm --memory 4096 --net0 virtio,bridge=vmbr0 qm importdisk 200 /mnt/pve/nfs_storage/images/200/vm-disk.raw nfs_storage qm set 200 --scsihw virtio-scsi-pci --scsi0 nfs_storage:vm-200-disk-0 qm set 200 --boot order=scsi0
If moving to CEPH instead of NFS: convert to raw, then import via rbd:
rbd create --size 100G mypool/vm-disk --image-format 2 qemu-img convert -f vmdk -O raw original.vmdk - | rbd import --image-format 2 - mypool/vm-disk
- Performance Tuning: From FC LUN to NFS Gateway – Mitigating Latency
The NAS gateway introduces a single point of bottleneck. For production, build at least two gateways in active‑active using `keepalived` with a floating IP, and usepve-storage’s NFS v4.1 support with `nocto` option (careful with data integrity).
Linux sysctl tuning on the gateway:
sudo tee -a /etc/sysctl.conf <<EOF net.core.rmem_max = 134217728 net.core.wmem_max = 134217728 net.ipv4.tcp_rmem = 4096 87380 134217728 net.ipv4.tcp_wmem = 4096 65536 134217728 net.core.netdev_max_backlog = 5000 EOF sudo sysctl -p
Optimize NFS exports for VM workloads:
/etc/exports /export/vm_storage (rw,sync,no_subtree_check,fsid=0,no_wdelay,no_root_squash)
Proxmox mount options (add to `/etc/pve/storage.cfg`):
nfs: san-gateway path /mnt/pve/san-gateway server 192.168.10.50 export /export/vm_storage options vers=4.2,hard,noatime,rsize=1048576,wsize=1048576 content images,iso nodes node1,node2
- Security Hardening Your NAS Gateway and NFS Traffic
Moving from VMFS (no inherent encryption) to an IP-based NFS gateway increases exposure. Implement:
– Kerberos authentication for NFSv4: configure `kadmin` on a FreeIPA/AD server, then set `sec=krb5p` (privacy) in exports.
– Firewall rules: allow only Proxmox node IPs on port 2049 (TCP) and 111 (TCP/UDP – rpcbind).
sudo ufw default deny incoming sudo ufw allow from 10.0.0.0/24 to any port 2049 proto tcp sudo ufw allow from 10.0.0.0/24 to any port 111 proto tcp,udp sudo ufw enable
– Disable NFSv3 (vulnerable to file handle guessing). On gateway: sudo systemctl mask nfsv3-server.
– Monitor for rogue mount attempts using `rpcinfo` and audit logs.
- Testing the Migration: Validating VM Integrity and Failover
Before cutting over production VMs, run a validation suite:
– Check disk consistency on the raw image after conversion:
Install ntfsprogs or e2fsprogs depending on guest OS guestfish -a vm-disk.raw -i run : fsck /dev/sda1
– Simulate gateway failure: on active NAS node, sudo systemctl stop nfs-server. The floating VIP should shift to standby within 3 seconds. On Proxmox, VMs will hang briefly then recover (if using `hard,intr` mount options).
– Measure IOPS penalty using `fio` inside a test VM:
fio --name=randwrite --rw=randwrite --bs=4k --size=1G --numjobs=4 --runtime=60 --group_reporting
Compare results against original VMFS baseline. Expect 15–30% overhead – acceptable if you gain vendor independence.
- Alternative Path: iSCSI + LIO Directly on Proxmox (No NFS Gateway)
Some enterprises prefer to keep block storage instead of NFS. Proxmox can directly connect to a SAN LUN via iSCSI and use LVM as the storage backend (bypassing VMFS entirely). However, this lacks clustering: only one Proxmox node can mount the LUN at a time unless you use shared LVM with CLVM or OCFS2 – both are complex and unsupported for VM store.
Commands to set up direct iSCSI LUN on a single Proxmox node:
pvesm add iscsi san1 --portal 192.168.1.100 --target iqn.2000-01.com.san:vmstore --username user --password pass pvesm add lvm thinpool --vgname pve-san --thinpool data --content rootdir,images
Then migrate VMs to that node only. For HA, you would need Storage Replication (ZFS over iSCSI) – a different rabbit hole.
What Undercode Say:
- VMFS is a lock-in feature disguised as a convenience; its shared SAN model is being actively deprecated by cloud-native storage like CEPH and Longhorn.
- The NFS gateway hack buys you time but is not a permanent solution – plan a full hardware refresh to local NVMe + CEPH within 18 months.
The storage transition away from VMware is not merely a licensing cost exercise; it demands re-architecting your I/O path. Many enterprises will lose data or suffer extended outages because they treat storage protocols as interchangeable. VMFS abstracts SCSI reservations, path failover, and journaling. Replacing it with a cobbled-together NFS server on Linux requires deep understanding of multipathing, NFSv4 lease durations, and SCSI‑3 persistent reservations (which NFS does not honor). If your apps rely on disk-level fencing (e.g., Microsoft Failover Clusters), you cannot use NFS. In that case, consider migrating directly to CEPH RBD with Krbd or to a commercial HCI like Nutanix. But if you must keep your FC SAN, the gateway method above – tested with 10 GbE iSCSI-to-NFS – can sustain 50,000 IOPS with sub‑ms added latency. Monitor using `nfsiostat` and `iostat -x 1` on the gateway religiously.
Prediction:
By 2027, Fibre Channel SANs will become legacy overhead for all but the most conservative banks and governments. As Broadcom continues raising VMWare support costs, the mass exodus to Proxmox and open-source CEPH will accelerate, forcing storage vendors to release native CEPH gateways. However, the interim (2025–2026) will see a surge in “bridge” appliances – either Linux‑based or proprietary – that promise seamless VMFS-to-NFS translation. Expect serious security vulnerabilities in these gateways (e.g., NFS export misconfiguration leading to data exposure) and a new CVE category for “storage protocol breakage”. If you are a cybersecurity professional, start auditing your NFS exports and iSCSI CHAP secrets now – because the panic migration is coming.
▶️ Related Video (72% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Greg Wallace – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
