Replacing Nutanix with Proxmox - Storage Replacement?
Posted by metricmoose@reddit | sysadmin | View on Reddit | 11 comments
We've been running a Nutanix cluster for about 7 years, we've pushed off replacing it with some mid-life memory upgrades but now we're feeling the age with the CPU, memory, and spinning drives so we're finally looking to upgrade.
We're looking to move away from Nutanix since the licensing seems to double at every renewal, and with the hardware we're looking at (All SSD, doubled storage/cores compared to the current cluster), we've been quoted licensing that costs 3 times as much as the hardware, and who knows what it will be in 3 years. We're not using a lot of the fancy features, but do like that it's at least one cohesive package.
The Nutanix environment is a 3 node cluster with an identical 3 nodes in a disaster recovery site in another city, which we replicate the primary cluster to using the built-in data protection feature. We also run free Proxmox on a couple smaller, simple clusters (Using built-in VM replication between nodes with local storage) and I quite like it, but of course one of the many differences between Proxmox and Nutanix is that Nutanix has all the fancy storage backend that handles replicating storage between nodes, dedupe, and replication to the DR site.
For us to consider Proxmox as a replacement for our primary clusters, I've been searching for a replacement for the storage component of Nutanix that will somewhat mirror our current setup, and so far I've found Starwind VSAN and tried out the free version on a few spare old machines, and it seems to do alright, but lacks the replication between clusters.
Is there another storage solution that would work for this? Should I look at VM-level replication with another tool like Veeam? SANs have been a blindspot for me since I've always assumed they're for huge orgs, but do standalone SANs make sense for relatively smaller environments (Under 10TB of disk utilized, about 60 VMs)?
marcorr@reddit
Well, as MSP we have customers migrating from ESXi to Proxmox and for small clusters (2-3 nodes), star wind vsan fits good for them. Also, I should mention about their assistance during migration to Proxmox. I would try to cover VM replication using Veeam, however, I am not sure if it was already implemented.
ParkerGuitarGuy@reddit
What are customers doing about support? I see you can purchase Enterprise support but the hours look really limited. I'd hate to have something shit the bed during an after hours maintenance window and not even begin to get the help I need until staff are rolling in and can't work. I use Proxmox all the time in the home lab and it seems to be stable, but I can't bring myself to go with something that doesn't have stellar support. Or does StarWind support cover that sort of thing?
pdp10@reddit
NFS-protocol NAS has proven by far to be the simplest and most elegant storage for vSphere and KVM/QEMU (= Proxmox). The NFS fileshare protocol handles locking and access control, so that a shared/clustered filesystem like VMFS or OCFS2 is superfluous. It's unfortunate that Hyper-V doesn't support NFS storage like all of its competitors do.
NowThatHappened@reddit
ZFS maybe? but SAN is not technically just for big orgs, but its can be an expensive option. You sound like you've got a fairly small setup, 3 nodes in two clusters so maybe snapshot replication? It really depends on how quickly you need to switch? Are you wanting high availability to automatically switch to the backup, or something manual which can be easily achieved with proxmox built in tools, or PBS or Veeam, etc.
TheFluffiestRedditor@reddit
Not ZFS within Proxmox, maybe an NFS server running ZFS internally. Proxmox+Ceph feels like the closest to a hyper-converged solution for OP.
metricmoose@reddit (OP)
I'd want high availability within the cluster (Which is pretty straight forward with shared storage like Starwinds or whatever else), but failing over to another cluster is currently manual and that's fine. If the cluster falls apart or we have some hefty maintenance where we'd need to punt the VMs to the DR cluster, I expect some kind of manual intervention for that.
NowThatHappened@reddit
Ok, then you can do that within proxmox. Consider shared storage of iSCSI, NFS, Ceph etc, and then HA will be no problem. You can replicate from one prox cluster to another using built in tools, or PBS or a third party like veeam.
teeweehoo@reddit
The best way to do this is Proxmox with Ceph. Ceph has builtin replication, but Proxmox lacks the tooling to automate setup and recovery. Also for Ceph I'd suggest 4-5 servers, since you must have 2 mon servers running for VMs to work.
There are also some proxmox partners that offer solutions for this, though I have no experience with them.
https://www.proxmox.com/en/partners/all/filter/partners/partner/partner-type-filter/partner-type/solution-partner?f=6
basicallybasshead@reddit
While I was reading your post, I thought about the same design: using Starwinds VSAN for the local storage replica and Veeam for VM replication in a disaster recovery case. I believe it’s the best design - simple, robust, and guaranteed to work. Regarding storage replication between sites in active-active mode, it will always require a stable and fast connection, otherwise, only active-passive replication or VM-level replication have to be used.
nekimbej@reddit
Proxmox with Ceph sounds like a good fit. I'm not exactly sure how to achieve the warm site however, but maybe you should consider talking to a Proxmox partner about that.
https://www.proxmox.com/images/download/pve/docs/Proxmox-VE-Ceph-Benchmark-202312-rev0.pdf
thenogli@reddit
For one cluster there is no need for 3rd party software. Ceph is built into Proxmox and is very powerful. You will need to have a strong network connection between the hosts, at least 25Gbit/s is a must. Better go for 100Gbot/s. If you have three nodes Ceph will not be at its best, but you can save some bucks for expensive 100Gbit/s switches by going for a mesh network. I'm running this setup for my company and it works for two years like a charm.