ESXi on an SD card

Do you know the phrase “don’t put all your eggs in one basket?” Well that’s part of it, but it also has to do with performance needs. The host OS needs very little space and very little IOPs so it makes sense to put it on separate disk (hence SD cards, media known for its incredible speeds /s). Imagine a single host with 50 VMs running, it would require blazing fast storage and a lot of capacity. Commonly you would allocate two disks in RAID-1 for OS, then a bunch in a high performance RAID-10 configuration with maybe one or two hot spares. A common 2U full depth sever will hold a max of 26 2.5” drives, so two for host OS, two hot spares, leaving 22 drives in RAID-10. I think 22x read and 11x write speeds should do the trick. Now, the OS array fails, the VM array is still intact and RTO will be faster, considering only the OS needs to be restored. If the VM data needed to be restored also, RTO would be slower simply. If the VM disk array failed, the host would continue running. You might ask, what good is the host running if all VMs are offline? In my opinion the silver lining of the host remaining up is simply from an observability perspective. Most likely, the host would be continuously monitored and the failure would be quickly detected. From there, you could potentially be able to troubleshoot the host while it’s running. But after a certain size, you move away from single node hosts on to clusters, with either external SAN storage or local storage in a software defined storage configuration. The host OS would obviously need to be separate in that case also. Hell, some servers literally don’t even have hard drives and instead they boot from a SAN. So these were just a few reasons to separate host OS and VM storage. Thank you for coming to my ted talk.