I have been successfully using consumer SSD in my ESXi for few years. After migration to Proxmox, I have tried to use ZFS for virtual machines. First few weeks I was very happy about it. Then I noticed very fast increasing SSD wearout levels on my new SSDs. Proxmox support forums and reddit are full of users mentioning proxmox is destroying SSD, but I thought this is just rumor. Where is the true?
According to my ZFS benchmarks with enterprise SSDs I’ve seen a write amplification between factor 3 (big async sequential writes) and factor 81 (4k random sync writes) with an average of factor 20 with my homeserver measured over months.
In my setup, I have just ZFS mirror, no raidz. I am running few virtual machines:
- zabbix, including mysql
- nextcloud (with another mysql instance)
- web & mail hosting
- samba fileserver
- Windows VM
This VMs gives nearly 1MB/s writes in average. But real writes to SSD are about 3.7-3.8 times higher!!! due ZFS write amplification. In this situation consumer SSD will die very soon. This is graph of one SSD disk in ZFS mirror:
And after migration to LVM (@mdadm raid1) and you can see difference. VM workload is same:
Don’t use consumer SSD with ZFS for virtualization. You can kill you drives in less then year, depending on your workload. I think Proxmox team should put big warning in their wiki about this problem. Their wiki is nearly prefect, but is missing this key point about ZFS and (consumer) SSD wear problem. It is only mentioned in ZFS SSD benchmark faq:
To quote the PVE ZFS Benchmark paper FAQ page 8 again:
Can I use consumer or pro-sumer SSDs, as these are much cheaper than enterprise-class SSDs?
No. Never. These SSDs wont provide the required performance, reliability or endurance. See the fio results from before and/or run your own fio tests.