We currently have three VMware clusters: production, secure and test-dev. Production and secure are both ESX 3.5 U4 running NFS against a NetApp FAS3020; test-dev is a ESX 4 cluster running NFS & iSCSI against a NetApp R200. While I’m pretty happy with the FAS3020’s performance, they are certainly not cheap — we’re paying (around) $36k for a shelf that gives us around 8TB usable. You don’t need a calculator to tell you that’s more than $4000/TB. Of course, that price includes support, and NetApp support for us has been nothing short of extraordinary. The R200, by comparison, is quite slow as it’s rapidly approaching it’s EOS and EOL terms. So, before the end of the year it’s going to be replaced.
However — and this came up at the recent New England VMUG — we are reaching an interesting paradigm in software, particularly virtualization, where the software itself (in this case, VMware vSphere) is doing enough of the “smarts” (e.g., snapshots and thin-provisioning) that you have to wonder if the price and performance you’re getting out of your filers makes sense from a cost/benefit viewpoint. While it’s great to have our NetApps do the snapshots and thin-provisioned volumes, it is a little wasteful; restoring from a snapshot typically means using something like rsync to pull back the portion of the NFS datastore you want, then either create a new VM or move the existing VM’s VMDKs out of the way. It’s not terrible, but it could be better.
For this reason, and some others, I’m going to give one of these Sun Fire x4540 storage servers a shot. They are mighty powerful in terms of CPU speed compared to a NetApp filer (six-core AMD Opteron. And you can have more than one!), but decidedly lacking in the traditional software support. However, you do get ZFS, and as you can read at Ben’s Cuddletech blog, you can do some pretty cool things with it. At any rate, VMware Data Recovery lets you do incrementalized, snapshot-based backups from within vSphere itself; it ships as a vCenter plug-in and looks to be a great thing. So, I’m going to give one of these Sun Fire servers a shot and see how I think it goes.
At this point, I’ve ordered a trial of the base model (8 CPUs, 32GB of RAM, 12TB of disk!) to see how it performs. I imagine I’ll leave Solaris on it, and simply use it to front some iSCSI and/or NFS LUNs (and store them in ZFS.) I presume that I’ll let VMware do most of the storage shenanigans via Data Recovery and its native support for snapshots, though I’ll probably take a look at how ZFS can do some of these things, too.