From the previous graph, we can see that when the ESX host does not have enough pRAM — and, by nature, that the VMs do not have unfettered access to enough vRAM — that when the memory balloon inflates, performance goes to hell. Note that I deliberately put only a small amount of pRAM in the ESX host, and then slightly overcommitted the vRAM in the two VMs. I did this for a reason; I wanted to see deliberately have one VM with a reservation actively use all its memory (via the very useful stress) tool, which will in turn cause the other VM — with no reservation for its vRAM — to suffer as soon as the first VM begins a memory-intensive workload.
I got the results that I was expecting in so far as the 2nd VM, with little vRAM, suffers drastically when it must make its memory balloon calls to disk. Now, the only thing left to do was to prove that I can address the problem by adding more pRAM to the ESX host. So, I added another 4GB of pRAM to the ESX host (bringing it to a total of 8GB), and left the two VMs as they were (one with 3GB of vRAM assigned & 2.5GB reserved, the other with 1GB of vRAM and 0GB reserved.)
The results were what I expected — because the ESX host has enough RAM to allow the second, starving VM to balloon to RAM instead of NFS disk, its performance when it balloons (due to the first VM having a high memory reservation and a memory-intensive workload) is virtually the same as when it doesn’t balloon.
I suppose that, for now at least, the moral of the story is that you have to watch your performance!
:-) Seriously though, just because there is some amount of free pRAM on the ESX host doesn’t mean the VMs can necessarily access it in a manner that keeps performance at acceptable levels. If you need to balloon your memory, you want to do it on the absolutely fastest storage (hopefully pRAM) that you can. Abstracting this away and doing it to disk via swap is going to have pretty serious performance implications; if you have many VMs all doing this at once then it is going to get very ugly very quickly.