More on mixing ESX & ESXi in the same cluster

A while back I wrote this post, which detailed some of my struggles as I moved from ESX to ESXi at the same time as we conducted some network reconfiguration. Now that vSphere 5 is almost ready to go gold, it’s imperative that I finish off our ESX->ESXi conversions as vSphere 5 is going to be ESXi-only. Previously, I had written the following:

So, now we have a mismatch in the architecture between ESX and ESXi.

While that is technically true, it turns out that it doesn’t really matter. Yes, your Service Console (ESX) is now your vmk0 (ESXi), which means your VMkernel(s) are going to be vmkN where N>0 because the VMkernels get created after the Management Network (ESXi’s new name for the Service Console). Now, this presents you with a small problem that you’ll see appear as this error dialog:

The vMotion interface of the destination host uses network ‘vmk1’, which differs from the network ‘vmk0’ used by the vMotion interface of the source host. Are you sure you want to continue? Yes / No

Normally, if you put a host in Maintenance Mode, the VMs will vMotion off said host automatically. However, if you see this warning when manually trying to vMotion a VM, they will not vMotion automatically as the host goes into Maintenance Mode! They will just sit there and be surly. However, if you vMotion the VMs yourself, it will be successful! You simply have to click Yes when the warning appears.

What I used to do, and what I now recommend against you doing, is removing the Management Network from your new ESXi host, making [a] new VMkernel(s) and then re-creating your Management Network in order to have your device names “match” — i.e., vmk0 on your ESX host is a VMkernel just like vmk0 on your ESXi host is a VMkernel. You don’t need to do it! Sure, you won’t be able to migrate your machines simply by putting them into Maintenance Mode, but the steps involved are a PITA and require iLO or physical console access. After all, how can you remove your Management Network in ESXi while you’re logged in to it? 🙂

Another thing to keep in mind is that your datastores must be mapped the same in ESXi as they were on your ESX servers — at least, if you don’t want to have to power off every VM before you migrate it. Here’s what I mean: I have two hosts, ESX and ESXi, and both are connected to an NFS datastore with the name nfsprod. When I try and vMotion a VM between the ESX and the ESXi host, it fails with the not-particularly-descriptive error message:

A general system error occurred: the source detected that the destination failed to resume

If you look in the VMkernel error log, you probably won’t see anything descriptive (at least, I didn’t when I looked.) But this blog post came up from The Googles, and it contained the answer I was looking for. Here’s what you see at the hypervisor level — remember, this is a datastore that appears as nfsprod on both the ESX & ESXi via the vSphere Client. Check it out:

[root@ESX ~]# esxcfg-nas -l
nfsprod is /vol/nfsprod from mounted

And now, the ESXi host:

~ # esxcfg-nas -l
nfsprod is /vol/nfsprod from blender

See? One is named via its IP, and one is named via its FQDN. And that is enough to make the vMotions fail!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s