A couple of days ago, I helped a customer get their cDOT system up and running using SAN storage. They had inherited a Cisco MDS switch running NX-OS, and were having trouble getting the devices to log in to the fabric.
As you may know, Clustered Data ONTAP ”’requires”’ NPIV when using Fiber Channel storage — i.e., hosts connecting to a NetApp cluster via the Fiber Channel protocol. NPIV is N-Port ID Virtualization for Fiber Channel, and should not be confused with NPV — which is simply N-Port Virtualization. Scott Lowe has an excellent blog post comparing and contrasting the two.
NetApp uses NPIV in order to abstract away the underlying hardware (i.e., FC HBAs) to the client-facing hardware (i.e., Storage Virtual Machine Logical Interfaces). The use of logical interfaces, or LIFs, allows us to not only carve up a single physical HBA port into many logical ports, but also for the WWPNs to be different. This is particularly useful when it comes to zoning — if you buy an HBA today, you’ll create your FC zone based on ”’the LIF WWPNs”’ and not the HBA’s.
For example, I have a two-node FAS3170 cluster, and each node has two FC HBAs:
dot82cm::*> fcp adapter show -fields fc-wwnn,fc-wwpn node adapter fc-wwnn fc-wwpn ---------- ------- ----------------------- ----------------------- dot82cm-01 2a 50:0a:09:80:89:6a:bd:4d 50:0a:09:81:89:6a:bd:4d dot82cm-01 2b 50:0a:09:80:89:6a:bd:4d 50:0a:09:82:89:6a:bd:4d
(Note that that command needs to be run in a privileged mode in the cluster shell.) But the LIFs have different port addresses, thanks to NPIV:
dot82cm::> net int show -vserver vs1
(network interface show)
Logical Status Network Current Current Is
Vserver Interface Admin/Oper Address/Mask Node Port Home
----------- ---------- ---------- ------------------ ------------- ------- ----
vs1
dot82cm-01_fc_lif_1
up/up 20:05:00:a0:98:0d:e7:76
dot82cm-01 2a true
dot82cm-01_fc_lif_2
up/up 20:06:00:a0:98:0d:e7:76
dot82cm-01 2b true
So, I have one Vserver (sorry, SVM!) with four LIFs. If I ever remove my dual-port 8Gb FC HBA and replace them with, say, a dual-port 16Gb FC HBA, the port names on the LIFs that are attached to the SVM will ”not” change. So when you zone your FC switch, you’ll use the LIF WWPNs.
Speaking of FC switches, let’s look at what we need. I’m using a Cisco Nexus 5020 in my lab, which means I’ll need the NPIV (not NPV!) license enabled. To verify if you have that license, it’s pretty simple:
nane-nx5020-sw# show feature | i npiv npiv 1 enabled
That’s pretty much it. For a basic fabric configuration on a Nexus, you need the following to work with cluster-mode:
- An NPIV license
- A Virtual Storage Area Network, or VSAN
- A zoneset
- A zone
I’m using a VSAN of 101; for most environments the default VSAN is VSAN 1. I have a single zoneset, which contains a single zone. I’m using aliases to make the zone slightly easier to manage.
Here is the zoneset:
nane-nx5020-sw# show zoneset brief vsan 101 zoneset name vsan101-zoneset vsan 101 zone vsan101-zone
You can see that the zoneset is named vsan101-zoneset, and it’s in VSAN 101. The member zone is rather creatively named vsan101-zone. Let’s look at the zone’s members:
nane-nx5020-sw# show zone vsan 101
zone name vsan101-zone vsan 101
fcalias name ucs-esxi-1-vmhba1 vsan 101
pwwn 20:00:00:25:b5:00:00:1a
fcalias name dot82cm-01_fc_lif_1 vsan 101
pwwn 20:05:00:a0:98:0d:e7:76
Note that I have two hosts defined by aliases, and that those aliases contain the relevant WWPN from the host. Make sure you commit your zone changes and activate your zoneset!
Once you’ve configured your switch appropriately, you need to do three four things from the NetApp perspective:
- Create an initiator group
- Populate that initiator group with the host’s WWPNs
- Create a LUN
- Map the LUN to the relative initiator group
When creating your initiator group, you’ll need to select a host type. This will ensure the correct ALUA settings, amongst others. After the initiator group is populated, it should look something like this:
dot82cm::> igroup show -vserver vs1 Vserver Igroup Protocol OS Type Initiators --------- ------------ -------- -------- ------------------------------------ vs1 vm5_fcp_igrp fcp vmware 20:00:00:25:b5:00:00:1a
We’re almost there! Now all we need to do is map the initiator group to a LUN. I’ve already done this for one LUN:
dot82cm::> lun show -vserver vs1
Vserver Path State Mapped Type Size
--------- ------------------------------- ------- -------- -------- --------
vs1 /vol/vm5_fcp_volume/vm5_fcp_lun1
online mapped vmware 250GB
We can see that the LUN is mapped, but how do we know which initiator group it’s mapped to?
dot82cm::> lun mapped show -vserver vs1 Vserver Path Igroup LUN ID Protocol ---------- ---------------------------------------- ------- ------ -------- vs1 /vol/vm5_fcp_volume/vm5_fcp_lun1 vm5_fcp_igrp 0 fcp
Now we have all the pieces in place! We have a Vserver (or SVM), vs1. It contains a volume, vm5_fcp_volume, which in turn contains a single LUN, vm5_fcp_lun1. That LUN is mapped to an initiator group called vm5_fcp_igrp of type vmware, over protocol FCP. And that initiator group contains a single WWPN that corresponds to the WWPN of my ESXi host.
Clear as mud?
