Getting started with Data ONTAP cluster-mode

First things first: this post (and subsequent posts) are built off the backs of many great NetApp employees, especially the PS organization and the cmode-ses mailing list. In particular I’d like to thank Luke Mun, Doug Moore, Barry Spencer and Pavel Sobiezczuk.

This post will describe how to convert a system that’s currently running Data ONTAP 8.1 7-mode into 8.1 cluster-mode. If you’re using 8.0 7-mode, I strongly recommend upgrading to 8.1 7-mode before the conversion: there’ll be a couple less steps involved. I’ll assume you’re using a Cisco Nexus 55xx switch as the private cluster interconnect. While this isn’t required per se, it is the only option that’s properly supported.

The Cisco switch setup is actually quite simple. You can get the full, NetApp-approved configuration from the NetApp support site here. Here are the relevant global options for the switch(es):

feature lacp

errdisable recovery interval 30
errdisable recovery cause pause-rate-limit

policy-map type network-qos cluster
  class type network-qos class-default
    mtu 9216
system qos
  service-policy type network-qos cluster

spanning-tree port type edge default
port-channel load-balance ethernet source-dest-port

You can see that we’re not doing a whole lot: enabling LACP (the Link Aggregation Contro Protocol), building a basic QoS policy and setting some error parameters. Now, here are the relevant port options for cluster nodes:

interface Ethernet1/1
  description Cluster Node 1
  no lldp transmit
  no lldp receive
  spanning-tree port type edge
  spanning-tree bpduguard enable

Here, you can see we’re disabling LLDP (the Link Layer Discovery Protocol) and configuring the switch ports for spanning-tree protocol. Last but not least, if you’re connecting two Cisco Nexus switches together, here are the relevant options for the inter-switch link:

interface Ethernet1/13
  description Inter-Cluster Switch ISL Port 13 (Port-channel)
  no lldp transmit
  no lldp receive
  switchport mode trunk
  channel-group 1 mode active

Not much to see here, either: disable LLDP, define STP (this time as trunk nodes) and note that the link is part of a channel group (i.e. an aggregated link). Once you’ve done the Cisco configuration, you’re off to the races!

From here, we boot the Filer. Once you hit the LOADER prompt, hit Ctrl-C to interrupt the boot process. Enter these options to configure the Filer to boot cluster-mode now and in the future:

setenv bootarg.init.boot_clustered true
setenv bootarg.init.usebootp false
setenv bootarg.bsdportname e0a

After you’ve set these options, boot Data ONTAP by entering boot_ontap. Once Data ONTAP starts booting, hit Ctrl-C again to get to the installation options. You’ll see a menu like this:

(1) Normal Boot.
(2) Boot without /etc/rc.
(3) Change password.
(4) Clean configuration and initialize all disks.
(5) Maintenance mode boot.
(6) Update flash from backup config.
(7) Install new software first.
(8) Reboot node.

Select option (4) Clean configuration and initialize all disks. This will re-zero the disks, which can take an hour or two. After re-zeroing, your Filer will reboot automatically and proceed to the setup wizard.

And that’s enough for now!