Deep diving into NetApp NFS operations — file operations

Following on from my last post about NFS operation types, I thought I’d show some basic filesystem commands and then use tcpdump to illustrate what operations they generate on a Filer. Here are some simple commands and their relevant output. Note that I will omit the non-NFS traffic from the tcpdump output. The client is a 32-bit RHEL VM, and the Filer in question is a NetApp FAS3240 running Data ONTAP 8.1.2 7-mode. It’s important to note that, despite the existence of NFS RFCs and associated standards, NFS servers (and clients!) vary in their behavior in the real world. That said, to keep things simple, I am only specifying a handful of mount options:

charles-e0a:/vol/blog	/mnt/blog	nfs	noac,vers=3,tcp,soft,bg	0 0

Now let’s jump into things! First, we’ll mount the /mnt/blog filesystem and see what happens (mount command omitted):


Not much to see when mounting the filesystem — just a bunch of acks. However, if we wait a little while, Linux appears to look at the new filesystem on its own volition:


Note the appearance of two new operations: fsinfo and fsstat. From my previous post, you’ll remember that these commands – not surprisingly, given their names – return information about filesystems. Now, let’s take the next step and cd into the NFS mount:


Nothing. OK, let’s try ls:


Interesting — we see a repeat of the fsstat operation, along with a a couple of new operations: getattr and access. getattr returns file attributes (such as ownership and timestamps), and access returns file access permission information. Note that, although we’re only running ls once, we’ve seen a total of 5 operations travel along the wire to our NFS server. Let’s try and create an empty file with touch:


More operations involving getattr and access, but interestingly no read or write traffic. Is that because the file is empty? Let’s write some data to it:


Even more operations! Now we’re up to a handful. We have the previously-seen getattr and access, along with some new ones: lookup, create and setattr. (I didn’t highlight every operation as it would have made the graphic a little unwieldy.) As their names suggest, lookup looks up a file’s handle, create creates a file and setattr is used to set a file’s attributes.

One curious thing to note is that, right after setting an attribute with a setattr, we go ahead and read it right back with getattr. Also curious is that we still haven’t seen the write operation.

Let’s try writing to a new file:


Finally, we see a write operation — highlighted in red for your viewing pleasure. But we still see a plethora of other operations that we’ve seen before, all to create and write to a single file: getattr, setattr, access, lookup and create.

Now that we have two files in our NFS mount, let’s run an ls and see what we see:


We see all the operations we’ve seen before, and a new operation called readdirplus. As mentioned in my previous post, readdirplus returns the names of a file in a directory and their attributes. Interestingly, we see the output of ls (files bar and foo) returned after four operations, but then we see another six operations (including readdirplus) occur after we’ve already seen the output.

Let’s append to a file that already has data in it:


Not a lot to see here, and no write command — just some access. Is Linux caching my request? Let’s try reading the file back to make sure the data is there:


Huh! Before the content of the file is returned to us, we see three operations: getattr, access and write. Then we see the file’s content, and then we see another getattr and another access.

For the sake of being thorough, let’s run an ls -al to see if the NFS operations are any different to a regular ls:


Yikes. Even though we have only two files in the directory, there are more than 20 operations before we even see the output of the directory (which I cropped from my screenshot). Nothing new, though; just a lot of getattr and access.

Let’s copy a non-empty file and see what happens:


We see largely the same output as our append process. There’s a whole lot of getattr and access, interspersed with a lookup — but no write! In order to see the write command, I’d have to cat the file to read its contents.

Let’s copy a file that should be too big to cache. We’ll use my favorite Linux calculator, bc, and copy it into our NFS mount:


By now, we expect to see something like the above. A bunch of metadata operations (access, getattr, lookup) followed by a bunch of writes. For the record, the bc binary is only ~70KB. Note that the write operations completed while the copy process was taking place – i.e., there is no filesystem caching going on here. Also note that the number of write operations we see is going to depend on the NFS block size we’re using, too.

For something a little different, let’s try to create a new symbolic link (horse) to point to a file that already exists (donkey):


Hmm. Nothing we haven’t seen before — which is weird, because I know a symlink operation exists in the NFS RFCs. Let’s try and look it up with ls -al:


Et voila! Buried in the output of ls -al, we see our new symlink operation, which is of course used to create symbolic links. Let’s try and follow the symlink:


Interesting. We see some typical getattr and access operations, but we do not see the readlink operation. What else can we do? Let’s try changing the permissions on bc:


Not a lot unusual there. We read the permissions attributes in (via access and getattr), then we set the permissions attributes via setattr. How about changing ownership?


Not much difference — it operates pretty much the same way that chmod operates, i.e. the setting of an attribute. Now, let’s try deleting a file:


Only one new operation here, and not much of a surprise that it’s called remove and neither a surprise that its function is to remove a file 🙂 What if I remove a symbolic link? If you remember, horse is a symlink to donkey:


Here we can see the remove operation doesn’t occur until we bother doing an ls. Last but not least, what about renaming a file?


Again, we see the operation (in this case, rename) after the ls. And that’s it for now! Up next: directories. As always, questions and comments and suggestions are welcome.


Understanding NFS operation types

During my day, I spend a lot of time analyzing NetApp performance statistics in an effort to help my customers better understand what their Filers are doing. Most of my customers prefer NFS, so that’s what I usually focus on. But just as every customer is different, so is every customer’s environment: they have different infrastructure to serve different data.

As the NFS article on Wikipedia explains, NFS traffic is actually made up of a variety of different operations, or “server procedures” in RFC language. There are reads, of course, and there are writes, but there are other things too: metadata needs to be looked up, file ownerships examined, directories enumerated and so forth. And you may well be surprised at how much of your Filer workload is spent doing “other” (i.e., neither reads nor writes) work.

Here is a graph of a Filer that does a lot of reads (blue) and writes (yellow), but look what else:


That Filer does an average of 1,500 IOps just on one type of metadata operation – in this case, GETATTR. So what are the various types of operations, and what do they do? Let’s have a look at operations valid in NFSv1 and NFSv2. Note that not all operations are shown in the graph, as they are rarely occuring:

  • GETATTR: return the attributes of a file. E.g. file type, permission, size, ownership, last access time etc.
  • SETATTR: set the atttributes of a file. E.g. permission, ownership, size, last access time etc.
  • STATFS: return the status of a file system. E.g., the output of df.
  • LOOKUP: lookup a file. E.g., when you open a file, you receive the file handle.
  • READ: read from a file.
  • WRITE: write do a file.
  • CREATE: create a file.
  • REMOVE: delete a file.
  • RENAME: rename a file.
  • LINK: create a hard link to a file.
  • SYMLINK: create a soft link to a file.
  • READLINK: read a symbolic link. E.g., find out where the link goes.
  • MKDIR: make a directory.
  • RMDIR: delete a directory.
  • READDIR: read a directory. E.g., the output of ls.

NFSv3 accepts the above operations, as well as:

  • ACCESS: check file access permissions.
  • MKNOD: create a UNIX special device.
  • READDIRPLUS: return the names of files in a directory and their attributes.
  • FSSTAT: return dynamic information about a filesystem. E.g., the filesystem mountpoint and capacity.
  • FSINFO: return static information about a filesystem. E.g., the maximum read & write size the filesystem supports.
  • PATHCONF: return POSIX.1 information about a file. E.g., whether or not the file has case sensitivity.
  • COMMIT: commit previous async writes to storage.

(I stole most of these summaries from W. Richard Stevens’ invaluable TCP/IP Illustrated, Volume 1. The NFS v3 RFC document is also very helpful.)

READDIRPLUS is in bold because it’s one of the most useful metadata operations available in NFSv3. Now hold on, as in my next post I’ll introduce tcpdump outputs.