Understanding NFS operation types

During my day, I spend a lot of time analyzing NetApp performance statistics in an effort to help my customers better understand what their Filers are doing. Most of my customers prefer NFS, so that’s what I usually focus on. But just as every customer is different, so is every customer’s environment: they have different infrastructure to serve different data.

As the NFS article on Wikipedia explains, NFS traffic is actually made up of a variety of different operations, or “server procedures” in RFC language. There are reads, of course, and there are writes, but there are other things too: metadata needs to be looked up, file ownerships examined, directories enumerated and so forth. And you may well be surprised at how much of your Filer workload is spent doing “other” (i.e., neither reads nor writes) work.

Here is a graph of a Filer that does a lot of reads (blue) and writes (yellow), but look what else:

2013-03-05_1437

That Filer does an average of 1,500 IOps just on one type of metadata operation – in this case, GETATTR. So what are the various types of operations, and what do they do? Let’s have a look at operations valid in NFSv1 and NFSv2. Note that not all operations are shown in the graph, as they are rarely occuring:

  • GETATTR: return the attributes of a file. E.g. file type, permission, size, ownership, last access time etc.
  • SETATTR: set the atttributes of a file. E.g. permission, ownership, size, last access time etc.
  • STATFS: return the status of a file system. E.g., the output of df.
  • LOOKUP: lookup a file. E.g., when you open a file, you receive the file handle.
  • READ: read from a file.
  • WRITE: write do a file.
  • CREATE: create a file.
  • REMOVE: delete a file.
  • RENAME: rename a file.
  • LINK: create a hard link to a file.
  • SYMLINK: create a soft link to a file.
  • READLINK: read a symbolic link. E.g., find out where the link goes.
  • MKDIR: make a directory.
  • RMDIR: delete a directory.
  • READDIR: read a directory. E.g., the output of ls.

NFSv3 accepts the above operations, as well as:

  • ACCESS: check file access permissions.
  • MKNOD: create a UNIX special device.
  • READDIRPLUS: return the names of files in a directory and their attributes.
  • FSSTAT: return dynamic information about a filesystem. E.g., the filesystem mountpoint and capacity.
  • FSINFO: return static information about a filesystem. E.g., the maximum read & write size the filesystem supports.
  • PATHCONF: return POSIX.1 information about a file. E.g., whether or not the file has case sensitivity.
  • COMMIT: commit previous async writes to storage.

(I stole most of these summaries from W. Richard Stevens’ invaluable TCP/IP Illustrated, Volume 1. The NFS v3 RFC document is also very helpful.)

READDIRPLUS is in bold because it’s one of the most useful metadata operations available in NFSv3. Now hold on, as in my next post I’ll introduce tcpdump outputs.

Advertisements

2 thoughts on “Understanding NFS operation types

  1. What’s the practical difference between ACCESS and GETATTR? I haven’t been able to find a comprehensive guide that explains the difference in detail (that is, “why is there ACCESS when GETATTR seems to do the same thing?” I’ve seen many systems where many clients/apps requests these metadata types the most).

    Thanks, solid intro!

    Reply
    • Andy, thanks for the post. I’m going to be posting some screenshots with tcpdump outputs so you can see what happens and when. I have to say, it is surprising just how much traffic is generated by a relatively simple command!

      Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s