During my day, I spend a lot of time analyzing NetApp performance statistics in an effort to help my customers better understand what their Filers are doing. Most of my customers prefer NFS, so that’s what I usually focus on. But just as every customer is different, so is every customer’s environment: they have different infrastructure to serve different data.
As the NFS article on Wikipedia explains, NFS traffic is actually made up of a variety of different operations, or “server procedures” in RFC language. There are reads, of course, and there are writes, but there are other things too: metadata needs to be looked up, file ownerships examined, directories enumerated and so forth. And you may well be surprised at how much of your Filer workload is spent doing “other” (i.e., neither reads nor writes) work.
Here is a graph of a Filer that does a lot of reads (blue) and writes (yellow), but look what else:
That Filer does an average of 1,500 IOps just on one type of metadata operation – in this case, GETATTR. So what are the various types of operations, and what do they do? Let’s have a look at operations valid in NFSv1 and NFSv2. Note that not all operations are shown in the graph, as they are rarely occuring:
- GETATTR: return the attributes of a file. E.g. file type, permission, size, ownership, last access time etc.
- SETATTR: set the atttributes of a file. E.g. permission, ownership, size, last access time etc.
- STATFS: return the status of a file system. E.g., the output of df.
- LOOKUP: lookup a file. E.g., when you open a file, you receive the file handle.
- READ: read from a file.
- WRITE: write do a file.
- CREATE: create a file.
- REMOVE: delete a file.
- RENAME: rename a file.
- LINK: create a hard link to a file.
- SYMLINK: create a soft link to a file.
- READLINK: read a symbolic link. E.g., find out where the link goes.
- MKDIR: make a directory.
- RMDIR: delete a directory.
- READDIR: read a directory. E.g., the output of ls.
NFSv3 accepts the above operations, as well as:
- ACCESS: check file access permissions.
- MKNOD: create a UNIX special device.
- READDIRPLUS: return the names of files in a directory and their attributes.
- FSSTAT: return dynamic information about a filesystem. E.g., the filesystem mountpoint and capacity.
- FSINFO: return static information about a filesystem. E.g., the maximum read & write size the filesystem supports.
- PATHCONF: return POSIX.1 information about a file. E.g., whether or not the file has case sensitivity.
- COMMIT: commit previous async writes to storage.
(I stole most of these summaries from W. Richard Stevens’ invaluable TCP/IP Illustrated, Volume 1. The NFS v3 RFC document is also very helpful.)
READDIRPLUS is in bold because it’s one of the most useful metadata operations available in NFSv3. Now hold on, as in my next post I’ll introduce tcpdump outputs.