Deep diving into NetApp NFS operations — file operations

Following on from my last post about NFS operation types, I thought I’d show some basic filesystem commands and then use tcpdump to illustrate what operations they generate on a Filer. Here are some simple commands and their relevant output. Note that I will omit the non-NFS traffic from the tcpdump output. The client is a 32-bit RHEL VM, and the Filer in question is a NetApp FAS3240 running Data ONTAP 8.1.2 7-mode. It’s important to note that, despite the existence of NFS RFCs and associated standards, NFS servers (and clients!) vary in their behavior in the real world. That said, to keep things simple, I am only specifying a handful of mount options:

charles-e0a:/vol/blog	/mnt/blog	nfs	noac,vers=3,tcp,soft,bg	0 0

Now let’s jump into things! First, we’ll mount the /mnt/blog filesystem and see what happens (mount command omitted):


Not much to see when mounting the filesystem — just a bunch of acks. However, if we wait a little while, Linux appears to look at the new filesystem on its own volition:


Note the appearance of two new operations: fsinfo and fsstat. From my previous post, you’ll remember that these commands – not surprisingly, given their names – return information about filesystems. Now, let’s take the next step and cd into the NFS mount:


Nothing. OK, let’s try ls:


Interesting — we see a repeat of the fsstat operation, along with a a couple of new operations: getattr and access. getattr returns file attributes (such as ownership and timestamps), and access returns file access permission information. Note that, although we’re only running ls once, we’ve seen a total of 5 operations travel along the wire to our NFS server. Let’s try and create an empty file with touch:


More operations involving getattr and access, but interestingly no read or write traffic. Is that because the file is empty? Let’s write some data to it:


Even more operations! Now we’re up to a handful. We have the previously-seen getattr and access, along with some new ones: lookup, create and setattr. (I didn’t highlight every operation as it would have made the graphic a little unwieldy.) As their names suggest, lookup looks up a file’s handle, create creates a file and setattr is used to set a file’s attributes.

One curious thing to note is that, right after setting an attribute with a setattr, we go ahead and read it right back with getattr. Also curious is that we still haven’t seen the write operation.

Let’s try writing to a new file:


Finally, we see a write operation — highlighted in red for your viewing pleasure. But we still see a plethora of other operations that we’ve seen before, all to create and write to a single file: getattr, setattr, access, lookup and create.

Now that we have two files in our NFS mount, let’s run an ls and see what we see:


We see all the operations we’ve seen before, and a new operation called readdirplus. As mentioned in my previous post, readdirplus returns the names of a file in a directory and their attributes. Interestingly, we see the output of ls (files bar and foo) returned after four operations, but then we see another six operations (including readdirplus) occur after we’ve already seen the output.

Let’s append to a file that already has data in it:


Not a lot to see here, and no write command — just some access. Is Linux caching my request? Let’s try reading the file back to make sure the data is there:


Huh! Before the content of the file is returned to us, we see three operations: getattr, access and write. Then we see the file’s content, and then we see another getattr and another access.

For the sake of being thorough, let’s run an ls -al to see if the NFS operations are any different to a regular ls:


Yikes. Even though we have only two files in the directory, there are more than 20 operations before we even see the output of the directory (which I cropped from my screenshot). Nothing new, though; just a lot of getattr and access.

Let’s copy a non-empty file and see what happens:


We see largely the same output as our append process. There’s a whole lot of getattr and access, interspersed with a lookup — but no write! In order to see the write command, I’d have to cat the file to read its contents.

Let’s copy a file that should be too big to cache. We’ll use my favorite Linux calculator, bc, and copy it into our NFS mount:


By now, we expect to see something like the above. A bunch of metadata operations (access, getattr, lookup) followed by a bunch of writes. For the record, the bc binary is only ~70KB. Note that the write operations completed while the copy process was taking place – i.e., there is no filesystem caching going on here. Also note that the number of write operations we see is going to depend on the NFS block size we’re using, too.

For something a little different, let’s try to create a new symbolic link (horse) to point to a file that already exists (donkey):


Hmm. Nothing we haven’t seen before — which is weird, because I know a symlink operation exists in the NFS RFCs. Let’s try and look it up with ls -al:


Et voila! Buried in the output of ls -al, we see our new symlink operation, which is of course used to create symbolic links. Let’s try and follow the symlink:


Interesting. We see some typical getattr and access operations, but we do not see the readlink operation. What else can we do? Let’s try changing the permissions on bc:


Not a lot unusual there. We read the permissions attributes in (via access and getattr), then we set the permissions attributes via setattr. How about changing ownership?


Not much difference — it operates pretty much the same way that chmod operates, i.e. the setting of an attribute. Now, let’s try deleting a file:


Only one new operation here, and not much of a surprise that it’s called remove and neither a surprise that its function is to remove a file 🙂 What if I remove a symbolic link? If you remember, horse is a symlink to donkey:


Here we can see the remove operation doesn’t occur until we bother doing an ls. Last but not least, what about renaming a file?


Again, we see the operation (in this case, rename) after the ls. And that’s it for now! Up next: directories. As always, questions and comments and suggestions are welcome.

5 thoughts on “Deep diving into NetApp NFS operations — file operations

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s