More on TCP/IP offloading & iSCSI performance

Spurred on by the surprising[ly poor] results I saw when previously testing TCP/IP offloads, I decided to run some more tests. This time, as a couple of people have suggested, I wanted to load up the server’s four cores (it’s a dual-socket, dual-core Opteron) before testing the network I/O. This way, we should see the benefits of the TCP/IP Offload Engine become clear as our host’s CPUs become busy dealing with application I/O.

Using Phoronix’s Test Suite, I ran the stressCPU2 test for a half-hour while running both TOE-enabled and TOE-disabled tests with Oracle’s ORiON. I did this with the same ORiON syntax as before, namely:

./orion -run advanced -testname mytest -num_disks 1 -write 50

Here are the results for 50% read, 50% write:

With TOE:

Maximum Large MBPS=47.94 @ Small=0 and Large=2
Maximum Small IOPS=1253 @ Small=5 and Large=0
Minimum Small Latency=3.74 @ Small=4 and Large=0

Without TOE:

Maximum Large MBPS=42.82 @ Small=0 and Large=2
Maximum Small IOPS=1098 @ Small=5 and Large=0
Minimum Small Latency=4.00 @ Small=1 and Large=0

Now that I’m seeing the opposite of yesterday’s results, I thought I’d re-run the tests. Like I did above, I ran the tests below back-to-back (changing the TOE driver out as soon as possible):

With TOE:

Maximum Large MBPS=38.95 @ Small=0 and Large=2
Maximum Small IOPS=1015 @ Small=5 and Large=0
Minimum Small Latency=4.71 @ Small=1 and Large=0

Without TOE:

Maximum Large MBPS=34.98 @ Small=0 and Large=2
Maximum Small IOPS=738 @ Small=5 and Large=0
Minimum Small Latency=5.62 @ Small=1 and Large=0

This is virtually the reverse of what we saw before. Now, we see that the host — which is CPU-bound due to our artificial load generation — is having trouble filling network I/O when not utilizing the TOE drivers. When the TOE drivers are used, the host I/O returns to normal.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s