Managing cDOT images and packages

When you’re upgrading cDOT clusters, we keep copies of both the installation files (i.e. the tarballs) and the installation packages.

Note that the downloaded images are kept on each node, hence we’re using a system image command to manipulate them:

dot83cm::> system image package show
             Package
Node         Repository     Package File Name
------------ -------------- -----------------
dot83cm-01
             mroot
                            823_q_image.tgz
                            831RC1_q_image.tgz
                            83P1_q_image.tgz
                            83P2_q_image.tgz
                            83RC2_q_image.tgz
dot83cm-02
             mroot
                            822P2_q_image.tar.gz
                            823_q_image.tgz
                            831RC1_q_image.tgz
                            83P1_q_image.tgz
                            83P2_q_image.tgz
                            83RC2_q_image.tgz
11 entries were displayed.

Those can be deleted simply:

dot83cm::> system image package delete -node * -package 83P1_q_image.tgz
1 entries were deleted.

You can delete them off one or multiple nodes. To verify that it’s gone:

dot83cm::> system image package show
             Package
Node         Repository     Package File Name
------------ -------------- -----------------
dot83cm-01
             mroot
                            823_q_image.tgz
                            831RC1_q_image.tgz
                            83P2_q_image.tgz
                            83RC2_q_image.tgz
dot83cm-02
             mroot
                            822P2_q_image.tar.gz
                            823_q_image.tgz
                            831RC1_q_image.tgz
                            83P2_q_image.tgz
                            83RC2_q_image.tgz
9 entries were displayed.

Note that we also keep the unpacked tarball, only this time it’s cluster-wide hence we’re using a cluster image command to see what’s out there:

dot83cm::> cluster image package show
Package Version  Package Build Time
---------------- ------------------
8.3.1            8/31/2015 08:42:08
8.3.2            2/23/2016 22:29:11
8.3.2P1          4/14/2016 11:58:25
8.3.2P3          6/22/2016 04:16:50
8.3.2RC1         11/5/2015 08:41:17
8.3.2RC2         12/8/2015 05:24:13
8.3P1            4/7/2015 12:05:35
8.3P2            5/19/2015 03:34:02
8 entries were displayed.

They can also be deleted:

dot83cm::> cluster image package delete -version 8.3P1      
                                                                               
Package Delete Operation Completed Successfully

Because it’s a cluster-wide operation, there’s no need to specify a node.

dot83cm::> cluster image package show
Package Version  Package Build Time
---------------- ------------------
8.3.2            2/23/2016 22:29:11
8.3.2P1          4/14/2016 11:58:25
8.3.2P3          6/22/2016 04:16:50
8.3.2RC1         11/5/2015 08:41:17
8.3.2RC2         12/8/2015 05:24:13
8.3P1            4/7/2015 12:05:35
8.3P2            5/19/2015 03:34:02
7 entries were displayed.
Advertisements

Why efficiency matters: NetApp versus Isilon for scale-out

Disclaimer: I am a NetApp employee. The views I express are my own.

As the marketplace for scale-out offerings heats up, it’s interesting to see the different approaches that different companies take with their product development. The two leaders in scale-out NAS & SAN are NetApp and Isilon, and they both take rather different approaches to scale-out technology when it comes to performance. In this post, I will attempt to quantify the result of those differences: how NetApp does more, with less, than a comparable Isilon offering.

In terms of reference material, I’ll be drawing from the SPECsfs2008_nfs.v3 submissions from both NetApp and Isilon. NetApp’s 4-node FAS6240 submission is here, and Isilon’s 28-node S200 submission is here. First things first: I picked the two submissions that most closely matched one another in terms of performance. As you can see from the sfs2008 overview, there are a lot of submissions to choose from. I chose NetApp’s smallest published cluster-mode offering, and then looked for an Isilon submission that was roughly equivalent.

As part of this exercise, I put together list prices based on data taken from here (NetApp) and here (Isilon). I chose list prices because there is no standard discount rate from one vendor, or one customer, to another. If you have an updated list price sheet for either vendor, please let me know. Here are the results:

NetApp

  • 260,388 IOps
  • $1,086,808 list
  • 288 disks for 96TB usable

Isilon

  • 230,782 IOps
  • $1,611,932 list
  • 672 disks for 172TB usable

Doing some basic math, that’s $4.17 per IOp for NetApp and $6.98 per IOp for Isilon.

And to get a roughly equivalent level of performance, Isilon needs more than twice as many disks to do so! That brings us full-circle to the point about efficiency. NetApp does more with less disk, which means we get significantly more performance per disk than Isilon does:

“But wait”, I hear you cry, “NetApp has FlashCache! That’s cheating because you’re driving up the IOps of the disks via cache!” It’s true – the submission did include FlashCache. 2TB in total; 512GB in each of the four FAS6240 controllers. Isilon’s submission had solid-state media of their own; 5.6TB in total from a 200GB SSD in each of the 28 nodes.

“But wait”, I hear you cry, “RAID-DP has all of that overhead! WAFL, snap reserve space; you must be short-stroking to get those numbers!” Wrong again – we’re using more space than the competition.

In order to meet roughly the same performance goal, Isilon needed to provision almost twice as much space as an equivalent NetApp offering. That’s hardly storage efficient. It’s not cost-efficient, either, because you have to buy all those spindles to reach a performance goal even if you don’t have that much data you need to run at that speed.

“But wait”, I hear you cry, “those NetApp boxes are huge! They must be chock-full of RAM. And CPUs. And NVRAM too!” True again; each NetApp controller has 48GB of RAM for a total of 192GB. By contrast, Isilon has 1344GB of RAM. Isilon does have slightly less NVRAM (14GB) compared to NetApp (16GB).

“But wait”, I hear you cry, “NetApp requires 10Gb Ethernet for that performance!”. Yes, and so does Isilon. Let’s see how efficient we get not only from the nodes themselves, but also from the load-generating clients, too:

Although 10Gb Ethernet switch ports are coming down in price, they’re still not particularly cheap. And look at the client throughputs: Isilon struggled to get more than 10,000 IOps from each client, which means you have to scale out your client architecture as well. Which, of course, means more money.

“But wait”, I hear you cry, “NetApp is still going to use more power and space with their big boxes!” Not true. Here are the environmental specs for both:

Isilon did use less rack space (56RU) than NetApp (64RU). The environmental data were taken from here (Isilon) and here (NetApp).

Every single graph pictured above was compiled with data taken only from the three sources listed: the SPECSFS submissions themselves (found via Google), the list-price lists (found via Google) and the environmental submissions (found via Google). I will gladly provide the .xls file from which I generated the graphs if anyone’s interested.

Thoughts and comments are welcome!

Why cache is better than tiering for performance

When it comes to adjusting (either increasing or decreasing) storage performance, two approaches are common: caching and tiering. Caching refers to a process whereby commonly-accessed data gets copied into the storage controller’s high-speed, solid-state cache. Therefore, a client request for cached data never needs to hit the storage’s disk arrays at all; it is simply served right out of the cache. As you can imagine this is very, very fast.

Tiering, in contrast, refers to the movement of a data set from one set of disk media to another; be it from slower to faster disks (for high-performance, high-importance data) or from faster to slower disks (for low-performance, low-importance data). For example, you may have your high-performance data living on a SSD, FC or SAS disk array, and your low-performance data may only require the IOps that can be provided by relatively low-performance SATA disks.

Both solutions have pros and cons. Cache is typically less configurable by the user, as the cache’s operation will be managed by the storage controller. It is considerably faster, as the cache will live on the bus — it won’t need to traverse the disk subsystem (via SAS, FCAL etc.) to get there, nor will it have to compete with other I/O along the disk subsystem(s). But, it’s also more expensive: high-grade, high-performance solid-state cache memory is more costly than SSD disks. Last but not least, the cache needs to “warm up” in order to be effective — though in the real world this does not take long at all!

Tiering’s main advantages are that it is more easily tunable by the customer. However, all is not simple: in complex environments, tuning the tiering may literally be too complex to bother with. Also, manual tiering relies on you being able to predict the needs of your storage, and adjust tiering automatically: how do you know tomorrow which business application will require the hardest hit? Again, in complex environments, this relatively simply question may be decidedly difficult to answer. On the positive side, tiering offers more flexibility in terms of where you put your data. Cache is cache, regardless of environment; data is either on the cache or it’s not. On the other hand, tiering lets you take advantage of more types of storage: SSD or FC or SAS or SATA, depending on your business needs.

But if you’re tiering for performance (which is the focus of this blog post), then you have to deal with one big issue: the very act of tiering increases the load on your storage system! Tiering actually creates latency as it occurs: in order to move the data from one storage tier to another, we are literally creating IOps on the storage back-end in order to accomplish the performance increase! That is, in order to get higher performance, we’re actually hurting performance in the meantime (i.e., while the data is moving around.)

In stark contrast, caching reduces latency and increases throughput as it happens. This is because the data doesn’t really move: the first time data is requested, a cache request is made (and misses — it’s not in the cache yet) and the data is served from disk. On it’s way to the customer, though, the data will stay in the cache for a while. If it’s requested again, another cache request is made (and hits — the data is already in the cache) and the data is served from cache. And it’s served fast!

(It’s worthwhile to note that NetApp’s cache solutions actually offer more than “simple” caching: we can even cache things like file/block metadata. And customers can tune their cache to behave how they want it.)

Below is a graph from a customer’s benchmark. It was a large SQL Server read, but what is particularly interesting is the behavior of the of the graph: throughput (in red) goes up while latency (in blue) actually drops!

If you were seeking a performance augmentation via tiering, there would have been two different possibilities. If your data was already tiered, throughput will go up while latency will remain the same. If your data wasn’t already tiered, throughput will decrease as latencies will increase as the data is tiered; only after the tiering is completed will you actually see an increase in throughput.

For gaining performance in your storage system, caching is simply better than tiering.

New job, new town, same look

As some of my more astute readers may know, I recently left Bowdoin College for a position down at NetApp in Waltham, MA. I’m a Systems Engineer in the Enterprise group; essentially I’ll be helping with both pre-sales consulting as well as post-sales review (with focuses on performance and VMware).

While I was sad to leave Bowdoin, I am super excited to be at my new position! It required a bit of a move down from Maine, as well as an environmental adjustment from academia to corporate, but so far I am having a great time.

I’ll be blogging here when I can, and (like previous posts) mostly focusing on issues surrounding VMware and performance. I have a couple of small updates in the pipeline, hopefully I’ll get to them this week!

The reality of economics in tech innovation

There are things in the economies of tech businesses that scale well (mass production; agglomeration of labor) and there are things that don’t scale well. Innovation is not something that scales well, and so I will try and point out a few reasons why.

A few days ago HP announced that it is killing its tablet and spinning off WebOS. Perhaps the only surprising thing is that it happened so quickly; the HP tablet was only a few months old.

On the surface, you would imagine — as many people did — that a company as large, talented and wealthy as HP could actually pull off a device to compete with the Apple iPad. However, I think this was a poor supposition to have been made at all, for reasons I will go into in this post.

HP’s competitive advantage in the tech industry is that, up until maybe the last decade or so, it had the best engineers in the business. While that is not arguably true anymore, it is important to note that it never really had a significant competitive advantage in the mobile/tablet market. The mobile/tablet market is dominated by Apple, of course, whose competitive advantage is simply selling great design and great UI. Apple have the most market share because they are obviously the best at it. When was the last time you said “This HP device is really great, and really easy to use?” I will go ahead and give you the answer, i.e. “Never”. They had tried mobile before, with the iPaq smartphones, and they sucked.

While HP’s lack of competitive advantage in the mobile/tablet sector doesn’t mean the HP tablet was destined to fail, it sure had a huge mountain to climb if it was to be unseat the iPhone & iPad. Unfortunately for HP, they failed.

It is also worth noting that competitive advantages, like everything else in the world, are dynamic. Research In Motion (RIM) had a competitive advantage in building mobile devices that a) worked well with enterprise software and b) were very network-efficient. Now, neither are really true; as RIM employees are starting to see the writing on the wall. RIM failed to notice that customers started wanting different things: as the mobile market expanded to “regular” consumers (as opposed to corporate consumers), people want things like cameras and MP3 players and don’t care about “true” multitasking or many things RIM think they care about. But RIM is wrong, and it will kill the company if they don’t reverse their course.

Innovation can come out of nowhere, and it can disappear just as quickly. Perhaps the first great innovator in the mobile space was Nokia; now their products are openly mocked. They had the best technology at the time the technology was emerging; once the technology became common and cheap they were quickly displaced by other manufacturers who had better devices (Samsung, Motorola, etc.) Businesses cannot rest on their laurels and presume that their advantages will live forever.

Deleting private messages queues in Windows 2008 Server

Ran into this little chestnut this morning. Windows 2008, in its infinite wisdom, does not let you delete private message queues; they are owned by the SYSTEM user and no one else. This was a roadblock to uninstalling a vendor’s HVAC platform, so here’s the way around it:

  1. Open Server Manager
  2. Click on the Features tree
  3. Click on the Message Queuing tree
  4. Expand the Private Queuing tree
  5. Right-click on the queue you want to delete and select Properties
  6. Click on the Security tab
  7. Click on the Advanced button
  8. Click on the Owner tab
  9. Click on your own account name, then click Apply
  10. Click on the Permission tab
  11. Click on the Add button and add yourself as a user
  12. Give yourself Full Control and click Apply
  13. Click OK
  14. Right-click on the queue you want to delete and select Delete

Et voilĂ ! No more queue.

Using Splunk with BlackBoard

I have been using Splunk for a couple of years and am pretty happy with it. At the BbWorld DevCon 2011 some people asked me about it, so I thought I’d write up how and why I use it.

Introduction

What is Splunk?

Splunk is a log aggregation & searching tool. Okay, so it does way more than just that, but that’s what we use it for. It lets you monitor pretty much anything: a file, a directory, a port, a socket, WMI, whatever; and collect the output. Then it lets you search the output. It is better at searching log files than anything I have ever used. It knows the difference between an Apache log file and an IIS log file and a Tomcat log file, so when you search them it’s aware of the correct formatting to use.

And then there are Apps.

How much does it cost?

Splunk is licensed by how many MB of data you process per day. There’s a free, lifetime license for <500MB/day users; anything over that and you have to pay. Quite a lot. The free version does have one caveat: it doesn't support authentication. So you'll have to firewall your Splunk server after the 30-day trial expires & the free lifetime license kicks in.

What does it run on?

Pretty much anything, it seems.

Installation

Installing Splunk on Linux

Because we’re monitoring files on our BlackBoard server, you should install Splunk on a central Splunk server, and also on the Bb app server itself. (If you want, you could do it standalone on your Bb server, but why bother?) Simply download the correct version and install it via RPM. Accept the license agreement, of course!

The default username for a new application installation is admin with a password of changeme.

Configuring the Splunk server

By default, it runs on port 3000; you may want to change that via this command:

sudo /opt/splunk/bin/splunk set web-port 80
sudo /opt/splunk/bin/splunk restart

Also, by default, Splunk does not accept data sent by other Splunk servers (or clients). You’ll need to enable that via this command:

/opt/splunk/bin/splunk enable listen 42099 -auth admin:changeme

(FYI, the default listening port is 9997.)

If you want to accept syslog messages (generally a good thing), run this command:

/opt/splunk/bin/splunk add udp 514 -sourcetype syslog -auth admin:changeme

Configuring your Splunk clients

This is a two-step process. You need to enable forwarding on your Splunk client, but you have an option to enable lightweight forwarding as well. The difference is, when you enable lightweight forwarding, it shuts down the Splunk GUI and removes some unnecessary services. This is a great thing to do on a Splunk “client” machine:

sudo /opt/splunk/bin/splunk enable app SplunkLightForwarder -auth admin:changeme
sudo /opt/splunk/bin/splunk add forward-server splunk.bowdoin.edu:42099
sudo /opt/splunk/bin/splunk restart

Monitoring inputs

Splunk is all about inputs. You can get a great of what – and how – Splunk can index data here. In Linux, inputs conf is located in /opt/splunk/etc/system/local/inputs.conf. The syntax is pretty simple, here is an example of what I monitor:

[default]
host = blackboard.bowdoin.edu

[monitor:///usr/local/blackboard/logs/bb-services-log.txt]
[monitor:///usr/local/blackboard/logs/bb-sqlerror-log.txt]
[monitor:///usr/local/blackboard/logs/tomcat/catalina-log.txt]
[monitor:///var/log/messages]

Note that you don’t have to monitor a series of files: you can monitor a whole directory, or files with wildcards, whatever. (On Windows, those paths would look like [monitor://C:\Logs\foo.log].)

Configuring the *NIX app

As mentioned above, Splunk is very extensible via its application collection. To collect the *NIX statistics (users, resource usage etc.) from a Splunk client, add this to /etc/hosts:

139.140.xxx.xxx         splunk  splunk.bowdoin.edu      LOGHOST

On your Splunk client, add this to /etc/syslog.conf:

*.*                                                     @LOGHOST

On your Splunk client, restart the syslog daemon:

sudo /sbin/service syslog restart

What do I do now?

Save a search. Create a dashboard. Make an alert. Go forth!