Differences between NetApp Flash Cache and Flash Pool

NetApp’s Flash Cache product has been around for several years, the Flash Pool product is much newer. While there is a lot of overlap between the two products, there are distinct differences as well. The goal of both technologies is the same: to accelerate reads by serving data from solid-state memory instead of spinning disk. Although they produce the same end result, they are built differently and they operate differently.

First of all, why do we bother with read (and write caching)? Because there’s a significant speed differential depending on the source form which we serve data. RAM is much faster than SSD, and SSD is much faster than rotating disk. This graphic illustrates the paradigm:


(Note that we can optimize the performance of 100%-rotating disk environments with techniques like read-aheads, buffering and the like.)

Sequential reads can be read off rotating disks extremely quickly, and very rarely needs to be accelerated by solid-state storage. Random reads are much better served from Flash than rotating disks, and very recent random reads can be served from buffer cache RAM (i.e., the controller’s RAM) — which is an order of magnitude faster again. Although Flash Pool and Flash Cache provide a caching mechanism for those random reads (and some other I/O operations), they do so in different ways.

Here is a quick overview of some of the details that come up when comparing the two technologies:

Detail Flash Cache Flash Pool
Physical entity PCI-e card SSD drive
Physical location Controller head Disk shelf
Logical location Controller bus Disk stack/loop
Logical accessibility Controller head Disk stack/loop
Cache mechanism First-in, first-out Temperature map
Cache persistence on failover Requires re-warm Yes
Cache data support Reads Reads, random overwrites
Cache metadata support Yes Yes
Cache sequential data support Yes, with lopri mode No
Infinite Volume support Yes No
Management granularity System-level for all aggregates & volumes Aggregate-level with per-volume policies
32-bit aggregate support Yes No
64-bit aggregate support Yes Yes
RAID protection N/A RAID-4, RAID-DP (recommended)
Minimum quantity 1 PCI-e card 3 SSD drives
Minimum Data ONTAP version 7.3.2 8.1.1
Removable Yes Yes, but aggregate must be destroyed

The most basic difference between the two technologies is that Flash Cache is a PCI-e card (or cards) that sits in the NetApp controller, whereas Flash Pool are SSD drives that sits in NetApp shelves. This gives way to some important points. First off, Flash Cache memory that is accessed via the PCI-e bus is always going to have a much higher potential for throughput than Flash Pool memory that sits on a SAS stack. Secondly, the two different architectures result in different means of logical accessibility — for Flash Cache, any and all aggregates that sit on a controller with a Flash Cache card can be accelerated. With Flash Pool, any and all aggregates that sit in a disk stack (or loop) with a Flash Pool SSD can be accelerated. This gives way to the fact that in the event of planned downtime (such as a takeover/giveback), the Flash Cache card’s cache data can be copied to the HA pair’s partner node (“rewarming”), but in the event of unplanned downtime (such as a panic), the Flash Cache card’s cache data will be lost. The corollary of this is that if a controller with Flash Pool fails, the aggregate will fail over to the other controller and the Flash Pool will be accessible on the other controller.

Another important difference is Flash Pool’s ability to cache random overwrites. First, let’s define what we’re talking about: a random overwrite is a small, random write of a block(s) that was recently written to HDD, and now we’re seeing a request for that block to now be overwritten. The fact that we’re talking only about random overwrites is important, because Flash Pools do not accelerate traditional and sequential writes — Data ONTAP is already optimized for writes. Rather, Flash Pool lets us cache these random overwrites by allowing the Consistency Points (CPs) that contain random overwrites to be written into the SSD cache. Writing to SSD is up to 2x quicker than writing to rotating disk, which means that the Consistency Point occurs faster. Random overwrites are the most expensive (i.e., slowest) operation that we can perform to a rotating disk, so it’s in our interest to accelerate them.

Although Flash Cache cannot cache random overwrites, it can function as a write-through cache via the use of the lopri mode. In some workloads, we may desire that recently-written data be immediately read just after being written (the so-called “read after write” scenario). For these workloads, Flash Cache can improve performance by caching recently-written blocks rather than having to seek the rotating disks for that recently-written data. Note that we are not writing the client data into Flash Cache primarily, but rather we are writing the data to rotating disk through the Flash Cache — thus, a write-through cache. Flash Cache serves as a write-through cache under two scenarios. Flash Cache will serve as a write-through cache if the Flash Cache is less than 70% full if lopri is disabled. If lopri is enabled, it will always serve as a write-through cache. The enabling of lopri mode in Flash Cache will also cache sequential reads, whereas Flash Pool has no mechanism for caching sequential data.

Continuing this exercise are the differences between how Flash Cache and Flash Pools actually cache data. Flash Cache is populated with blocks of data that are being evicted from the controller’s primary memory (i.e. its RAM) but are requested by client I/O. Flash Cache utilizes a first-in, first-out algorithm: when a new block of data arrives in the Flash Cache, an existing block of data is purged. When the next new block arrives, our previous block is now one step closer to its own eviction. (The number of blocks that can exist in cache depends on the size of your Flash Cache card.)

Flash Pool is initially populated the same way: when blocks of data match the cache insertion policy and have been evicted from the controller’s primary memory but are requested by client I/O. Flash Pool utilizes a temperature map algorithm: when a new block of data arrives in the Flash Pool, it is assigned a neutral temperature. Data ONTAP keeps track of the temperature of blocks by forming a heat map of those blocks.

This is what a temperature map looks like, and shows how a read gets evicted:


You can see the read cache process: a block gets inserted and labeled with a neutral temperature. If the block gets accessed by clients, the scanner sees this and increases the temperature of the block — meaning it will stay in cache for longer. If the block doesn’t get accessed, the scanner sees this and decreases the temperature of the block. When it gets cold enough, it gets evicted. This means that we can keep hot data while discarding cold data. Note that the eviction scanner only starts running when the cache is at least 75% utilized.

This is how a random overwrite gets evicted:


The hotter the block, the farther right on the temperature gauge and the longer the block is kept in cache.¬†Again, you can see the process: a block gets inserted and labeled with a neutral temperature. If the block gets randomly overwritten again by clients, the block is re-inserted and labeled with a neutral temperature. Unlike a read, a random overwrite cannot have its temperature increased — it can only be re-inserted.

Note that I mentioned Flash Pool actually has RAID protection. This is to ensure the integrity of the cached data on disk, but it’s important to remember that read cache data is just that – a cache of read data. The permanent copy of the data always lives on rotating disks. If your Flash Pool SSD drives all caught fire at the same time, the read cache would be disabled but you would not lose data.