First up–who the heck is SNIA?  Usual blogger shortcut here–a quote from their website:

“SNIA [advances] IT technologies, standards, and education programs for all IT professionals. Made up of some 400 member companies spanning the global storage market, the SNIA connects the IT industry with end-to-end storage and information management solutions.

As a not-for-profit association, the SNIA enables our members to develop robust solutions for storing and managing the massive volumes of information generated by today’s businesses. … As a result, the SNIA has adopted the role of industry catalyst for the development of storage solution specifications and technologies, global standards, and storage education.”

What I’m really looking forward to the most (in no particular order) is:

•    Meeting with others in the industry in an informal, free-exchange atmosphere
•    Working to bring some coherence to the messaging and marketing around solid state hard drives
•    Peeking under the covers at the other SNIA technical working groups (Cloud and Green storage are a couple of the big topics for example)
•    Helping to define and develop concise standards around SSD performance measurement

Why are we interested in events like this?  Simple: establishing standards results in industry progress– it enables customers and manufacturers to better engage and innovate around a technological concept, prototype, or even a finished product.  Because the fact is, measuring performance can be done through many different lenses–for example, a certain perspective can make a given device look better (to drive sales) or make another solution look worse (FUD to hamper adoption by competing technology).  So having the industry get together and agree on standards helps shore up the rules of engagement; like how one benchmarks and tests a device, what state the device is in when the tests are run, and so on.  They help ease some of the obfuscation one often encounters when comparing devices from different manufacturers, or in different usage models.

Of course the SNIA isn’t the only standards group working on this for SSDs.  We are also deeply involved with these efforts at JEDEC as well.  We see these kinds of focused dialogues as win-win.

Anyway, I’ll be in Chicago talking, listening, taking notes—don’t wait up. I’ll post a follow up once I’m back.

If you used you car as a vanpool, and if you were loaded to the gills already, probably not.  But what if I also told you that this rule applied to their larger vehicles too—you could get a massive 18-passenger maxi-van with double the typical gas mileage if you  were willing to order one with 13 seats instead.  Suppose I also told you that this trick worked on every car they made.

What about now?  Would you do it? For those of you that think I’ve lost it there is a computer analogy coming (you knew there would be, right?). Suppose I told you that you could  as much as double the performance of your solid state drive (SSD) if you gave up 25% of the capacity.  Would you do it?  Suppose I also told you that the drive will last longer as a bonus.  How about now?

You can do all this. How? Overprovisioning.

First, a little background on overprovisioning—the basic principle here is to put more NAND on the drive that the drive actually reports.  Why on earth would one do that?  Performance.

Suppose we have a drive that actually is built with 64GB of NAND.  Suppose we “fix” the drive such that it knows to tell the operating system (Windows, Linux, doesn’t matter) that it is “really” a 50GB drive and that the drive owns the extra 14GB of NAND.  So physically we have 64GB on the drive, but logically the drive only shows it has 50GB.  Where did the other 14GB go?  Who stole my NAND?

Well, nobody–the NAND is still there.  That extra 14GB is reserved for the drive; it is there only for the drive’s private use.  This is the principle of overprovisioning. What does the drive use the extra space for?  Several background bookkeeping functions that are hidden from the user, but that are essential to a solid state drive’s performance over the life of the drive.

In a nutshell, NAND-based SSDs like to group data together–it’s a feature of NAND’s personality, the chips like to have data grouped.  They also like to leave little pieces of data scattered about the drive.  A contradiction?  Yes, NAND is like that.  The contradiction comes from the physical way NAND is erased and programmed.  The short version: when NAND is erased, it has to be erased in large segments.  But applications and operating systems like to write data in small blocks.  So even though the NAND prefers the data in large chunks, applications and operating systems aren’t always so obliging.  Therefore we get scattered bits of data on the drive.

But, NAND drives are smart. They know that they like their data well organized and rely on the NAND controller to group these little scattered bits and place them all nice and neat together.  We use a somewhat unflattering term for this process: garbage collection.

Another important feature of NAND-based SSDs is “wear leveling” – which ensures that all the NAND cells on the drive work together to share the storage load (wouldn’t it be nice if people worked that way?).  In order to ensure all the cells share the work, the NAND controller often has to move data around–moving data from cells that are heavily used to less-used cells to make sure all cells are worn evenly.  Hence the fancy name.

In order for garbage collection and wear leveling to work, they both need free space on the drive.  This free space is a temporary spot to put data during the garbage collection and wear-leveling processes.  The data sits there for a bit until it finds a more permanent home.

On an empty drive, free space is abundant, but what about a full drive?  What happens to free space–the working space the garbage collection and wear leveling processes use–as the drive fills?  It goes away.  Without free space, these processes struggle to keep up with demand. What’s a drive to do?  It needs to collect the garbage (by the way, I don’t like that term, but it is well adopted) and do its wear-leveling thing.
Enter overprovisioning.  By reserving a bit of NAND (typically 25% in an enterprise drive and 7% in a client drive) for these background processes–effectively hiding that “extra” space for the OS–the NAND controller is ensured that it has enough empty space to do its thing and not get in the way of the normal data storage processes.

Cool, isn’t it?  But what’s the real gain?  I mean, you’re essentially giving up 25% of the drive to this reserved space–so what do you really get?

First let me throw out a caveat: YMMV (Your Mileage May Vary). Because the actual improvement is heavily dependent on workload, the OS, how full the drive really is, and a number of other factors.  However–in the lab I’ve seen performance improvements up to 4X,  and–the drive lasts longer too. For us, overprovisioning was a no-brainer. With the rapid and regular increases in NAND density, giving up a small slice of capacity for better performance, higher reliability, and longer life was an easy call. I’ll discuss all the details more in a later post.