NetApp NearStore VTL: Now Guaranteed to Cause Data Loss!?

Edit: Note there is revised math in the comments section.

Well, the bloggers at NetApp have been threatening to introduce deduplication for their VTL for a while now, so it should come as no surprise that they finally did it. And just like that, they have assumed the unenviable position of being the last major vendor to announce a dedup product for backup.

(Just a quick role call here: EMC. Check. HP. Yep. IBM. You bet. Sun. Check. Dell. Roger that. Data Domain. Naturally. Sepaton. Yes. Falconstor. Yes. So NetApp, welcome, at last, to the game.)

Naturally, I had a lot of questions when I heard the announcement. Would it do replication? Is it a variable or fixed block scheme? In-line or post-process? How does it protect the hash index? And, most importantly of all, did NetApp decide to stick with VTL RAID rather than moving to RAID DP?

The last question is really the most crucial.

NetApp apparently believes that RAID 5 (of which VTL RAID is a subset) is 4,000 times as likely to lead to catastrophic data loss as RAID DP. We can put that more precisely: any single parity RAID scheme with 7 data drives is 3,955 times more likely, over 5 years, to lead to data loss. Where data loss means that your RAID group is unrecoverable. For those who are interested in how NetApp came to this conclusion, the paper can be found here.

So why does this matter with deduplication? It matters because of the way a deduplication appliance necessarily lays out data across the RAID groups.

Deduplication ultimately reduces all backup data to segments. Those segments can be hashed, indexed, parity protected, self-describing, and so on, but ultimately all backups become reduced to a large number of segments. Segements are often between 8-128 kB in size. So you can see that a 10 TB backup will have a lot of segments associated with it. About 1.5 million of them, if they were 64 kB, or 3 million if they were 32 kB.

So a LUN will contain many, many segments. And each segment will almost certainly have many pointers to it--one for every duplicate instance of that data. Think of it this way: an appliance that is getting 20:1 deduplication will have an average of 20 pointers to each segment of data.

Those pointers will come from lots of different backups, and be associated with 20 different virtual cartridges. At the end of the day, all this means that segments from every backup will likely be found on every LUN. Every virtual tape cartridge associated with that system will likely be spread over every LUN and RAID group.

So what happens if a RAID group fails?

If a RAID group fails, you will lose all of the segments associated with that RAID group. But those segments are almost certainly referred to by every backup, and every virtual cartridge, ever stored on that system.

So if you lose a RAID group, you will lose all virtual cartridges. You will lose all your backups.

You don't just lose some of your data, you lose it all. A single RAID group failure is equivalent to a complete systems failure.

NetApp claims that VTL RAID is 4,000 times more likely to lose data than RAID 6, and is therefore dangerous to your data.

Now it seems that once you add deduplication to their VTL it is much more than 4,000 times more dangerous. It is actually 4,000 time more dangerous, multiplied by the number of RAID groups. In a system with 100 TB of useable capacity, this would be equivalent to 66,667 times more dangerous.

Lets put some specifics on this. NetApp thinks that a single parity RAID scheme with 7 drives has a 6% probability of data loss over 5 years. The probability of a system failure when the failure of any individual component causes system failure can be expressed as:

P(A) + P(B) + ... P(x) - P(A)*P(B)*...*P(x)

In the case of a 48 TB useable system, we will have 8 RAID groups of 6 TB useable each (that is 8 6+1 RAID 5 groups of 7 disks total each).

The probability of failure is:

6% + 6% + 6% + 6% + 6% + 6% + 6% + 6% - 6%^8 = 48%

In the case of a 104 TB useable system, we will have 17 RAID groups of 6 TB useable each (that is 17 6+1 RAID 5 groups of 7 disks total each). The probability of failure is 100%.

So I have a suggestion (tongue firmly in cheek). NetApp seems pretty eager to offer a data deduplication guarantee for space savings on their filers. How about they offer a data loss guarantee with their NearStore VTL? Because it does seem pretty much guaranteed.

One last thought: how do the rest of us mitigate the risk of a failure leading to data loss? Well honestly, you have two choices: RAID-6 and replication. NetApp doesn't do the former. And unfrotunately, they don't replicate either. That's right. No replication with the NearStore VTL. Which means that if you want to replicate, you don't get any of the bandwidth savings that deduplication offers, and you would have to replicate using your backup application, which means that you will consume 20 times more bandwidth than you would if you only replicated the deduplicated data.

Comments

[John]

Wow! You are VERY precise in your deductions. Just seem to lack some intellect on the facts. Were you denied a job at NetApp or maybe lived in a cyber closet for the last several years? Either way your rhetoric reminds me of a politician. Obama, is that you??? So for your list of vendors above, how many of them offer RAID-5 as a preferred or standard RAID with their offerings. How many are using RAID-6 as best practice. (None- due to poor performance unlike when implemented as NetApp RAID-DP) (Just a quick role call here: EMC. Nope. HP. Not even close. IBM. Not even. Sun. wishes they could. Dell. Not even close. Data Domain. Ha! Sepaton. Nope. Falconstor. Do they have disk sub system??? )

Thanks for the useless, misrepresented and un-satisfying facts though! What a very skewed view you have from your back-up mountain top. Get some clarity and present an unbiased postions with a fair representation of all the facts before spewing such worthless non-sense, please!!!

[Scott Waterhouse]

John;

The facts are I am using NetApp numbers, and probability theory.

So there are really only two options here, without any wiggle room I can see.

Option 1: NetApp is wrong about the probability of failure of RAID 5. If that is the case, perhaps they can stop criticizing everybody else for using and not using RAID 6.

Option 2: If they are not wrong, then show me where I have made a mistake with the probability theory. If the risk of failure really is 6% then the rest just follows logically.

Actually, let me introduce another option: contrary evidence or reasoning that would disprove my approach.

I would be happy to discuss based on any of those three options. Heck, I will even retract if it turns out that I am mistaken. But given that you didn't even attempt to address me on the basis of any of those points, I am going to have to leave things at that...

[Cleanur]

John "HP. Not even close"

Really you should check out the VLS6000 and VLS9000 line from HP http://h18006.www1.hp.com/storage/disk_storage/disk_to_disk/vls/index.html
Both use Raid 6 and DeDupe. The only one's not sporting Raid 6 are the lower end entry level boxes where price/capacity are key.

Haven't even bothered to look at the other vendors you mentioned, some are gateway products so will rely on the underlying arrays ability, of which most do support Raid 6. But it looks like you need to do some more research before flinging mud indiscriminately.

[Scott Waterhouse]

Alright, I now have an update on the math. Thanks to an astute reader (Aaron) with a better understanding of probability theory than me, this is what it should be:

Reliability = 1 - (probability of failure of an individual RAID group ^ # of RAID groups). Meaning that a single RAID group system will be 94% reliable over 5 years (only 6% likely to experience catastrophic data loss), and a 2 RAID group system will be 88.36% reliable. The full list, up to 20 RAID groups, is as follows, where the first number is the number of RAID groups, and the second is the probability as a percentage of total failure (total data loss on the NearStore VTL):

1 6.00%
2 11.64%
3 16.94%
4 21.93%
5 26.61%
6 31.01%
7 35.15%
8 39.04%
9 42.70%
10 46.14%
11 49.37%
12 52.41%
13 55.26%
14 57.95%
15 60.47%
16 62.84%
17 65.07%
18 67.17%
19 69.14%
20 70.99%

[JL]

John, your lack of disk knowledge is painful. I work for SEPATON and can confirm that we ship RAID 6. Most other dedupe vendors do as well with the obvious exception of NetApp.

[Scott Waterhouse]

Cleanur;

I was giving John the benefit of the doubt (and not trying to further antagonize him) and assuming that he was talking about primary storage, and not VTL/backup systems. For backup systems, the following vendors offer RAID 6:

Data Domain
EMC
HP
IBM
Sepaton

The ones that DO NOT offer RAID 6 are:

Sun (they use ZFS--and I am not about to go into a full analysis of that vs. RAID 5 here!)
Falconstor (because they don't sell hardware)

And the one I don't know for sure:

Dell (not indicated on the specs for the PowerVault DL2000)

[W. Curtis Preston]

I agree with you that NetApp's blogs (and common sense) dictate that they should be using RAID 6. I also agree that it's disappointing that they released their product without RAID6 or replication. (I’m sure they’re working on it furiously.) It really takes the wind out of the sails of their launch.

I don’t agree with the “last major vendor” comment. EMC, HDS, HP, & Sun are simply putting their own storage behind somebody else’s hard work. That’s hardly the same as building your own target dedupe system from scratch – or (in the case of IBM) having the commitment to acquire the company you’re going to use to do target dedupe.

It’s true that NetApp was the last major OEM disk vendor to ship target dedupe for their VTL, but they were the first major OEM disk storage vendor (i.e. EMC, IBM, HDS, HP, NetApp & Sun) to have their own (not OEMd) VTL and the first vendor of any kind to ship dedupe for primary storage (developed it themselves). Before that, they supported dedupe for NetBackup with that same product. (EMC was the first, and so far the only, such vendor to acquire and ship a source dedupe product.) NetApp is the second such vendor to ship their own (not OEMd) target dedupe product (IBM beat them by a few months via their acquisition of Diligent) , and the first such vendor to develop it themselves. So while they may look like the last one to the party, they’re the first to come with a monogamous wife, versus others who are coming with a friend (EMC/Quantum), or caught in a love triangle (IBM/HDS/Diligent or Sun/COPAN/Falconstor).

Have I given my NetApp contacts serious grief for them not shipping dedupe until now? Of course I have. Am I now giving them more grief for shipping it with RAID5 and no replication? Of course I am. But I also think it’s important to give them props for having the commitment to do it themselves. I think that counts for something, and I just blogged about why:
http://www.backupcentral.com/content/view/199/47/

[Scott Waterhouse]

I think there is a lot we can agree on here. However, I would challenge your supposition EMC is "simply putting their own storage behind somebody else's hard work." That is very far from the truth. In two ways.

First, there is real, significant incremental value add in terms of customer integration and code work that goes on. My favorite example of this is the Write Cache Coalescing that happens on a DL4000 (Clariion storage, Falconstor code base). This is unique to EMC, and it can make a very significant performance difference. This is just one example of dozens on our EDL line. There absolutely is unique integration going on--we don't just grab the code, stuff it on an Intel server, and call it a day!

Second, there is an issue of quality. When EMC puts its name on something, there is a standard of quality, support for multi-vendor environments, and supportability. From testing, to QA, to bug fixes, product support, and our interoperability lab, there is a vast amount of work that goes into a product release. There is a very good reason why EMC EDL code versions have no relationship to Falconstor VTL code versions! And this applies irrespective of whether it is an internally developed product, or has an OEM component to it. The standards are the same.

So OEM'ing code is not necessarily bad, and doing it yourself is not necessarily good. Your relationship metaphors made me smile however.

[W. Curtis Preston]

There's no doubt that the 4000 has some unique code to make it work. It's a unique architecture. But I'll judge that system when it ships. When should we expect that?

But other than the testing, the 3000 series is basically Quantum code in an Intel server in front of Clarion storage, right? While I don't want to minimize the amount of effort testing takes, there's a big difference between testing someone else's product and building your own product and testing IT.

You're right. Being homegrown doesn't necessarily make something better. BUT, all other things being equal, I'll take a homegrown solution over an OEMd solution any day.

I also found it amusing that in all the vendors you listed, you didn't list the one that you're using: "(Just a quick role call here: EMC. Check. HP. Yep. IBM. You bet. Sun. Check. Dell. Roger that. Data Domain. Naturally. Sepaton. Yes. Falconstor. Yes. So NetApp, welcome, at last, to the game.)"

My point of contention was simply that they AREN'T the last vendor to the game. They actually have actually been doing it for a while, just not in their VTL. And their delay is simply because they wanted to do it themselves vs what you EMC did, and I think that choice (while it caused a significant delay during one of the hottest technology trends in recent years) is admirable.

As to my relationship metaphors, I had a laugh typing them myself. I'm glad you can have a sense of humor about this. I forgot one, though:

...or coming to the party with TWO dates, BOTH of which are in a love triangle (EMC/Falconstor/Quantum). Sorry, it was just too good not to include. ;)

[Scott Waterhouse]

The 4000 has been shipping for 2 years now!

If you mean the 4000 with data deduplication, that has been shipping for 3 months at this point.

With respect to the 4000, we do have a great deal of unique code aside from any integration component with deduplication.

With respect to the 1500/3000 there is a small, but growing amount of unique IP there. Over time you will see increasingly greater differentiation between our offering and Quantum's, and that excludes features and functions which we, ah, encourage (!) Quantum to add.

Finally: if both of my dates are supermodels, I know I am pretty happy that I have two of them. ;)