NetApp's NearStore VTL: Even More Dangerous

My previous post on this subject incited some really good discussion. A lot of is was in the comments, and much of it was far ranging and discursive, so I thought I would attempt to summarize the dialogue so far, and try to come to some conclusions.

My original post stemmed from a comment by a NetApp blogger that claimed RAID-5 was dangerous, because it would inevitably lead to data loss due to double drive failure. The lynch pin of this argument is the risk associated with double drive failures, and their potential for data loss. Therefore, the real claim is that any RAID schema which does not use double parity is dangerous and will inevitably lead to data loss. There was one important caveat Alex had, and I will repeat it here: big drive sizes increase this risk.

So the question becomes, why does NetApp use a single parity RAID scheme in their NearStore VTL if that is what they believe? And more importantly, I guess, why would you ever deploy a device that was "dangerous" and would "inevitably" have data loss?

The comments that came back from NetApp commentators (but not Alex), to my surprise, only reinforced their stance: RAID with single parity is dangerous. In fact, it is a well established and long held belief over at NetApp. Not only was it echoed in comments: Geert singled out RAID without DP as a bad thing, and claimed that unlike all other forms of RAID, RAID DP: "dramatically increases the *data availability* by surviving the "inevitable" dual disk failure". Val Bercovici has also historically held the same stance: dating back to his dialogue with StorageMojo at least.

So full points for consistency guys. It seems like RAID-6 DP is the way to go in NetApp's book, and everything else is unacceptably dangerous. Everything else will inevitably experience double disk failure, data loss is a given, and in general all other forms of RAID should be avoided like the plague.

OK...

If that is true, the original question holds now more than ever: why would NetApp sell, and why would anybody buy, a NetApp product like the NearStore VTL that doesn't use DP? After all, you are going to lose your backup data, it is only a matter of time.

Well, to that question, Alex basically said (and I am paraphrasing): "sure, NearStore is not RAID DP, and sure that's really bad, but it is not as bad as it could be because VTL RAID isn't quite as bad as normal single parity RAID". Why? Alex gave 3 reasons: 1) VTL RAID stops writing to the degraded RAID group. 2) VTL RAID rebuilds proceed much faster. 3) VTL Self tuning.

So lets deal with them in order:

1) VTL RAID stops writing to a RAID group after in fails. That's great. The issue here is, however, how long are you exposed, and is this too long? Long enough that a double disk failure becomes inevitable? And when we put it that way, reason 1 really becomes a contributing factor to reason 2, but isn't of itself a reason that not having RAID-DP on a Near Store is any more tolerable.

2) VTL RAID rebuilds proceed faster. And as evidence of which, Alex quotes an article 2 years old. That used 320 GB drives. (And by the way, more on the article in a little bit.) In fact, even with 320 GB drives, the rebuild time took 4.5 hours. So it seems reasonable to assume that with 1 TB drives, we would be looking at a rebuild time of 13.5 hours. So much for much faster. In fact, with currently shipping 1 TB drives, the Near Store VTL leaves you exposed for a much longer period of time than NetApp admits at first. And if 13.5 hours of exposure is not enough to  be called "a serious and measurable risk" what is? Which bring us to point #3.

3) Self Tuning. Self tuning increases the risk of a RAID group failure. Although the table cited in Alex's follow up post raise as many questions as answers (how big were the drives sizes on the arrays? how long had they been in production? does the dispersal of virtual cartridges change over time as the array ages?) in does show that in practice, cartridges will be distributed across 2-4 RAID groups. Lets assume that we are emulating LTO-1: 100 GB/cartridge. So a VTL-RAID group of 6 TB would have capacity for 60 cartridges, but would likely have data from 120-240 virtual cartridges. In other words, the inevitable double disk failure will impact as many as 240 virtual cartridges.

So lets do some math. Lets assume NetApp is right about the rebuild time for an EDL (they are not, but I will get to that): 20 hours. Lets assume RAID group sizes are the same: 6 TB. Assume virtual cartridge sizing is the same. If you look at this from a risk over time perspective, the Near Store poses a risk to 120-240 cartridges for 13.5 hours. The comparable array from EMC would risk 60 cartridges for 20 hours. From a risk perspective, the Near Store poses 135% to 270% more risk. Of course, the EMC EDL actually uses RAID-6, so the only array with risk of data loss is NetApp's.

So, none of the three factors cites by Alex mitigate the risk. According to NetApp, single parity RAID is dangerous, and data loss inevitable. VTL RAID is no exception--RAID rebuilds are barely any faster than RAID rebuilds for RAID-5, and the amount of data exposed during a failure is 2-4 times greater than in standard RAID-5.

Overall, VTL-RAID is riskier than standard "dangerous" single parity schemes like RAID-4 or RAID-5.

What NetApp's explanations demonstrate is this: not only is VTL-RAID no better than the alternative single parity schemes, it is worse.

Long footnote about that article NetApp referenced: No surprise, NetApp and Veritest did it again, yet another "impartial" report that favors NetApp. Amazing. But dig a bit (and it doesn't take a lot of digging) under the covers, and you will find so many errors as to render the study worthless:

1. Outdated equipment. When the report was published, the EMC array test was N-1, and had been for almost a full year.
2. Outdated software. Similarly, the firmware levels on the array were N-2. With newer (then current) firmware, they would have encountered much faster RAID rebuilds on the EMC array.
3. Incorrect procedures. None of the configuration methods used by Veritest are supported by EMC. Specific commands must be run to add capacity, for example, and Veritest did not use them. This would have a severe negative impact on performance.
4. Block sizes for tests are inconsistent between the two arrays, and not generally reflective of real world backup block sizes.
5. Specific tuning parameters were selected to exclude key EMC performance features--roughly equivalent to turning off NetApp self tuning.
6. Veritest added capacity to the EMC array in a way which ensured it was not likely to add additional performance. Despite this, the EMC array gained performance as capacity was added, and the NetApp array began to decline in performance (after only 4 shelves--25% of total capacity).

I could go on, but I don't really think it is necessary. A quick perusal by anybody knowledgeable about EMC Disk Libraries would reveal so many errors in method and fact as to make any conclusions drawn from them meaningless. Enough said.

Comments

[Storagezilla]

I had to laugh when I saw that paper dragged out of it's cold grave but it's NetApp SOP in such cases.

When they can't compete on product alone they go for stunt paperwork.

[Stephen McDonald]

Hi Scott,

VTL's don't stripe to all array sets at once. Inbound IO streams are directed to one raid group unless there is a performance benefit to switching groups. What you are stating is the worst case and not the average case.

[Scott Waterhouse]

Yep, as I discussed above, I just used Alex's numbers for how many RAID groups are, on average, used by a virtual tape. His own chart (a little suspicious to me, but I went with it) shows it is from 2-4 RAID groups per virtual cartridge. So what I am stating is the actual case, using NetApp's data. If he wants to come back and say that the data is wrong, fine. But I am just using what he put out there. I stand by the analysis on that basis: NetApp poses from 135-270% greater chance of data loss per RAID group than an equivalently sized standard RAID-5 group. (Start reading after point #3 for the next 4 paragraphs.)

[Alex McDonald]

I'm not quite following on the arguments you make on my table.

http://blogs.netapp.com/shadeofblue/2008/08/i-blame-the-wea.html

The average is 1 RAID group per cartridge.

Take the first row;

. Virtual Tape Volumes (304)
. 2 RAID groups (10)
. 3 RAID groups (0)
. 4 RAID groups (0)
. >4 RAID groups (0)

and the difference (294 cartridges) on 1 RAID group.

Because of space constraints, I didn't show the "Maximum RAID Groups Used per Virtual Tape Volume" for a count of 1, but I assumed it would be clear.

Obviously not.

[Scott Waterhouse]

It was not. :)

But that is OK. At the end of the day, all the nice folks over at NetApp continue to label RAID-5, and by extension any single parity RAID, as professional malpractice. All explanations taken into account, the data shows there is no substantive difference between VTL RAID and any other single parity RAID in terms of rebuild times, reliability, etc. In short, all the criticisms that NetApp levels against single parity protection apply to the Near Store VTL.

If the reliability and risk associated with single parity is acceptable is an individual subjective evaluation, and probably only a part (small part?) of the evaluation that would select a VTL. But at the end of the day, NetApp at least has a consistency issue, a worst has an issue of responsibility (why sell something you really believe is dangerous?) and an issue of reliability to the extent that the need for dual parity protection on a VTL with 1 TB drives is real.

[Alex McDonald]

Better talk to Chuck Hollis then. Apparently RAID5 (8+1) is good enough for Exchange, a Tier1 application. We're consistent in our message, yours is all over the place.

http://chucksblog.typepad.com/chucks_blog/2008/08/your-storage-mi.html

[Scott Waterhouse]

The only way you could possibly be consistent is to withdraw the Near Store VTL from the market (because it is professional malpractice to sell single parity large drive systems) or admit that in some cases, single parity is appropriate.

My apologies if the utilization discussion wasn't helpful, however, try this out for irony: the only way to mitigate the risk with 1 TB drives to a level equivalent to what you claim for rebuild times (1/3 of the actual time) of your dangerous VTL RAID implementation would be to ensure that capacity never went above 34%. Maybe it isn't only filers that have a problem...

OK, relax, that was a joke. True, but funny.

Finally, since a necessary part of your argument is that for single parity RAID to be dangerous it must use large drives, and since the implementation discussed around Exchange and RAID-5 *does not* use large drives, this is, in classic NetApp fashion, another red herring.

[Alex McDonald]

"Finally, since a necessary part of your argument is that for single parity RAID to be dangerous it must use large drives, and since the implementation discussed around Exchange and RAID-5 *does not* use large drives, this is, in classic NetApp fashion, another red herring."

Good point, and worth exploring in a lot more detail. I'll address this as soon as I can. I write my blog out of company hours, and time is short right now due to travel and non-work stuff, and I want to get the second part of the RAID-DP piece out of the way first.

I'll be back!