The Backup Blog

NearStore VTL: Requiem for a Dream

In a recent posting, Curtis Preston wrote some pretty insightful words. I was waiting for the appropriate moment to reference them, and this seems as good as any. Since Val Bercovici referenced them recently, it seems doubly appropriate. Curtis wrote:

I treat vendor blogs as a printed version of a sales rep talking to me. I expect the sales rep to tell me all the best things about their product and none of the bad.  I expect them to try to bash the competition at some point (although I don't like it when they do), but I also believe that the best source about their competition isn't them; it's their competition. So when they tell me that the other vendor's blood is green, I don't believe them.  I grab the nearest sales rep and prick them with a needle and look very closely.

My thoughts on this? This is all true, with one important exception: where a product has a specific and significant shortcoming. To (unfairly?) perpetuate a stereotype of sales reps, we wouldn't expect that rep to immediately admit, or even admit under anything less than extreme duress, a critical failing of a product, would we? So, after thinking a lot about this, I think there is a valuable role for the competitive blog post: to expose a critical short coming. Hopefully this can be accomplished without any unnecessary histrionics, vaudevillian showmanship, or needless hyperbole. I regard this as similar to the role of the press in a democracy, or the role of the opposition in government: to provide checks and balances, and a contrary opinion. To keep everybody honest.

In the blogosphere, we have the added luxury that this can be a dialogue. If a competitor disagrees, there is every opportunity to engage in conversation, and point out the failings of the post. Does the reasoning hold up or does it not? Is it sound and relevant, or is it much ado about nothing? And ultimately I hope the readers bring their critical reading skills to the table so they can judge each position on its own merit.

So, when I posted that the NearStore VTL was practically guaranteed to lose data, I hoped that the analysis would: a) stand up to scrutiny by NetApp, b) that they would engage in dialogue if, in their opinion, I was wrong, and that, c) it was relevent. Well, like Meatloaf said, 2 out of 3 ain't bad.

NetApp NearStore VTL: Now Guaranteed to Cause Data Loss!?

Edit: Note there is revised math in the comments section.

Well, the bloggers at NetApp have been threatening to introduce deduplication for their VTL for a while now, so it should come as no surprise that they finally did it. And just like that, they have assumed the unenviable position of being the last major vendor to announce a dedup product for backup.

(Just a quick role call here: EMC. Check. HP. Yep. IBM. You bet. Sun. Check. Dell. Roger that. Data Domain. Naturally. Sepaton. Yes. Falconstor. Yes. So NetApp, welcome, at last, to the game.)

Naturally, I had a lot of questions when I heard the announcement. Would it do replication? Is it a variable or fixed block scheme? In-line or post-process? How does it protect the hash index? And, most importantly of all, did NetApp decide to stick with VTL RAID rather than moving to RAID DP?

The last question is really the most crucial.

NetApp NearStore VTL: 4000 Times as Dangerous as an EMC EDL

I have previously discussed one of the big problems with the NetApp NearStore VTL here and here: it uses a type of RAID that is more dangerous than RAID-6 (including RAID-DP), and is, in fact, more dangerous than RAID-5.

Now I was content to describe the NearStore VTL as merely "more dangerous" than EMC's EDL at the time. After discussing the issues with Alex McDonald on The Missing Shade of Blue, I think I came to the conclusion that:

1. The NearStore VTL is, basically RAID-5. Yes, it is called VTL RAID by NetApp marketing, and yes there are some differences, but fundamentally, the risk and rebuild times are equivalent to RAID-5 (unless you aren't actually using the disk volume!).
2. In some respects, VTL RAID is worse than RAID-5 in terms of risk of data loss, as it tends to disperse some virtual cartridges across more than one RAID group. Meaning that the failure of a RAID group can impact a greater number of virtual tape cartridges than its capacity might indicate.

All this lead me to characterize VTL RAID as having 135% to 270% more risk of catastrophic data loss due to double disk failure than a RAID-5 volume of the same size. And all I was further able to conclude about the risk of a NearStore VTL vs. an actual EDL was that that the risk of a NearStore is much greater than an actual EDL that uses RAID-6.

It appears now however that we can be very specific about how much "much greater" is when it comes to the risk of a NearStore vs. an EDL.

According to the latest data cited by Alex: "Over a period of 5 years, RAID-5 is 4,000 times more likely to suffer from data loss than RAID-DP." His emphasis. Quote source is here.

So They Do Understand...

It took a while, but the nice folks over at NetApp have attempted to describe what exactly Tape Smart Sizing does. I could be wrong, but I think they agree with me. I guess since I agreed with them that using single parity RAID on a VTL with 1 TB drives is dangerous (yes their Near Store VTL does exactly that...) they decided to return the favor? Perhaps...

In any event, Absolute Architect decided to pen a response which was completely complementary to my analysis. To the best of my understanding, she agrees that Tape Smart Sizing is a "confusing ... technology that won't remotely impact your life" (her words). Which pretty closely echoes my claim that it is "unnecessary and irrelevant". And I mean that literally. For most, if not all organizations, the "feature" is either unnecessary, irrelevant, or both.

Much like a square wheel bike.

Do square wheels work? Yes, if the ground is shaped like an inverted catenary. Are square wheels necessary or relevant? Not even remotely.

The analogy is perfect. Does Tape Smart sizing work? Yes. Is Tape Smart Sizing necessary or relevant? Not even remotely.

The good stuff is always in the why.

NetApp's NearStore VTL: Even More Dangerous

My previous post on this subject incited some really good discussion. A lot of is was in the comments, and much of it was far ranging and discursive, so I thought I would attempt to summarize the dialogue so far, and try to come to some conclusions.

My original post stemmed from a comment by a NetApp blogger that claimed RAID-5 was dangerous, because it would inevitably lead to data loss due to double drive failure. The lynch pin of this argument is the risk associated with double drive failures, and their potential for data loss. Therefore, the real claim is that any RAID schema which does not use double parity is dangerous and will inevitably lead to data loss. There was one important caveat Alex had, and I will repeat it here: big drive sizes increase this risk.

So the question becomes, why does NetApp use a single parity RAID scheme in their NearStore VTL if that is what they believe? And more importantly, I guess, why would you ever deploy a device that was "dangerous" and would "inevitably" have data loss?

Tape Smart Sizing: Another NetApp Myth

One of the more interesting replies to my recent discussion of NetApp's dangerous VTL--which NetApp apparently feels is likely to cause significant loss of backup data--concerned the concept of "tape smart sizing". To quote NetApp's CTO: "I can also confirm that Tape Smart Sizing works exactly as advertised."

Since I dismissed it as little more than a David Blaine type trick, it might be useful to ask: what exactly is advertised? Again, just quoting NetApp:

"[tape smart sizing] sample[s] the data for compressibility as it’s being written to the VTL and then dynamically adjust the size of that virtual tape to match the estimated compressed capacity of the physical tape it will be written to. The LTO-3 virtual tape is now automatically adjusted depending on what type of data is being written to it and how it will compress in the Real-Tape World. Welcome to not wasting money or space on extra physical tapes when using a VTL."

Incidentally, NetApp also claims that:

Every other VTL out there requires you to to set a fixed size on the virtual tape when configuring the virtual library.

The consequence of this is, according to NetApp, that if you write 200 GB data that is 2:1 compressible to any other VTL, you will use 200 GB of disk space; write it to tape, and you use 100 GB of tape (because tape is compressed). So, it seems that you would require 2 virtual LTO-1 cartridges. Take those 2 virtual cartridges, and they will each half fill a physical cartridge. Thus, wasted tape.

So we have two different bloggers claiming that tape smart sizing is useful, and unique in terms of its ability to save space of physical media. Basically, that "every other VTL" will require that you waste physical tape space as described above.

Let me state this unequivocally, and for the record: this is not true. EMC's virtual tape products have no such issue.

NetApp's VTL is "Dangerous"

It is not often that I will admit NetApp is right in this blog.

The explanation for this amounts to one of, or a combination of, two factors: 1) NetApp doesn't sell a lot of virtual tape, and they don't offer deduplication with their virtual tape, so our paths rarely intersect; 2) when they do intersect, I find their marketing claims tend to have more style than substance. Try dissecting "tape smart sizing" for example. David Blaine would be proud.

However, when somebody is right they are right. And in this case, Alex McDonald at NetApp is right!

So don't take this as a sign of the apocalypse!

What does Alex have to say at "The Missing Shade of Blue" in his post "Ray Day Pay"?

Well he is talking about the advantages of RAID-6 over RAID-5. And he correctly points out that RAID-6 offers greater protection from disk failure than RAID-5, by virtue of the fact that there are two parity blocks per RAID rank. In fact, he notes that:

"The advantages of dual parity are in robust data protection; specifically, RAID-6 can sustain two simultaneous drive failures in any RAID group without loss of data."