The Backup Blog

Deduplication APIs?

Curtis Preston raised an interesting point over at Backup Central regarding the need for a common set of standards for the interaction of deduplication appliances and backup applications.

Now, on one hand I agree with Curtis' basic premise (even if I do slightly disagree with some of his historical characterizations). In fact, I have twice made pleas on this blog to similar effect, here and here. Having said that, I will readily admit that W. Curtis Preston is taking the request to the next level. Rather than just ask backup application vendors to implement the means to take advantage of the target side replica of a deduplication appliance, he is asking to community to have a common set of APIs, an open standard, around this.

So, a couple of things. First: everything in this post is strictly my opinion. Nothing in here has been read, endorsed, or reviewed by anybody else at EMC. Having said that: Mr. Preston is right. This is needed, and needed badly. I have been saying it for a while, and his request is the logical extension of the arguments I have made before. So rather than arguing with Curtis about history or debating how to do this, let me state clearly which I think such an effort should achieve, with the hope that this can constructively advance the conversation.

How Fast is Fast

One of the toughest issues to address with deduplication is that of performance. Backup performance can be difficult to characterize under the best of circumstances. There are many things which can contribute to improving (or degrading) performance in a backup environment. This holds true for the performance of deduplication appliances as well. There are a host of factors that can impact performance, and quantifying them can be difficult in the best of circumstances, not least because the thing which is driving data to the appliance--the backup application--can be difficult to quantify and measure reliably as well.

Having said that, there are four or five key questions that I think users should ask when it comes to the issue of performance and deduplication:

1. How fast is my ingest speed with in-line deduplication?
2. How fast is my deduplication speed if I am using delayed or scheduled deduplication? And that is really a two part question: how fast will be initial writes be? and how fast will the post-process deduplication be?
3. How fast are restores? Again, this is a two part question: how fast are restores from deduplicated data? and how fast are restores from native, or fully hydrated data, if it exists on the system?
4. How fast is replication?
5. And finally the most interesting question of all: can you characterize the performance of a system under multiple workloads? What if a system is ingesting, deduplicating, and replicating all at once? What happens to performance for each of those tasks in that case?

I have written a great deal in this blog about the performance of the EMC systems in these different circumstances, and I have also tried to highlight cases in which the competition was either not, in my estimation, being very straightforward with their performance characterizations, or worse, where they made no meaningful characterization at all.

So when JL over at Sepaton said, in our last conversation on the subject, that further details on Sepaton's performance would be forthcoming, I was inclined to believe him. Now, JL has revealed the further details. And this amounts to a single metric: 25 TB per day per node.

Only one image is appropriate at this point, and it absolutely is a little old lady screaming "Where's the beef?"

Come on Sepaton, any intelligent customer is going to need more detail than that! Given that Sepaton does post-process deduplication (only), we have to wonder how fast are initial writes? And how fast does deduplication proceed after the fact? How fast does replication occur? What if it is bi-directional replication? When does a virtual cartridge become eligible for replication? How fast are restores from anything more than a day old?

Until you understand numbers like this, and until your vendor can have meaningful performance conversations that include discussing performance in all these different aspects, you could be in for some unpleasant surprises in enterprise deployments of deduplication technologies.

Virtual Tape Revisionist History

It seems that Tony Pearson over at IBM is up to it. Again. Once again (and just about one year later) he has failed to acknowledge the difference between Open Systems virtual tape and mainframe oriented virtual tape. His claim: "Today, IBM is the leader in storage virtualization, both for disk and tape."

Really, Tony?

I don't think so. It has been a while since we looked at the topic, but to refresh: mainframe and open systems virtual tape are two very different beasts. They have different use cases, attach to different platforms, have different performance objectives, and so on. And while it is true that IBM has done admirably well in the area of mainframe virtual tape (although EMC's MDL continues to gain market share), it is also true that IBM has been a virtual no-show in the area of open systems virtual tape. To claim leadership would be a massive distortion of the facts.

Unequivocally, EMC has shipped more Open Systems virtual tape than any other vendor. And when compared to any other single vendor, like IBM, the disparity is usually about 10:1 in EMC's favor.

Of course, Tony is also claiming a "resurgence" in interest in tape. In the same year that IBM reported a 31% year over year decrease in tape sales revenue. Hmmm.

Performance Tips For EDL

One of the conversations I have most frequently with customers regarding virtual tape and the EMC Disk Library (EDL) is this: how can we make it go fast? Well, there are a few simple things that you can do to make sure your EDL is running as fast as it can. After that, you may find some other bottlenecks elsewhere in your environment... but that isn't necessarily a bad problem to have. As I like to say: EDLs are fast enough that it is very unlikely they will be the bottleneck in backup.

So what should you do for optimal EDL performance? Here are a few simple tips:

How Do You Spell Irony? Does It Start With A "V"?

Recently one of the good bloggers over at NetApp decided to start throwing stones (again) at the Centera platform. Stephen Todd and Barry Burke have done a great job of demonstrating that the alleged flaw with Centera, well, wasn't. Wasn't a flaw. And wasn't with Centera.

So our unnamed NetApp persona seems to being getting awfully wound up about something that had less than a 1 in 1 billion chance of happening (every thousand years) originally. After the update to the platform (5 years ago), that chance dropped to nothing.

(For reference, 1 in 1 billion is equivalent to 0.0000001%)

Which makes me wonder, why would somebody get so concerned about a platform that has had no chance of data loss (due to hash/object collisions), when they are still shipping a product that has a 99.64% chance of data loss in everyday use?

Things that make you go hmmmm.

Another Big Name Reports Drop in Tape Sales

Steve Duplessie reports that IBM tape sales dropped 31% year over year. Interesting. Now before I go any further with this, let me say that Steve writes on of the smartest, and funniest blogs in IT. And I would sooner cross a cranky underfed alligator while stranded in the everglades with both arms tied behind my back than I would him. (Which is as close as I can really get to a good Duplessie metaphor--he would undoubtedly invoking smoking, Germans, and midgets to make his point.)

Having said that Steve then writes: "At the highest end of the market, tape hasn't gone away – and actually it's the opposite." Well, yes and no. I would say based on my observation that it is true that it hasn't gone away at all due to its critical role in very long term retention of backups and off-siting. One could argue that these roles are more visible and more important than ever, and thus the perceived relevance of tape remains high.

However, sales are off. IBM's were down, and so were HP's earlier. I see fewer and fewer big new tape buys even in the high end of the enterprise where some form of VTL, either with or without deduplication, is not also being considered. From the perspective of operational recovery, tape is being marginalized. From a budget perspective, it is being severely eroded: let's face it, I need a lot fewer tape drives to make a monthly/quarterly/annual backup image once a month, than I need to backup up everything everyday.

Not its not dying. It is just slowly tottering off into the sunset land of uninteresting and nearly irrelevant technologies. Just like dumb terminals and SCO: there are still a few around, but nobody cares much about them any more.

Welcome to Two New Bloggers

Just a quick welcome to two new bloggers. The first covers a topic especially near and dear to my heart: Networker. The NetWorker Blog also has a nifty url--a bit on an in joke for real NetWorker people. Preston has some really good insights too; if you are a NetWorker admin this is definitely worth checking out. Also, a shout out to fellow EMC'er Edward Newman who is writing on all matters infrastructure.