The Backup Blog

How Much Is Too Much?

You can never be too thin or too rich. (Although I don't think whoever came up with that had Nicole Richie or the Sultan of Brunei in mind.)

And in the world of backup, you can never be too fast. It is just not possible.

Rich Colbert and Daniel Budiansky have posts regarding the importance of speed to backup. You can read those here and here.

The premise of their approach seems to me two-fold: one, the DD880 is fast enough that it obsoletes post-process deduplication technologies; and two, that this speed is high enough that for the first time in-line deduplication will no longer be a bottleneck in the backup process for the vast majority of customers.

Now I don't entirely disagree with these arguments. For a long time I have been saying of a DL4206 and DL4406 when asked of performance: "They are so fast that I can virtually guarantee that they will not be the bottleneck in your backup process." At up to 2,200 MB/s, we can see that there are many opportunities for backups to degrade before the EDL becomes the problem: data must be read from disk, through a client, across a network, sometimes mediated by an database or application interface, to a backup server, meta-data must be processed, and then it must traverse another network before it is written to a target. There are so many obstacles here (including the presence of an OS, a filesystem, different disk architectures, etc.) that there is very little chance that the EDL will be a bottleneck. Certainly not for a single backup server or storage node or media server.

Welcome Aboard Data Domain

As of this morning, it is official, EMC has acquired a majority intrest in Data Domain, and from here forward they will be run as an product division within EMC, with the DD structure remaining intact, and Frank Slootman will head the new division.

So first things first: welcome aboard to all Data Domain employees and customers. As I have said here previously, I think EMC is a great place to work, and I believe that you will all find that to be the case too. To Data Domain customers--I think you will find EMC has a lot to offer, that we really care about our customers, and that we have a lot to talk about.

Having said that, I think most people will have two big questions for EMC: 1) How do DD and Avamar fit together? And why do you guys now have two different deduplication technologies? 2) How is DD going to fit within a unified product line?

The answers here I will give are my personal answers. This shouldn't be mistaken for some sort of official EMC statement, or any sort of look into our product roadmap! Having said that...

In answers to the first question(s): my firm belief is that source deduplication (Avamar) and target deduplication (Data Domain) are entirely complementary. Complementary in both a business sense, and a technology sense.

Putting The Pieces Together: Deduplication Technologies

Disclaimer: everything that follows is my opinion. In no way should it be mistaken for any sort of official EMC position. Any resemblance to that is purely coincidental.

Amidst the chaos and confusion of the EMC offer to buy Data Domain, I think the biggest unanswered question has been: why would EMC want to have three or four or five different deduplication technologies?

Truthfully, I think the question has a profoundly simple answer: because backup sucks.

Mmmm, irony.

Don't believe me? Show of hands: how many people actually like their existing backup application?

Given the name of this blog, and that I have spent the last 15 years plus on backup and recovery, I think I can appreciate the irony as well as any.

Even more ironic when we remember that Data Domain's original slogan was: tape sucks.

And by the way, there is a back handed answer in there as to why it makes sense for Data Domain to be acquired by somebody. If you claim that tape sucks, and you try to "fix" this with another piece of hardware, all you are really doing is building a better tape drive. You are trying to be StorageTek, only better. Where better means faster and cheaper. But does a faster and cheaper tape drive make backup suck less? Maybe. But probably not nearly as much as it could--if you had a broader perspective and reach with your technology roadmap, one that includes CDP, primary storage, a backup application, and some virtualization capability. More on what I would do with all that later. But one final question: if your objective is to fix backup, completely, and you think that you need access to all those components to do that, who is going to be in a better position to do this? EMC? Or NetApp?

Having said that, the biggest obstacle to fixing backup is not technology. It is inertia. It is cultural. It is fear of change. It is ingrained process. It is the fact that we have done things one way for so long that the reason we are going things has been forgotten.

EMC DL and Oracle RMAN Filters

The second new feature of the DL that I would like to highlight this week is even more straight forward than the last: Oracle Filters. These Oracle Filters give the DL a nearly 2:1 capacity advantage over competitive systems when backing up Oracle RMAN data streams.

To understand what the Oracle Filter is, we need to step back a second and understand how deduplication works. When a data stream is sent to a deduplication appliance, to achieve optimal deduplication efficiency, that appliance should apply a filter that is specific to the type of backup application and data type that is being backed up.

A Comprehensive Misunderstanding

Data Domain is at it again: they have made another plea to please, please (please!) think of them as an enterprise solution. And boy are they ready to do disaster recovery. Honest.

Except that what they call disaster recovery, well, isn't.

Not by a long shot.

Now I am not sure if I care if they don't get it. Bummer for them, but ultimately, so what?

Well the so what in this case is I think they are unfairly dumbing down the collective discussion on backup and disaster recovery. In other words, they can make whatever claims they want, but when they attempt to redefine common understandings of backup and recovery and disaster recovery, in such a way that these terms mean less than they did before, they are doing themselves and the market a huge disservice.

I Am Feeling Underwhelmed

A recent post by somebody over at Data Domain's Dedupe Matters (Especially When That Is All You Do) Blog waxed enthusiastic about their ability to leverage deduplication for disaster recovery.

Reading through the post, I couldn't help but feel a little underwhelmed. The way the issue is discussed it reminds me more of Inspector Clouseau than Sherlock Holmes.

The analysis seemed to me to indicate that Data Domain is very much a hardware company struggling to understand the nuances of software, and the critical role operations and process play in a backup and recovery operation.

Digging a little deeper, the post claimed that using Data Domain appliances for disaster recovery is:

1. Cost effective and efficient;
2. Easier and offers faster recovery SLAs compared to primary storage replication

Now I will be the first to admit that I think that data deduplication for backup has two major attractions: reducing the amount of disk required to store backups, and reducing the amount of network traffic required to replicate backups.

Having said that, I think that Data Domain's position pretty badly misunderstands the process and operational components involved in using backup for disaster recovery. Further, I think their hardware only approach fails to deliver on the real promise of remote deduplication (for remote office sites).

Deduplication Efficiency

There are a lot of different deduplication choices these days. And lots of ways to shove these technologies into places they weren't meant to go (or shouldn't have been). Even if we focus just on deduplication for backup--and this is the backup blog!--then there are still a couple big choices to make.

The biggest choice, of course, is do you want to deduplicate at the source or the target?

In EMC product terms, do you want to use Avamar or DL3D? Just for fun, lets note that there are no real significant alternatives in the source deduplication space these days (as it is my perception that Symantec's PureDisk is retreating to a target based model). On the target side, Data Domain retains a strong position but continues to lose significant market share to EMC.

Avamar on a Roll

Avamar has done exceptionally well over the last year. The product has definitely come along, and happens to be hitting its stride at the same time that customers are realizing how problematic certain types of backup are: VMware in particular. All this has contributed to massive growth, and a sudden realization that source based deduplication is a powerful alternative to target based deduplication. It turns out that we at EMC aren't the only ones who feel this way. From a recent analyst report on Data Domain, we have the following:

"analysts see a problem looming on Data Domain's horizon: competition. Since data protection software is a highly saturated market, Slootman says the players are in a "ground war" over space in the data center. And EMC's (EMC) purchase of deduplication startup Avamar Technologies two years ago brought one of the data center's behemoths into the picture... Data Domain is still the dominant player in the field, and the company has kept innovating to ensure a superior offering."

So lets take a look at this. The analyst is acknowledging that Avamar is competing effectively against DD. And we are. Avamar works, and works very well. What is more, Avamar offers a couple of compelling values that DD can't: significantly reduced I/O and CPU utilization on the backup host. DD does nothing for either--you still use the same amount of CPU, the same amount of network bandwidth, and you are still tied to the mediocre capabilities of a traditional backup application in the VMware world.

Data Protection Advisor

Note: 2nd of 3 updates on new product announcements for this week. The 3rd installment will cover what is new with NetWorker 7.5.

I thought I would take the opportunity in the second of my product update discussion to discuss one of the lesser know, but most relevant and useful products in the lineup: Data Protection Advisor (formerly known as EMC Backup Advisor, and prior to that as WysDM). Data Protection Advisor, or DPA, fills a big gap for most backup and recovery administrators: the reporting, diagnostics, and analysis of their environment in a useful way.

Notice that I say useful: having 400 canned "reports" does nobody any good if the reports are not useful and relevant.

DL3D: The Benefits of Immediate Deduplication

One of the most powerful features of the EMC DL3D is its ability to do both in-line and post-process deduplication. It is one of the few (the only?) deduplication appliance that offers users this flexibility, and in fact lets you do both on one appliance, depending on the requirements of the data being backed up.

Certain competitors have done their best to portray this flexibility as a weakness. Huh? Since when is choice a bad thing?

Well, it isn't a bad thing at all. Choice is good. Flexibility is good.

The other criticism that gets leveled against the DL3D here is that the in-line is not really in-line. In particular, Data Domain attempts to claim that EMC doesn't do in-line deduplication at all--that we write to a disk cache, and that this lets us be deceptive with our performance numbers.

Nothing could be more misleading.