The Backup Blog

Something Doesn't Add Up (And Never Will)

When considering what to do to upgrade your backup system, I have always maintained that one of the most important questions you should be asking is: what is wrong with what I got? (Proving nothing more than that the English language will be the first victim of this post.)

What is wrong can have many answers. It is not reliable. It breaks. It is slow. It costs too much. And so on. But one of the very frequent answers is this: my backup takes too long.

As I have said many times before, there are a lot of things that can cause a backup to bottleneck. And there are a lot of things that you can make infinitely fast and your backup will still be no faster overall (yes, LTO3 and LTO4, I am talking about you.)

And until we understand the big picture, and which components in that big picture are contributing to a slow backup, it is a little irresponsible for any of us on the vendor side of things to come out and claim that disk will "fix" your backup and make it faster. It might. Or it might not.

How Much Is Too Much?

You can never be too thin or too rich. (Although I don't think whoever came up with that had Nicole Richie or the Sultan of Brunei in mind.)

And in the world of backup, you can never be too fast. It is just not possible.

Rich Colbert and Daniel Budiansky have posts regarding the importance of speed to backup. You can read those here and here.

The premise of their approach seems to me two-fold: one, the DD880 is fast enough that it obsoletes post-process deduplication technologies; and two, that this speed is high enough that for the first time in-line deduplication will no longer be a bottleneck in the backup process for the vast majority of customers.

Now I don't entirely disagree with these arguments. For a long time I have been saying of a DL4206 and DL4406 when asked of performance: "They are so fast that I can virtually guarantee that they will not be the bottleneck in your backup process." At up to 2,200 MB/s, we can see that there are many opportunities for backups to degrade before the EDL becomes the problem: data must be read from disk, through a client, across a network, sometimes mediated by an database or application interface, to a backup server, meta-data must be processed, and then it must traverse another network before it is written to a target. There are so many obstacles here (including the presence of an OS, a filesystem, different disk architectures, etc.) that there is very little chance that the EDL will be a bottleneck. Certainly not for a single backup server or storage node or media server.

Welcome Aboard Data Domain

As of this morning, it is official, EMC has acquired a majority intrest in Data Domain, and from here forward they will be run as an product division within EMC, with the DD structure remaining intact, and Frank Slootman will head the new division.

So first things first: welcome aboard to all Data Domain employees and customers. As I have said here previously, I think EMC is a great place to work, and I believe that you will all find that to be the case too. To Data Domain customers--I think you will find EMC has a lot to offer, that we really care about our customers, and that we have a lot to talk about.

Having said that, I think most people will have two big questions for EMC: 1) How do DD and Avamar fit together? And why do you guys now have two different deduplication technologies? 2) How is DD going to fit within a unified product line?

The answers here I will give are my personal answers. This shouldn't be mistaken for some sort of official EMC statement, or any sort of look into our product roadmap! Having said that...

In answers to the first question(s): my firm belief is that source deduplication (Avamar) and target deduplication (Data Domain) are entirely complementary. Complementary in both a business sense, and a technology sense.

TSM, Virtual Tape, Deduplication and Reclamation

Well that is kind of a mouthful.

Quick technical tip for TSM users. When TSM is finished with a tape volume (virtual or physical) the default behaviour is to not relabel it, but instead to reuse it with the existing label (volume ID).

This happens to be not a very good thing if you are using a virtual tape library with deduplication. Why? Because virtual tape libraries don't reclaim space--at the block or segment level that is no longer referenced by valid backup data--on a volume that is still in use. Even if "in use" in this case means sitting in the (virtual) scratch pool doing nothing.

So, the good folks over at IBM figured this out, and have a way to force TSM relabel a volume when it is returned to the scratch pool. In terms, the VTL now sees that the data referenced by this volume is no longer valid, and can be freed up for reclamation, which will free up physical space on your deduplicated VTL. Good stuff.

In order to take advantage of this, you need to set "relabelscratch=yes" in your library definition. You can find additional detail here: http://www-01.ibm.com/support/docview.wss?uid=swg21330242. Note that if you are at TSM 5.5.1.0 you will need to apply a patch for this to work, if you are at 5.5.2.x and higher, it should work without further patching.

Putting The Pieces Together: Deduplication Technologies

Disclaimer: everything that follows is my opinion. In no way should it be mistaken for any sort of official EMC position. Any resemblance to that is purely coincidental.

Amidst the chaos and confusion of the EMC offer to buy Data Domain, I think the biggest unanswered question has been: why would EMC want to have three or four or five different deduplication technologies?

Truthfully, I think the question has a profoundly simple answer: because backup sucks.

Mmmm, irony.

Don't believe me? Show of hands: how many people actually like their existing backup application?

Given the name of this blog, and that I have spent the last 15 years plus on backup and recovery, I think I can appreciate the irony as well as any.

Even more ironic when we remember that Data Domain's original slogan was: tape sucks.

And by the way, there is a back handed answer in there as to why it makes sense for Data Domain to be acquired by somebody. If you claim that tape sucks, and you try to "fix" this with another piece of hardware, all you are really doing is building a better tape drive. You are trying to be StorageTek, only better. Where better means faster and cheaper. But does a faster and cheaper tape drive make backup suck less? Maybe. But probably not nearly as much as it could--if you had a broader perspective and reach with your technology roadmap, one that includes CDP, primary storage, a backup application, and some virtualization capability. More on what I would do with all that later. But one final question: if your objective is to fix backup, completely, and you think that you need access to all those components to do that, who is going to be in a better position to do this? EMC? Or NetApp?

Having said that, the biggest obstacle to fixing backup is not technology. It is inertia. It is cultural. It is fear of change. It is ingrained process. It is the fact that we have done things one way for so long that the reason we are going things has been forgotten.

EMC World Wrap Up

Post EMC World 2009 I thought I would offer a couple of observations on what I saw, and what customers were asking about.

The show was very well attended, and we were busy in the backup and dedup booth.

Two key trends emerged.

The first is that Avamar is becoming more and more mainstream. I saw interest in the technology from companies both small and large. Moreover--and this is just a subjective impression--I would saw interest was as strong in Avamar as it was in target based deduplication solutions. People are absolutely considering making the transition from their current backup application to Avamar where it makes sense (ROBO, VMware environments, and file servers primarily).

DL4000 3D Explained, Part 3

In an effort to respond to the observations of W. Curtis Preston in his posts here, here and his comments here; and additionally in an effort to satisfy those of you who wanted a more direct response to those observations, I post some (final?!) comments below. Two comments first:

1. The dialogue has been going on for a long time, and the material comes from a few different sources. So the responses are more point form, and may move from point to point without a lot of regard for the previous point--there is no longer any strong unifying theme.
2. All the material below should, in my opinion, be placed firmly in the context of my VTL: What Matters? post. I feel quite strongly that those are the over-riding concerns, and that the following material is interesting primarily in terms of its' ability to determine how well the device meets the criteria described in "What Matters?"

With that said, on with the show!

Does Global Deduplication Matter?

Curtis Preston penned something last week asking if global deduplication matters. I am not sure if Curtis would count me in the group that questions global deduplication's reality, or not...

Of course I believe it is real!

The real issues are actually two-fold: does it matter, and if so, how much?

Mr. Preston claims that it only matters if you back up more than 10-20 TB. Well, that is a pretty broad range. I would actually amend this to a more specific number: for a single EMC DL3000 that is 18 TB (12 hours of deduplication), and for a DL4000 with deduplication enabled it is 30 TB. Why the larger number? Because with deferred deduplication and faster ingest speeds, it is perfectly legitimate to deduplicate throughout the day. For the sake of operational realities, I am assuming you are deduplicating for 20 hours with this number. That still leaves ample time for other niceties, like restore, for example.

Why Do We Build A DL4000 3D?

I have had a lot of comments, mostly constructive, on some of posts about performance and the EDL 4000 3D. (The translation of that product name, by the way, is the EMC Disk Library 4000 with deduplication.) W. Curtis Preston has been one of the most vocal critics, and while I think he is dead wrong--nothing personal, Curtis, that is just my opinion of your opinion--I don't think his comments are so trivial that I am going to dismiss them out of hand. I also suspect that they are roughly congruent with the initial reaction some users might have to the device.

So I wanted to take the opportunity to address some of these comments in a more structured fashion than just the back-and-forth in the comments section. Giving Mr. Preston his due, I will quote him directly and then respond to the objections.

Deduplication APIs?

Curtis Preston raised an interesting point over at Backup Central regarding the need for a common set of standards for the interaction of deduplication appliances and backup applications.

Now, on one hand I agree with Curtis' basic premise (even if I do slightly disagree with some of his historical characterizations). In fact, I have twice made pleas on this blog to similar effect, here and here. Having said that, I will readily admit that W. Curtis Preston is taking the request to the next level. Rather than just ask backup application vendors to implement the means to take advantage of the target side replica of a deduplication appliance, he is asking to community to have a common set of APIs, an open standard, around this.

So, a couple of things. First: everything in this post is strictly my opinion. Nothing in here has been read, endorsed, or reviewed by anybody else at EMC. Having said that: Mr. Preston is right. This is needed, and needed badly. I have been saying it for a while, and his request is the logical extension of the arguments I have made before. So rather than arguing with Curtis about history or debating how to do this, let me state clearly which I think such an effort should achieve, with the hope that this can constructively advance the conversation.