The Backup Blog

Copies and Backups Revisited

Mark (aka Storagezilla) Twomey wrote in his blog last week about copies and backups, and concluded that point in time copies are backups.

Hrmm. As I have said before, I am not sure that I agree with this.

I have discussed the issue here with W. Curtis Preston too, and I think there is a disconnect happening somewhere. I am fundamentally uneasy with the idea of calling a copy a backup. It seems to me that a copy is a necessary part of a backup, but not sufficient.

Would a tar ball be a backup? A gzip?

Again, I think the answer is, at best, "sort of".

And why do I differentiate between a copy and the tar ball?

Because I think there is not only a difference between copies and backups, but I think there is a difference between a backup and a backup and recovery system.

So lets start with fundamentals.

Something Doesn't Add Up (And Never Will)

When considering what to do to upgrade your backup system, I have always maintained that one of the most important questions you should be asking is: what is wrong with what I got? (Proving nothing more than that the English language will be the first victim of this post.)

What is wrong can have many answers. It is not reliable. It breaks. It is slow. It costs too much. And so on. But one of the very frequent answers is this: my backup takes too long.

As I have said many times before, there are a lot of things that can cause a backup to bottleneck. And there are a lot of things that you can make infinitely fast and your backup will still be no faster overall (yes, LTO3 and LTO4, I am talking about you.)

And until we understand the big picture, and which components in that big picture are contributing to a slow backup, it is a little irresponsible for any of us on the vendor side of things to come out and claim that disk will "fix" your backup and make it faster. It might. Or it might not.

Putting The Pieces Together: Deduplication Technologies

Disclaimer: everything that follows is my opinion. In no way should it be mistaken for any sort of official EMC position. Any resemblance to that is purely coincidental.

Amidst the chaos and confusion of the EMC offer to buy Data Domain, I think the biggest unanswered question has been: why would EMC want to have three or four or five different deduplication technologies?

Truthfully, I think the question has a profoundly simple answer: because backup sucks.

Mmmm, irony.

Don't believe me? Show of hands: how many people actually like their existing backup application?

Given the name of this blog, and that I have spent the last 15 years plus on backup and recovery, I think I can appreciate the irony as well as any.

Even more ironic when we remember that Data Domain's original slogan was: tape sucks.

And by the way, there is a back handed answer in there as to why it makes sense for Data Domain to be acquired by somebody. If you claim that tape sucks, and you try to "fix" this with another piece of hardware, all you are really doing is building a better tape drive. You are trying to be StorageTek, only better. Where better means faster and cheaper. But does a faster and cheaper tape drive make backup suck less? Maybe. But probably not nearly as much as it could--if you had a broader perspective and reach with your technology roadmap, one that includes CDP, primary storage, a backup application, and some virtualization capability. More on what I would do with all that later. But one final question: if your objective is to fix backup, completely, and you think that you need access to all those components to do that, who is going to be in a better position to do this? EMC? Or NetApp?

Having said that, the biggest obstacle to fixing backup is not technology. It is inertia. It is cultural. It is fear of change. It is ingrained process. It is the fact that we have done things one way for so long that the reason we are going things has been forgotten.

A Comprehensive Misunderstanding

Data Domain is at it again: they have made another plea to please, please (please!) think of them as an enterprise solution. And boy are they ready to do disaster recovery. Honest.

Except that what they call disaster recovery, well, isn't.

Not by a long shot.

Now I am not sure if I care if they don't get it. Bummer for them, but ultimately, so what?

Well the so what in this case is I think they are unfairly dumbing down the collective discussion on backup and disaster recovery. In other words, they can make whatever claims they want, but when they attempt to redefine common understandings of backup and recovery and disaster recovery, in such a way that these terms mean less than they did before, they are doing themselves and the market a huge disservice.

I Am Feeling Underwhelmed

A recent post by somebody over at Data Domain's Dedupe Matters (Especially When That Is All You Do) Blog waxed enthusiastic about their ability to leverage deduplication for disaster recovery.

Reading through the post, I couldn't help but feel a little underwhelmed. The way the issue is discussed it reminds me more of Inspector Clouseau than Sherlock Holmes.

The analysis seemed to me to indicate that Data Domain is very much a hardware company struggling to understand the nuances of software, and the critical role operations and process play in a backup and recovery operation.

Digging a little deeper, the post claimed that using Data Domain appliances for disaster recovery is:

1. Cost effective and efficient;
2. Easier and offers faster recovery SLAs compared to primary storage replication

Now I will be the first to admit that I think that data deduplication for backup has two major attractions: reducing the amount of disk required to store backups, and reducing the amount of network traffic required to replicate backups.

Having said that, I think that Data Domain's position pretty badly misunderstands the process and operational components involved in using backup for disaster recovery. Further, I think their hardware only approach fails to deliver on the real promise of remote deduplication (for remote office sites).

Backup Service Catalog

My previous post Is a Copy a Backup? was written after I got to thinking about a backup services catalog. What is a backup services catalog? Basically it is two things.

First, it is an admission that we have a lot more choices in terms of what technologies to use to provision backup services than ever before. 10 years ago, the backup technologies was really backup technology: tape. It was pretty easy to design a such a catalog, because you really only had one choice: tape. And it didn't really matter how rich, thin, or beautiful you were, you still only had one choice: tape.

Second, it is a recognition than different applications have different levels of importance to the business, and that given we have an array of technology choices with which to backup them up and recover them, we should probably choose different technologies for the different applications. Let me put that a different way: your SAP/Oracle application which runs AP, AR, and inventory is probably way more important than a miscellaneous SQL server some user threw up for ad hoc reporting, or an edge web server that is one of a few dozen others just like it.

So if you have different tiers of applications, shouldn't you have different tiers of backup and recovery? You bet you should.

Is a Copy a Backup?

And why is this an important question?

I started out thinking about today's post as an attempt to describe how to structure a backup services catalog. A backup services catalog would specify what application tiers are present in the business, and what Service Level (SLA) of backup and recovery they each receive. A tier one application (the production SAP database, for example) would get a higher level of backup and restore service than a tier three application (a miscellaneous SQL database set up for ad hoc queries that services a couple users).

Do You Know How Much You Backup?

And another question, just as important: do you know what you backup?

To those who do know, the question may seem trivial--even insulting. Of course I know! But do you? Really?

Two recent cases make me think that some backup admins who think they know how much they are backing up, in fact, don't know.

Sort of a case of however much you think you are backing up I do not think you are backing up what you think you are backing up.

In each of these cases, the backup admin was off by about a factor of 3. One thought he was backing up about 1.5 TB per day. It is actually 4.7 TB per day (and climbing). Another was convinced that he was backing up about 2 TB per day. In that case, he actually backs up about 7 TB per day.

Avamar is Fast

Every time I speak in front of customers about Avamar, I emphasize how much faster it is, and how much less data it sends over the network, than competitive products.

In the end, I think most people end up believing that Avamar can reduce their network traffic by 99% or better.

But I think a lot of people are skeptical about my claim that backups are much faster. So here is another data point that I can offer as "proof". All end user names have been removed to protect the innocent, and a huge shout out to Keith Lawrence for putting the data together.

The comparison is to TSM. The datasets, clients, and so on, are identical in each case. Basically we ran the TSM client and the Avamar client back to back, and had a look at how much data each sent, and how long it took to complete the backup. There are two amazing things here: one, TSM only does progressive incrementals (no full backups) and it still gets smoked by Avamar in terms of how much data is transmitted. Two, again, these TSM backups are not full backups, but they still take much longer than Avamar.

Here are the results:

	TSM Data Sent	Avamar Data Sent	TSM Elapsed Time	Avamar Elapsed Time
Day 1	148.79 MB	15.36 MB	17:45.0	9:36.0
Day 2	149.41 MB	14.54 MB	37:58.0	9:35.0
Day 3	156.56 MB	14.70 MB	39:26.0	9:37.0

So, despite being matched against an incremental backup, Avamar still manages a 90% improvement in amount of data transmitted (and, correspondingly, network utilization). And it still manages to take approximately 25% of the time to finish a backup.

Avamar. Nitrous for your backup.

DL4000 3D Explained, Part 3

In an effort to respond to the observations of W. Curtis Preston in his posts here, here and his comments here; and additionally in an effort to satisfy those of you who wanted a more direct response to those observations, I post some (final?!) comments below. Two comments first:

1. The dialogue has been going on for a long time, and the material comes from a few different sources. So the responses are more point form, and may move from point to point without a lot of regard for the previous point--there is no longer any strong unifying theme.
2. All the material below should, in my opinion, be placed firmly in the context of my VTL: What Matters? post. I feel quite strongly that those are the over-riding concerns, and that the following material is interesting primarily in terms of its' ability to determine how well the device meets the criteria described in "What Matters?"

With that said, on with the show!