The Backup Blog

VMware Backup with Avamar and vSphere Part 2

On the previous post I had a chance to review the major backup and recovery options for VMware. I put the discussion in terms of Avamar, but a lot of that is just for convenience. If you want, you can substitute your favorite backup application in place of Avamar, and you will get more or less the same result (minus the deduplication), albeit perhaps with greater operational pain and expense.

In any event, now that I have covered off the mechanics of the three major options, it is time to take a look at the why and the how of it all. Why would I want to choose one option over the other? How do we differentiate between them? Simply put: what is the best way to back up VMware?

I am going to start by breaking one of the cardinal rules of rhetoric. I am going to state my conclusion before my premises. And my conclusion is this:

Everybody that runs transactional applications in a VM should do guest level backup. You may also do image level backup as an additional method of data protection, but guest level backup is necessary. If you are protecting anything other than a transactional application with a Windows guest OS, then an image level backup (generated by vSphere 4.0 and Avamar 5.0) is sufficient. If you are protecting a Linux guest OS machine that is not running a transactional application, then you will likely find that guest level backups are sufficient.

Now why would I insist that guest level backup is the preferred methodology if you are running a transactional application within your guest OS?

VMware Backup with Avamar 5.0 and vSphere 4.0

One of the most important, and most frequent questions I get asked is this: what is the best way to back up VMware?

Well, "best" tends to be a bit of a slippery term. Usually the answer to the question should be "it depends." But I thought I would take the time to explore what it really means when we say "it depends." What does it depend on? What are the pros and cons of the different approaches? What are the most important considerations? And what new choices do we have with vSphere 4.0?

Now I am going to explore the answer in the context of Avamar. Mostly I am going to do so because I work for EMC and it is convenient to be able to put a label on these products and processes. Having said that, almost everything I say about the costs and benefits, the value of the various approaches with respect to each other, and the drawbacks of the various approaches is true irrespective of the backup product you are using. Guest backup has certain costs and certain benefits--no matter if you use Avamar or something else. Where benefits are specific to Avamar I will identify them. And certain things that that I identify as unique may only be unique as of the time of this writing; clearly some of the functionality is so desirable that other backup application vendors will be trying very hard to get this capability into their shipping product.

All that said, I think there are two primary methods of protecting a VM: guest level backup and image level backup. There are a couple of ways in which the number of options multiply however--do you use source deduplication or target deduplication? Do you want to use the array to generate a copy for image level backup, or do you want to rely on ESX/vSphere?

Avamar v5.0

Today EMC formally announced the availability of Avamar 5.0, which is loaded with new features and functionality. I am going to briefly review some of them here, focusing on the changes that I think will be most significant from a customer's point of view.

Having said that, please don't hesitate to ask questions or post feedback in the comments section if any of you are interested in any clarification on the material.

So what is new?

The big news is: bigger nodes, best server side task scheduling, more efficient checkpoints, new clients, desktop and laptop support, major performance enhancements for a variety of clients, and enhanced support for VMware environments.

That is the high level. Let's dive into these in a little more detail.

Source Deduplication Issues and Concerns

The always interesting and informative Preston de Guise has a discussion of source and target deduplication on his blog "The NetWorker Blog". Which is just about as imaginatively named as this blog. Well, he may not be a marketing mad man, but Preston is one of the most informative and most informed people I know writing about backup and recovery.

However, he does raise a couple of concerns about source deduplication that I would like to address. Now source deduplication for EMC means Avamar. Actually, for most everybody it means Avamar, because there really is no other credible source deduplication alternative today. There may be software deduplication solutions--but most of them tend to be target solutions in one way or the other. And not terribly technically or architecturally credible. (Yes, I am thinking of you, TSM.) With vanishingly small market share to boot.

So Avamar it is.

The Remote Office: CDP or Backup?

A colleague sent me the following link the other day: The Case Against Tape Backup. The link gets you to a white paper written by Double Take Software, which argues that using tape for remote office backup is a bad idea.

Their conclusion? That instead of tape, we should employ CDP software, and that this is much more cost effective.

Unfortunately the paper is deeply flawed. Don't get me wrong, I would be very happy if the general conclusions they drew were true. I would be just as happy if they were content to conclude only that tape was bad--meaning too expensive and too unreliable. 

However, they are not content to reach such a conclusion, and in their attempt to make the leap from "tape is bad" to "CDP is good" they go dramatically awry in two respects. First, they posit a false dichotomy. It is not like the only alternative to tape is CDP for remote backup. And, in fact, some of the solutions that they (conveniently) ignore, are substantially less expensive than their proposed CDP solution. Second, their financial reasoning is pretty badly flawed. And I have never been a fan of make your own solution look good by falsely making the alternative solution look bad.

However, I do think that following through the line of reasoning in the white paper is pretty interesting--and by doing so we can expose some better data about the costs of remote office backup and the alternatives to tape.

Putting The Pieces Together: Deduplication Technologies

Disclaimer: everything that follows is my opinion. In no way should it be mistaken for any sort of official EMC position. Any resemblance to that is purely coincidental.

Amidst the chaos and confusion of the EMC offer to buy Data Domain, I think the biggest unanswered question has been: why would EMC want to have three or four or five different deduplication technologies?

Truthfully, I think the question has a profoundly simple answer: because backup sucks.

Mmmm, irony.

Don't believe me? Show of hands: how many people actually like their existing backup application?

Given the name of this blog, and that I have spent the last 15 years plus on backup and recovery, I think I can appreciate the irony as well as any.

Even more ironic when we remember that Data Domain's original slogan was: tape sucks.

And by the way, there is a back handed answer in there as to why it makes sense for Data Domain to be acquired by somebody. If you claim that tape sucks, and you try to "fix" this with another piece of hardware, all you are really doing is building a better tape drive. You are trying to be StorageTek, only better. Where better means faster and cheaper. But does a faster and cheaper tape drive make backup suck less? Maybe. But probably not nearly as much as it could--if you had a broader perspective and reach with your technology roadmap, one that includes CDP, primary storage, a backup application, and some virtualization capability. More on what I would do with all that later. But one final question: if your objective is to fix backup, completely, and you think that you need access to all those components to do that, who is going to be in a better position to do this? EMC? Or NetApp?

Having said that, the biggest obstacle to fixing backup is not technology. It is inertia. It is cultural. It is fear of change. It is ingrained process. It is the fact that we have done things one way for so long that the reason we are going things has been forgotten.

EMC DL and Oracle RMAN Filters

The second new feature of the DL that I would like to highlight this week is even more straight forward than the last: Oracle Filters. These Oracle Filters give the DL a nearly 2:1 capacity advantage over competitive systems when backing up Oracle RMAN data streams.

To understand what the Oracle Filter is, we need to step back a second and understand how deduplication works. When a data stream is sent to a deduplication appliance, to achieve optimal deduplication efficiency, that appliance should apply a filter that is specific to the type of backup application and data type that is being backed up.

I Am Feeling Underwhelmed

A recent post by somebody over at Data Domain's Dedupe Matters (Especially When That Is All You Do) Blog waxed enthusiastic about their ability to leverage deduplication for disaster recovery.

Reading through the post, I couldn't help but feel a little underwhelmed. The way the issue is discussed it reminds me more of Inspector Clouseau than Sherlock Holmes.

The analysis seemed to me to indicate that Data Domain is very much a hardware company struggling to understand the nuances of software, and the critical role operations and process play in a backup and recovery operation.

Digging a little deeper, the post claimed that using Data Domain appliances for disaster recovery is:

1. Cost effective and efficient;
2. Easier and offers faster recovery SLAs compared to primary storage replication

Now I will be the first to admit that I think that data deduplication for backup has two major attractions: reducing the amount of disk required to store backups, and reducing the amount of network traffic required to replicate backups.

Having said that, I think that Data Domain's position pretty badly misunderstands the process and operational components involved in using backup for disaster recovery. Further, I think their hardware only approach fails to deliver on the real promise of remote deduplication (for remote office sites).

Avamar is Fast

Every time I speak in front of customers about Avamar, I emphasize how much faster it is, and how much less data it sends over the network, than competitive products.

In the end, I think most people end up believing that Avamar can reduce their network traffic by 99% or better.

But I think a lot of people are skeptical about my claim that backups are much faster. So here is another data point that I can offer as "proof". All end user names have been removed to protect the innocent, and a huge shout out to Keith Lawrence for putting the data together.

The comparison is to TSM. The datasets, clients, and so on, are identical in each case. Basically we ran the TSM client and the Avamar client back to back, and had a look at how much data each sent, and how long it took to complete the backup. There are two amazing things here: one, TSM only does progressive incrementals (no full backups) and it still gets smoked by Avamar in terms of how much data is transmitted. Two, again, these TSM backups are not full backups, but they still take much longer than Avamar.

Here are the results:

	TSM Data Sent	Avamar Data Sent	TSM Elapsed Time	Avamar Elapsed Time
Day 1	148.79 MB	15.36 MB	17:45.0	9:36.0
Day 2	149.41 MB	14.54 MB	37:58.0	9:35.0
Day 3	156.56 MB	14.70 MB	39:26.0	9:37.0

So, despite being matched against an incremental backup, Avamar still manages a 90% improvement in amount of data transmitted (and, correspondingly, network utilization). And it still manages to take approximately 25% of the time to finish a backup.

Avamar. Nitrous for your backup.

Web Demos of EMC Backup, Recovery, and Archive Products

For a while now the good folks at Avamar have been offering once a week demonstrations of the product via webex. The schedule has now expanded to include some of the other products in our backup, recovery, and archiving solution set.

The schedule is now as follows:

Times	Monday	Tuesday	Wednesday	Thursday	Friday
8:00 –9:30 am (pacific)	HomeBase (bare metal recovery)	EmailXtender (email archiving)	NetWorker (backup and recovery)	RepliStor (remote file replication)	Avamvar (source based dedup)
10:00-11:30 (pacific)				Data Protection Advisor (backup reporting and analysis)

Complete URLs, dial-in numbers, and product details can be found below. You can also find longer descriptions and product links below.