The Backup Blog

Avamar and VMware Backup Revisited

I thought it might be useful to circle around on this one more time with some definitive thoughts. My apologies for the delay on this--a week of intensive BRS training (EMC's new Backup and Recovery Systems business unit formed out of the union of BURA, the previous EMC group to focus on backup, recovery, and archive, and Data Domain) took me away from my blogging.

So where we left things was this: if you want to do image level backups with vSphere 4.0 the present state of the art does not allow an application consistent backup (without the injection of an agent into the guest OS). The underlying reason for this has to do with the present state of VSS support in VMware Tools. Although I am optimistic that VMware will improve their support in the future, and address this issue to the benefit of backup applications like Avamar, at present they only support VSS snaps on Windows 2003 guests (and do not support the extended application VSS writers at all under Windows 2008).

In practical terms, that means we cannot get application consistent backups of Windows applications. However, we can get crash consistent backups. This is actually a pretty important distinction, and I wanted to raise it because I am not sure W. Curtis Preston did enough to describe this distinction in his post on the subject. And crash consistency is a heck of a lot better than corrupted backups (which is what I fear that many people might infer happens if you don't get application consistency).

VMware Backup with Avamar and vSphere

Edit: I am taking the unusual step of adding this after the original post was finished. As of the moment, it may be best to regard the conclusions in the post as tentative only. The post has generated a flurry of questions and controversy, with several people saying that I am wrong. Unfortunately, nobody has been able to provide a piece of definitive technical documentation (a white paper, or a support document, or relevant piece of text from an administration guide) that clearly describes the issue. Let me be clear: I am OK with being wrong. I kind of hope that I am--it would be just fantastic if we could get image level application consistent backups. But it is not OK that the answer to such an important issue is so obscure. So at the end of the day, if I am wrong but manage to get a clear and well documented answer, that is good enough for me! S.W.

Continuing the posts here and here, I wanted to have a very quick discussion of application consistent backup. This particular issue came up as a question to the other posts--and simultaneously in a discussion with W. Curtis Preston. And the issue is this: can you get .vmdk level application consistent backups of Microsoft guests under VMware ESX 4.0 with vSphere?

So, long question! Why are we asking this? Remember I concluded that you could not get an image level, application consistent backup, that is guaranteed to be consistent--even with vSphere 4.0.

Now Mr. Preston and my smart reader rightly pointed out that VMware says you can. Thanks in advance to W. Curtis: you can find those claims here and here  on the VMware site.

So what is going on here? Was I dead wrong? Does VMware actually permit application consistent image level backups via VMware Tools and support for the MS VSS provider?

VMware Backup with Avamar and vSphere Part 2

On the previous post I had a chance to review the major backup and recovery options for VMware. I put the discussion in terms of Avamar, but a lot of that is just for convenience. If you want, you can substitute your favorite backup application in place of Avamar, and you will get more or less the same result (minus the deduplication), albeit perhaps with greater operational pain and expense.

In any event, now that I have covered off the mechanics of the three major options, it is time to take a look at the why and the how of it all. Why would I want to choose one option over the other? How do we differentiate between them? Simply put: what is the best way to back up VMware?

I am going to start by breaking one of the cardinal rules of rhetoric. I am going to state my conclusion before my premises. And my conclusion is this:

Everybody that runs transactional applications in a VM should do guest level backup. You may also do image level backup as an additional method of data protection, but guest level backup is necessary. If you are protecting anything other than a transactional application with a Windows guest OS, then an image level backup (generated by vSphere 4.0 and Avamar 5.0) is sufficient. If you are protecting a Linux guest OS machine that is not running a transactional application, then you will likely find that guest level backups are sufficient.

Now why would I insist that guest level backup is the preferred methodology if you are running a transactional application within your guest OS?

VMware Backup with Avamar 5.0 and vSphere 4.0

One of the most important, and most frequent questions I get asked is this: what is the best way to back up VMware?

Well, "best" tends to be a bit of a slippery term. Usually the answer to the question should be "it depends." But I thought I would take the time to explore what it really means when we say "it depends." What does it depend on? What are the pros and cons of the different approaches? What are the most important considerations? And what new choices do we have with vSphere 4.0?

Now I am going to explore the answer in the context of Avamar. Mostly I am going to do so because I work for EMC and it is convenient to be able to put a label on these products and processes. Having said that, almost everything I say about the costs and benefits, the value of the various approaches with respect to each other, and the drawbacks of the various approaches is true irrespective of the backup product you are using. Guest backup has certain costs and certain benefits--no matter if you use Avamar or something else. Where benefits are specific to Avamar I will identify them. And certain things that that I identify as unique may only be unique as of the time of this writing; clearly some of the functionality is so desirable that other backup application vendors will be trying very hard to get this capability into their shipping product.

All that said, I think there are two primary methods of protecting a VM: guest level backup and image level backup. There are a couple of ways in which the number of options multiply however--do you use source deduplication or target deduplication? Do you want to use the array to generate a copy for image level backup, or do you want to rely on ESX/vSphere?

Putting The Pieces Together: Deduplication Technologies

Disclaimer: everything that follows is my opinion. In no way should it be mistaken for any sort of official EMC position. Any resemblance to that is purely coincidental.

Amidst the chaos and confusion of the EMC offer to buy Data Domain, I think the biggest unanswered question has been: why would EMC want to have three or four or five different deduplication technologies?

Truthfully, I think the question has a profoundly simple answer: because backup sucks.

Mmmm, irony.

Don't believe me? Show of hands: how many people actually like their existing backup application?

Given the name of this blog, and that I have spent the last 15 years plus on backup and recovery, I think I can appreciate the irony as well as any.

Even more ironic when we remember that Data Domain's original slogan was: tape sucks.

And by the way, there is a back handed answer in there as to why it makes sense for Data Domain to be acquired by somebody. If you claim that tape sucks, and you try to "fix" this with another piece of hardware, all you are really doing is building a better tape drive. You are trying to be StorageTek, only better. Where better means faster and cheaper. But does a faster and cheaper tape drive make backup suck less? Maybe. But probably not nearly as much as it could--if you had a broader perspective and reach with your technology roadmap, one that includes CDP, primary storage, a backup application, and some virtualization capability. More on what I would do with all that later. But one final question: if your objective is to fix backup, completely, and you think that you need access to all those components to do that, who is going to be in a better position to do this? EMC? Or NetApp?

Having said that, the biggest obstacle to fixing backup is not technology. It is inertia. It is cultural. It is fear of change. It is ingrained process. It is the fact that we have done things one way for so long that the reason we are going things has been forgotten.

Not So Fast

I saw this little note on Byte and Switch the other day: Arkeia Shows Virtual Backup Appliance. According to the press release: Arkeia "delivers the first true virtual appliance for backup that gives customers free choice of hardware."

I can't think of any easy way of letting them down, so I will just come out and say it: Arkeia is most definitely not the first to have a virtual backup appliance. For some time now Avamar has been shipping Avamar Virtual Edition (AVE).

Now the point of this post is not to say that Arkeia are all a bunch of goofs or their software is no good or anything like that. Rather, I had two things in mind:

1. Let's set the record straight. Press releases should at least be accurate, and I know there is all kinds of soft language they use so they can say pretty much anything and be open enough to interpretation that company representatives can keep a straight face when repeating them.  But this one is just way off. Every characteristic they list for their "virtual appliance for backup" is already available in AVE. Hrmph. 'Nuff said.
2. AVE is good enough that it is worth repeating what it is and what makes it so good. AVE is a virtual machine that can run on any ESX server and utilizes any storage on any ESX volume that is attached to that ESX server. It can accept a backup from any Avamar client. It is a full fledged Avamar server, and can replicate to any other Avamar server over IP. It also happens to be insanely great as a way to deploy a backup server to a regional office or retail location--by utilizing disk instead of tape, and VMware technology, it is dead easy to deploy, install, and maintain, and provides for local recovery capability while still letting you replicate to a remote (central) server for DR and compliance purposes. Finally, it is dead easy to deploy, and is therefore also the perfect way to get your feet wet with Avamar and deploy it in a lab or POC to validate that I am not full of smoke and that Avamar really is as good as I keep saying it is!

So, in this case, I would have to conclude that the good folks at Arkeia are just a little too late to the party to be making any claims of "first"!

Data Protection Advisor

Note: 2nd of 3 updates on new product announcements for this week. The 3rd installment will cover what is new with NetWorker 7.5.

I thought I would take the opportunity in the second of my product update discussion to discuss one of the lesser know, but most relevant and useful products in the lineup: Data Protection Advisor (formerly known as EMC Backup Advisor, and prior to that as WysDM). Data Protection Advisor, or DPA, fills a big gap for most backup and recovery administrators: the reporting, diagnostics, and analysis of their environment in a useful way.

Notice that I say useful: having 400 canned "reports" does nobody any good if the reports are not useful and relevant.

Free Avamar for VMware

I thought I would return to the subject of Avamar for a quick discussion of one of my favorite pieces of backup software. The geek in me loves new technology and new gadgets, and the contrarian in me loves the thought of doing things different--so long as different is also better.

Avamar meets both of those criteria.

I have written about Avamar for VMware before, and how it is such a good thing. And six months after writing that, it is still a good thing. It doesn't matter if you opt to deploy it in a guest VM, or along side VCB proxy backup, it makes backup of a VMware environment vastly better than it is with traditional backup.

So call it an unintended consequence, but there is something else that happens when you elect to back up an ESX server with Avamar. That is, something else in addition to a reduction in the stress levels of backup administration roughly equivalent to a 6 month stay on Bora Bora, a reduction in storage requirements equal to a mountain of tape, and a reduction in bandwidth requirements of equally gargantuan proportions.

Backup and Recovery Webinars

Thought a few people might be interested in these. Yes I am advertising. But I am advertising for knowledge. This is all good stuff if you need to worry about VMware or bare metal recovery. If you don't? Nothing to see here, move along...

The sessions are:

1 ) A Smarter Approach to Server Recovery

Thursday, September 18, 2008 – 3:00 pm ET. How to implement a complete solution to quickly recover from server hardware failures.

2) Best Practices for Backup and De-duplication in a VMware Environment

Thursday, Sept 24^th, 2008 – 3:00 pm ET. With VMware Consolidated Backup (VCB) and EMC Avamar de-duplication, EMC provides leading-edge backup methodologies for virtual environments. Backup that reduces bandwidth contention, improves availability of CPU and memory resources, performs faster backups, and saves you time and money. EMC Avamar quickly and efficiently protects virtual machines with global data de-duplication technology.

3) The Ideal Solution for Protecting Physical and Virtual Environments.

Thursday, October 9, 2008 – 3:00 pm ET. How to optimize your backup system by Integrating deduplication into your VMware environment.

Toronto Seminar

This August 27th I will be speaking in Toronto, Canada on the subject of deduplication. The session will focus on two key technologies: source deduplication--embodied in Avamar for our purposes, and target deduplication--with the EMC 3D deduplication appliances. I will spend some time as well on how each of these technologies is relevant to VMware backup. Finally, I will talk briefly about Total Cost of Ownership of these new deduplication solutions versus traditional backup solutions that use tape.

Yes, it is partially a marketing session, but it will be technical--maybe not to the degree that I sometimes get into in this blog, but certainly not the typical marketing fluff either. Of course, you are welcome to grab me after the session, and we can take the conversation up a notch or down a notch, which ever is your preference. Or you can just show up and heckle!

Follow the "continue reading" link for a full description, or just click here to register.