VMware Backup with Avamar and vSphere

Edit: I am taking the unusual step of adding this after the original post was finished. As of the moment, it may be best to regard the conclusions in the post as tentative only. The post has generated a flurry of questions and controversy, with several people saying that I am wrong. Unfortunately, nobody has been able to provide a piece of definitive technical documentation (a white paper, or a support document, or relevant piece of text from an administration guide) that clearly describes the issue. Let me be clear: I am OK with being wrong. I kind of hope that I am--it would be just fantastic if we could get image level application consistent backups. But it is not OK that the answer to such an important issue is so obscure. So at the end of the day, if I am wrong but manage to get a clear and well documented answer, that is good enough for me! S.W.

Continuing the posts here and here, I wanted to have a very quick discussion of application consistent backup. This particular issue came up as a question to the other posts--and simultaneously in a discussion with W. Curtis Preston. And the issue is this: can you get .vmdk level application consistent backups of Microsoft guests under VMware ESX 4.0 with vSphere?

So, long question! Why are we asking this? Remember I concluded that you could not get an image level, application consistent backup, that is guaranteed to be consistent--even with vSphere 4.0.

Now Mr. Preston and my smart reader rightly pointed out that VMware says you can. Thanks in advance to W. Curtis: you can find those claims here and here  on the VMware site.

So what is going on here? Was I dead wrong? Does VMware actually permit application consistent image level backups via VMware Tools and support for the MS VSS provider?

It turns out the answer is simple: no they don't. VMware's support for VSS is for the basic file system level VSS provider. Not for the extended application VSS provider used by Exchange and SQL. So, even when you take an image level (.vmdk) backup of a guest OS, and even when you have VMware Tools installed, all you can quiesce is the file system, not the application. As a result, you may get an application consistent backup, or you may not. It is certainly not guaranteed--because the application itself is not quiesced.

Therefore, final answer: if you need application consistent backups, you must do a guest level backup. An image level backup is simply not good enough. Even with a MS Windows 2003 or higher guest, even though VMware supports VSS. Yes you may do image level backups too, but they will only complement, no replace, guest level backups.

And, as I concluded before, if you need to do guest level backups, there is no better choice than Avamar. The combination of its dramatic reduction in network I/O for backup by as much as 99.5% over a standard full backup, the reduction in backup duration by as much as 90%, and the very significant reduction in net CPU utilization demonstrated by Avamar are simply an unbeatable combination for the backup of consolidated systems like large VMware infrastructures.

Comments

[jarrad]

VCB with VSS does quiesce exchange and sql when an image backup is taken, assuming you have installed the VSS provider installed with vmware tools (by default its not selected). check your windows event logs, you can see the SQL/Exchange quiesce events happening when a vcb backup runs.

[W. Curtis Preston]

I spent most of today looking into this (and I had OTHER things to do, so thanks for bringing this up...). I'll elaborate further about it on my blog at www.backupcentral.com . FWIW, this limitation of VMware is not found in its main competitor: Hyper-V. Hyper-V offers application-consistent, totally supported backups in Windows VMs for any application that has a VSS writer (e.g. SQL, Exchange, Sharepoint, and Oracle). This is the first major competitive advantage I've found of Hyper-V over VMware. And, frankly, I'm appalled that it exists, given that VMware completely rearchitected the product last year to include better backup support. This is just one of the reasons why Hyper-V gained 25 pts of market share last year.

[W. Curtis Preston]

What jarrad says is the first solid proof that VMware IS doing the right thing. I also agree with your updated "tentative" statement now -- and that it's so bad that this is not talked about or documented very much. And I take back my "appalled" comment above as well until this is verified one way or another.

[VMDoug]

Scott,

I agree that VMware's implementation of VSS is not complete (it works fine for Windows 2003 for for W2K8 it's only file-level). This is why some backup vendors, including Veeam, have their own VSS implementation. Yes, this does require a run-time agent to be installed on the Windows OS but it's not the same as a guest level backup. Microsoft's VSS implementation is well documented and any vendor is free to write their own VSS implementation to properly quiesce the OS as well as VSS Aware applications, Veeam decided to do this rather than rely on VMware. An important consideration on VSS is not only the backup, but also the recovery. Only a proper VSS implementation will make sure that a VSS enabled snapshot recovers properly, very important for Exchange and Active Directory. For real-world examples, check out Veeam's forum on this exact topic, from real users: http://www.veeam.com/forums/viewtopic.php?f=2&t=2594 or check out our documentation at http://www.veeam.com/vmware-esx-backup.html#fragment-4

[Chuck Hollis]

Now that's what I like about you, Scott -- you dig into the details that matter, rather than the surface treatments we get from many of the "experts" out there.

Keep up the good work!

-- Chuck

[Sharney]

I am an Avamar and Networker user with ESX and we looked at backup methods before we bought Avamar. We ruled out VCB proxy + avamar as way too complex vs simple guest level backups via Avamar. And honestly, guest level backups remain simple. They're easy for my staff that need to perform data recoveries. They're easy to control. The notion that guest level backups via agents are somehow more difficult to manage is kind of a red herring. And if you have to do guest level backups at all, I see no reason to set up even the virtual guest + vcb option.

If you get image level backups that understand application consistency (and it can be done the same way you do for, say SRDF/A + Oracle DB data on Symm) then that's a nice option. But I, for one, don't see it as some kind of holy grail.

Guest level backups with Avamar work great and have all the advantages you report. Why should I allocate a single lun from my SAN infrastructure to all this VCB complexity? Why is it presumed that image level backups are better. My real world experience tells me otherwise.

[Scott Waterhouse]

Steve;

I agree with everything you said. However, my (very) unscientific survey says: backup admins like guest level, VMware admins like image level. And as a backup guy I often have to articulate to VMware admins why image level has some issues.

[Tina Perez]

That is an excellent point on backup admins preference vs. VMware admins. Here is a question I have.....
How well does the image level backup dedupe compared to the guest level?

[Scott Waterhouse]

Tina; the deduplication ratios should be roughly the same between the two methods. Certainly there is nothing so significant that you would ever choose one over the other for this reason.