Avamar and VMware Backup Revisited

I thought it might be useful to circle around on this one more time with some definitive thoughts. My apologies for the delay on this--a week of intensive BRS training (EMC's new Backup and Recovery Systems business unit formed out of the union of BURA, the previous EMC group to focus on backup, recovery, and archive, and Data Domain) took me away from my blogging.

So where we left things was this: if you want to do image level backups with vSphere 4.0 the present state of the art does not allow an application consistent backup (without the injection of an agent into the guest OS). The underlying reason for this has to do with the present state of VSS support in VMware Tools. Although I am optimistic that VMware will improve their support in the future, and address this issue to the benefit of backup applications like Avamar, at present they only support VSS snaps on Windows 2003 guests (and do not support the extended application VSS writers at all under Windows 2008).

In practical terms, that means we cannot get application consistent backups of Windows applications. However, we can get crash consistent backups. This is actually a pretty important distinction, and I wanted to raise it because I am not sure W. Curtis Preston did enough to describe this distinction in his post on the subject. And crash consistency is a heck of a lot better than corrupted backups (which is what I fear that many people might infer happens if you don't get application consistency).

So what does it mean to get a crash consistent backup?

Well, a crash consistent backup differs in two ways:

• First, it can be recovered to an application consistent state if you are willing to do apply the logs to the recovered application. Essentially the logs will play to determine the last consistent state of the application, and that is where you will be left. Generally that is all you will have to do (although for Active Directory there may be additional actions required).
• Second, and somewhat tangentially, you may need to groom the application post-backup. This kind of backup does not truncate the logs, inform the application that it has, in fact, been backed up. Nor does it flip the archive bit so that other backups or applications will detect that a backup has happened. Of all those, the first is probably the most important: it means that a script or administrative action will have to occur to trim the logs so they don't overflow the storage space.

So, to conclude: an application consistent backup is not possible at the image level, but this is not all bad. We now have an answer that is more shades of grey--or as us technical guys will often say when asked a question: "it depends." Do you value application consistency first and foremost? Then Avamar and guest level backups are the right approach for you? Do you desire the value image level backups more than guest level, and are you willing to tolerate a crash consistent machine state? Then image level backups are the right approach.

From my perspective, my suspicion is that most people will opt for guest level backups. They are mature, stable, and offer a wide range of tools that support APIs for application backup like VSS. And with Avamar they have a trivial impact on CPU resources, even less of an impact on network resources, and happen as much as 90% faster than with a traditional backup application. And most compelling of all: you don't have to change anything about your backup methodology as you move from physical to virtual. Never underestimate the power of inertia. Again, my opinion only, but I think image level backups will be reserved for DR and archival purposes for the near to medium term.

Comments

[W. Curtis Preston]

I don't think I missed that distinction at all. I just don't think that crash-consistent backups have any place in a datacenter: http://www.backupcentral.com/content/view/293/47/

[Tim Linerud]

The problem with guest level backups in an VMware environment is scalability and management. The advantage of using the native VMImage backup is that you don't have to install Avamar agents on each and every guest machine. That in itself might be worth the down side of not getting a "crash consisitent" backup, but if your deploying guest via templates, then the application is already covered, it's just the application data.

This is for "file system" backups, but if you need modules like SQL, SharePoint, Oracle, or Exchange, then you'll need to do guest level backups, with the appropriate plugin for those applications.

[VMDoug]

Yes, all very interesting indeed. What is the definition of agent? Is it something that is required to be installed on the OS and managed (persistent)?

What about vendor VSS implementations (like Veeam) that use a non-persistent, backup runtime executable that doesn't actually backup the GUEST but perform the PROPER VSS application quiescing accross Windows 2003, 2008, 2008 RS 32 and 64?

I agree that VMware does NOT provide this, but other vendors DO properly support VSS at the application level and there's no need for a guest AND image level backup for Windows Servers and applications that support VSS.

You DON'T need to do guest level backups if your image level backup provider (like Veeam) properly implements VSS and doesn't rely on VMware to support Microsoft systems.

[Chad Skinner]

I just purchase a Gen3 Avamar Data Store, and am running the latest public release of 5.0 software. I am a big fan so far on most fronts, but am beyond disappointed by the absurdly slow pace at which EMC updates the software to support OS releases and service packs. Two of the Domain Controllers in my environment are 2008 or later. First a Windows Server 2008 server with service pack 2 installed, and second a Windows Server 2008 R2 server. Both complete their backups "with exceptions" because Avamar does not yet support service pack 2 or 2008 R2 yet. Had I known this before purchasing Avamar, I may not have purchased it. I guess it is shame on me for not doing detailed enough research before buying. I can understand no support for 2008 R2, but lack of 2008 service pack 2 support, is just embarrassing (or should be) for EMC. I guess my point is this: sure you have a much better chance of getting application consistent backups by doing guest level backups, but with Avamar you had better be willing to wait around for an eternity for support. Service pack 2 for 2008 was released to the public in May of 2009. Over six months have passed, and still no support from Avamar. Sad, just sad.