VMware Backup with Avamar and vSphere Part 2

On the previous post I had a chance to review the major backup and recovery options for VMware. I put the discussion in terms of Avamar, but a lot of that is just for convenience. If you want, you can substitute your favorite backup application in place of Avamar, and you will get more or less the same result (minus the deduplication), albeit perhaps with greater operational pain and expense.

In any event, now that I have covered off the mechanics of the three major options, it is time to take a look at the why and the how of it all. Why would I want to choose one option over the other? How do we differentiate between them? Simply put: what is the best way to back up VMware?

I am going to start by breaking one of the cardinal rules of rhetoric. I am going to state my conclusion before my premises. And my conclusion is this:

Everybody that runs transactional applications in a VM should do guest level backup. You may also do image level backup as an additional method of data protection, but guest level backup is necessary. If you are protecting anything other than a transactional application with a Windows guest OS, then an image level backup (generated by vSphere 4.0 and Avamar 5.0) is sufficient. If you are protecting a Linux guest OS machine that is not running a transactional application, then you will likely find that guest level backups are sufficient.

Now why would I insist that guest level backup is the preferred methodology if you are running a transactional application within your guest OS?

Simply for this reason: no other backup method guarantees (in a strong sense) application consistent recovery. Yes I need a backup agent that ties into the application APIs (RMAN for Oracle, VSS for Microsoft applications, and so on). And yes this entails complexity and additional components in my backup process--although this complexity is no more than what we all have to deal with in an environment that is not virtualized, because it is exactly the same process as we would use in an environment that has not been virtualized. And it may entail additional expense (for licensing, if you are not using Avamar--which has free application agents). But it does have the unique virtue of guaranteeing application consistency.

And if I choose either of my other two methods of backup--meaning the two variations on image level backup that I described in my previous post--you do not get such a guarantee. Not really. I will admit that it is very likely you will get a consistent backup. Almost guaranteed. 99.99% probable, perhaps. Nearly certain. But the technical person in me is not satisfied with that. Nearly? Almost? Probable?

Not good enough.

vSphere 4.0 and Avamar 5.0 will backup changed blocks from the .vmdk associated with a guest. That will result in an image level protection. That image can be restored to the original ESX host or elsewhere. It is portable. It takes relatively few components to generate. The proxy system that mounts the LUNs containing the changed blocks can now be run in a VM too, further reducing the infrastructure commitment. It offloads the CPU effort of backup from the guest VM, which is a very good thing. All these are true. And all these represent distinct advantages over guest level backup.

But it still does not provide the iron clad assurance that I am looking for that my application will be restartable.

Now I have heard from others that they have tried this and never had a problem. If that is good enough for you, then that is fine with me! But I can't help but get nervous, and I would certainly never recommend that somebody adopt the approach of taking (only) image level backups without understanding this issue, and explicitly accepting that image level backups imply that application consistency is not guaranteed.

As a result, I think that there may be some times when we want to do both. We want to do guest level backups to capture an application consistent backup (and perhaps capture application change logs on a more frequent basis). This will be the basis for our operational recovery.

We may also want to capture an image level .vmdk backup for longer term retention, or archival purposes, or even compliance reasons. Perhaps once a month we will capture these images and archive them (likely to tape--as it is not the purpose of deduplicated backup targets--either Avamar or target deduplication appliances--to hold long term archival data as a rule; nor is it cost effective in most cases). Depending on how we use VMware, and whether it is our primary mechanism for site recovery, disaster recovery, and high availability, these images may also take on other purposes. But their primary intent should not be for operational recovery of a transactional application.

For those VMs who guest systems that do not host transactional applications, Avamar 5.0 and vSphere 4.0 provide the best, most efficient, least complex method of providing backup. I do not need agents in each VM. I can mount the backup data (which is much smaller than under a VCB due to containing block level differences only) to a VM proxy. So I no longer need a physical host to act as the proxy backup server. That proxy is significantly more efficient than the VCB proxy server. In the case of Avamar, the proxy can also deduplicate the data prior to sending it to a backup target. And finally, I can do both file level restore for Windows systems, and image level restore, from this one backup process.

This is a very good thing. This is the best of both worlds. The best of guest level backup (file level restore) with the best of image level backup (simplicity, reduced infrastructure, speed, and minimal to no impact on the CPU of the guest VM for backup purposes).

And as soon as we can guarantee application consistency with this approach, I think that guest level backups will no longer be either required or desired. Finally! (And no, that is not a statement or admission that this capability is coming from either EMC or VMware. I have no idea if it is or not. I do know however that this would be such an enormous step forward for data protection that it has to happen eventually. The advantages are just too big, too overwhelming, to ignore.)

Lastly, for those that host Linux as a guest OS within a VM, your decision tree on which backup methodology to choose is a little more complex. Does the VM contain a transactional application? Then you must do, minimally, guest level backups. Do you need file level restore? Again, you need guest level backups to do file level restore. But if you are not running a transactional application, don't care about guaranteed application consistency, or don't care about file level restore, then image level backup is the right approach.

By the way, there are lots of other great points of integration between vSphere 4.0 and Avamar 5.0 There is automated policy protection for VMs (automatically protect new VMs as they are added--which just might mean that I won't find environments that have "forgot" to protect 20% of the their systems any more!). There is a way of browsing the type of protection a VM has. The image level backups are automatically generated (no more or the scripting that accompanied VCBs previously). All kinds of good stuff.

But set all that aside. Before any of that matters you have to decide how to back up a VM. And I hope that this post and the previous one help you to better understand those choices. Choosing Avamar specifically should be secondary to the architectural decision on how to back up.

But what other backup application will reduce the time to backup by 90%, the net CPU load by 95%, and the amount of data transmitted over the network by 99.5% or better? All of which are uniquely beneficial for VMware...

Comments

[Madhav Mutalik]

Scott Waterhouse wrote:
"Simply for this reason: no other backup method guarantees (in a strong sense) application consistent recovery. Yes I need a backup agent that ties into the application APIs (RMAN for Oracle, VSS for Microsoft applications, and so on)."

Is this true if you are using vSphere 4.0 (with vStorage APIs)? I thought vStorage APIs were integrated with VSS on Windows guest operating systems, and thus would guarantee application consistency on Windows.

[Scott Waterhouse]

Madhav;

It is an excellent question. A couple other smart people have asked me this as well, and I haven't posted up anything yet because I don't have a definitive answer yet. There seems to be a discrepancy between what I get from some sources vs. others. I am actively pursuing this and will post something definitive when I think I have that definitive answer. Stay tuned!

[Peter Elliman]

Scott - If I use the approach you suggest here (I assume vStorage backups), and I have to recover a file amongst 100s of VMs, but don't know on which of 5 VMs a file exists, how do I find that file? Could you walk me through that recovery process?

First, how do I find the file amongst the VMs?

Second, how long does it take to recover that one file if I've been doing incrementals for 2 weeks?

Third, is there any difference in recovery if I moved the data to tape (or maybe that's not possible with Avamar - I can't remember)?

Thanks.

[Scott Waterhouse]

Peter;

Well, I am not sure how you could have a missing file but not even know where it is from?

Having said that, I can search for it in Avamar (assuming I know the name... if you don't know that and you don't know where it is from how do you know it is missing?).

With Avamar it would take about 2 seconds to recover--there is no penalty for an "incremental" because there is no such thing as an incremental with Avamar.

It is possible, and no, conceptually, there is no difference (just the seek/load time that makes tape so awful to deal with in the first place).