The Backup Blog

Inside Avamar: Fast NAS and NDMP Backup

Last time out, we went over the reasons why Avamar is different and faster than other backup applications for backing up unstructured data sets.

This time out, I thought we should turn our attention to NAS (filer) backup under Avamar. Again, we see big differences in how Avamar handles NDMP backups. Again, we see that Avamar backups can take only 10% of the time of a standard NDMP backup. But it turns out that the reasons for this are totally different than the reasons that Avamar is so fast at unstructured data backups.

First, a bit of background. Avamar doesn't really do full or incremental backups. Avamar does a backup. That backup data is written together with existing backup data on an Avamar server in a way that a full system image is present for every day on which a backup occurred.

Inside Avamar: Fast Backup and Finding Backup Candidates

Last time out, we talked about Avamar's global client deduplication. This time, I want to go into some more depth on what makes Avamar so fast for some types of backups.
 
Saying something is "fast" is always a bit subjective. I know that some things like Lamborghinis are fast. I know that Valentino Rossi is fast. But how fast is Avamar?
 
On average, Avamar is about 90% faster than most other backup products to make a backup of an unstructured data set. And in backup, speed matters. In fact, I would argue that it is the most important thing a backup application or platform can offer. So finding a solution that lets you cut backup times by 90% is important. And in some cases, it is crucial. For platforms and infrastructure that have expreienced 24+ hour backup windows, this is a decisive factor.
 
How does Avamar do it? Well, Avamar has a pretty elegant way to reduce the amount of time that it takes to determine what needs to be backed up.

Inside Avamar: Global Client Deduplication

I spend a lot of time talking to people about Avamar. I think it is a great application that pretty radically reinvents backup architectures. It does so many things well, and has consistently been ahead of the curve in terms of delivering advanced backup technologies. For example, Avamar was one of the first to market with a true disk based deduplication solution for backup.
 
And often I tell people that Avamar's overt association with deduplication is one of its' challenges too—in the sense that Avamar actually does two things at the client level that are really important. It is not just deduplication, but the duration and resource utilization of the Avamar client that are so impressive. Not only does deduplication reduce the amount of data to be backed up by 99% or more in many cases, but the duration of an Avamar backup is usually 90% less than the duration of a standard backup. To those of us that are concerned about growing amount of data and the impact on backup windows, this is a fundamentally important characteristic, and one that is, in my opinion, just as important as deduplication itself. (And one that I will discuss in a follow up post.)

EMC Avamar v6.0

Today EMC is announcing a new version of EMC Avamar software and hardware. With the latest release come a ton of new features, bigger, better, faster hardware, and support for integrating Avamar with Data Domain.

So, without further preamble, let's jump into what is new in Avamar v6.0:

On the hardware side, the new release offers higherdensity nodes with greater performance. Nodes are now offered in 4 different capacities: 1.3 TB, 2.6 TB, 3.9 TB, and 7.8 TB. The standard Utility and Accelerator nodes continue to be available and serve the same function in Avamar v6.0 as they did in previous releases. What is more, is the performance of the node relative to important internal system maintenance and upkeep scales with capacity. The 7.8 TB node offers twice the performance of the 3.9 TB node. For some important tasks like garbage collection and file system checks this means there is no longer any good reason not to use the highest density nodes even for data with exceptionally high change rates.

Avamar and VMware Backup Revisited

I thought it might be useful to circle around on this one more time with some definitive thoughts. My apologies for the delay on this--a week of intensive BRS training (EMC's new Backup and Recovery Systems business unit formed out of the union of BURA, the previous EMC group to focus on backup, recovery, and archive, and Data Domain) took me away from my blogging.

So where we left things was this: if you want to do image level backups with vSphere 4.0 the present state of the art does not allow an application consistent backup (without the injection of an agent into the guest OS). The underlying reason for this has to do with the present state of VSS support in VMware Tools. Although I am optimistic that VMware will improve their support in the future, and address this issue to the benefit of backup applications like Avamar, at present they only support VSS snaps on Windows 2003 guests (and do not support the extended application VSS writers at all under Windows 2008).

In practical terms, that means we cannot get application consistent backups of Windows applications. However, we can get crash consistent backups. This is actually a pretty important distinction, and I wanted to raise it because I am not sure W. Curtis Preston did enough to describe this distinction in his post on the subject. And crash consistency is a heck of a lot better than corrupted backups (which is what I fear that many people might infer happens if you don't get application consistency).

VMware Backup with Avamar and vSphere

Edit: I am taking the unusual step of adding this after the original post was finished. As of the moment, it may be best to regard the conclusions in the post as tentative only. The post has generated a flurry of questions and controversy, with several people saying that I am wrong. Unfortunately, nobody has been able to provide a piece of definitive technical documentation (a white paper, or a support document, or relevant piece of text from an administration guide) that clearly describes the issue. Let me be clear: I am OK with being wrong. I kind of hope that I am--it would be just fantastic if we could get image level application consistent backups. But it is not OK that the answer to such an important issue is so obscure. So at the end of the day, if I am wrong but manage to get a clear and well documented answer, that is good enough for me! S.W.

Continuing the posts here and here, I wanted to have a very quick discussion of application consistent backup. This particular issue came up as a question to the other posts--and simultaneously in a discussion with W. Curtis Preston. And the issue is this: can you get .vmdk level application consistent backups of Microsoft guests under VMware ESX 4.0 with vSphere?

So, long question! Why are we asking this? Remember I concluded that you could not get an image level, application consistent backup, that is guaranteed to be consistent--even with vSphere 4.0.

Now Mr. Preston and my smart reader rightly pointed out that VMware says you can. Thanks in advance to W. Curtis: you can find those claims here and here  on the VMware site.

So what is going on here? Was I dead wrong? Does VMware actually permit application consistent image level backups via VMware Tools and support for the MS VSS provider?

VMware Backup with Avamar and vSphere Part 2

On the previous post I had a chance to review the major backup and recovery options for VMware. I put the discussion in terms of Avamar, but a lot of that is just for convenience. If you want, you can substitute your favorite backup application in place of Avamar, and you will get more or less the same result (minus the deduplication), albeit perhaps with greater operational pain and expense.

In any event, now that I have covered off the mechanics of the three major options, it is time to take a look at the why and the how of it all. Why would I want to choose one option over the other? How do we differentiate between them? Simply put: what is the best way to back up VMware?

I am going to start by breaking one of the cardinal rules of rhetoric. I am going to state my conclusion before my premises. And my conclusion is this:

Everybody that runs transactional applications in a VM should do guest level backup. You may also do image level backup as an additional method of data protection, but guest level backup is necessary. If you are protecting anything other than a transactional application with a Windows guest OS, then an image level backup (generated by vSphere 4.0 and Avamar 5.0) is sufficient. If you are protecting a Linux guest OS machine that is not running a transactional application, then you will likely find that guest level backups are sufficient.

Now why would I insist that guest level backup is the preferred methodology if you are running a transactional application within your guest OS?

VMware Backup with Avamar 5.0 and vSphere 4.0

One of the most important, and most frequent questions I get asked is this: what is the best way to back up VMware?

Well, "best" tends to be a bit of a slippery term. Usually the answer to the question should be "it depends." But I thought I would take the time to explore what it really means when we say "it depends." What does it depend on? What are the pros and cons of the different approaches? What are the most important considerations? And what new choices do we have with vSphere 4.0?

Now I am going to explore the answer in the context of Avamar. Mostly I am going to do so because I work for EMC and it is convenient to be able to put a label on these products and processes. Having said that, almost everything I say about the costs and benefits, the value of the various approaches with respect to each other, and the drawbacks of the various approaches is true irrespective of the backup product you are using. Guest backup has certain costs and certain benefits--no matter if you use Avamar or something else. Where benefits are specific to Avamar I will identify them. And certain things that that I identify as unique may only be unique as of the time of this writing; clearly some of the functionality is so desirable that other backup application vendors will be trying very hard to get this capability into their shipping product.

All that said, I think there are two primary methods of protecting a VM: guest level backup and image level backup. There are a couple of ways in which the number of options multiply however--do you use source deduplication or target deduplication? Do you want to use the array to generate a copy for image level backup, or do you want to rely on ESX/vSphere?

Avamar v5.0

Today EMC formally announced the availability of Avamar 5.0, which is loaded with new features and functionality. I am going to briefly review some of them here, focusing on the changes that I think will be most significant from a customer's point of view.

Having said that, please don't hesitate to ask questions or post feedback in the comments section if any of you are interested in any clarification on the material.

So what is new?

The big news is: bigger nodes, best server side task scheduling, more efficient checkpoints, new clients, desktop and laptop support, major performance enhancements for a variety of clients, and enhanced support for VMware environments.

That is the high level. Let's dive into these in a little more detail.

Source Deduplication Issues and Concerns

The always interesting and informative Preston de Guise has a discussion of source and target deduplication on his blog "The NetWorker Blog". Which is just about as imaginatively named as this blog. Well, he may not be a marketing mad man, but Preston is one of the most informative and most informed people I know writing about backup and recovery.

However, he does raise a couple of concerns about source deduplication that I would like to address. Now source deduplication for EMC means Avamar. Actually, for most everybody it means Avamar, because there really is no other credible source deduplication alternative today. There may be software deduplication solutions--but most of them tend to be target solutions in one way or the other. And not terribly technically or architecturally credible. (Yes, I am thinking of you, TSM.) With vanishingly small market share to boot.

So Avamar it is.