DL3D: The Benefits of Immediate Deduplication

One of the most powerful features of the EMC DL3D is its ability to do both in-line and post-process deduplication. It is one of the few (the only?) deduplication appliance that offers users this flexibility, and in fact lets you do both on one appliance, depending on the requirements of the data being backed up.

Certain competitors have done their best to portray this flexibility as a weakness. Huh? Since when is choice a bad thing?

Well, it isn't a bad thing at all. Choice is good. Flexibility is good.

The other criticism that gets leveled against the DL3D here is that the in-line is not really in-line. In particular, Data Domain attempts to claim that EMC doesn't do in-line deduplication at all--that we write to a disk cache, and that this lets us be deceptive with our performance numbers.

Nothing could be more misleading.

So I wanted to take on the issue of why we took the design approach we did. Because if we understand what is going on underneath the covers, we can get a glimpse of just how powerful the DL3D can be, and how much better the device can meet the practical day to day needs of backup and restore administrators.

So what is really going on when you write the data to a DL3D?

Simple, there are two basic modes of operation.

The first is in-line or immediate deduplication. Immediate deduplication waits for 250 MB of data (or about 3-4 seconds at Gigabit Ethernet speeds) before deduplication begins. Data is written in its "native" format, and deduplicated in-line. When this mode is enabled, we can sustain the performance metrics I described here. No smoke and mirrors. Immediate deduplication can sustain speeds faster than a Data Domain DD690. We don't use cache to beat their benchmark. We don't need to.

The second approach is delayed deduplication. We can also call it deferred, or out-of-band. No matter what you call it, the approach is simple: all backup data is written in its native format, and is stored that way until such time as your DL3D policy determines that it is time to deduplicate it. Why do we do this? Again, pretty simple: because a native write can happen a lot faster. We can get the CPU out of the data path, and deduplication is no longer bottlenecking throughput. Typically Data Domain doesn't have a lot to say about this mode because, well, they don't offer it. However, for reference, you can read here and see some of the strengths and weaknesses of this approach relative to in-line deduplication.

Now here is the kicker: regardless of which approach you take, the DL3D will have data written in native format. And it will not immediately delete this data, even after it has been deduplicated. This process, called "truncation" will happen on a scheduled basis. And there is a very good reason for keeping the original, native format data around: fast restores.

Again, Data Domain misses the point, because they don't ever have native data on disk. So when it comes time to restore data from a DD box, you are always restoring deduplicated data. Which is relatively slow when compared to the backup speed. Since Data Domain doesn't publish restore speeds, I will have to rely on what customers tell me: data reads from a DD system are typically 25-33% of the speed of data writes. If you have a DD690, this means that you can't expect to be able to restore data faster than about 100-120 MB/s.

But this is where they get it wrong when it comes to the DL3D. In their haste to assume that the DL3D is all bark and no bite, they have assumed that whenever EMC discusses restore performance, we are discussing restore from native data to make us look better than we actually are. This is, of course, untrue.

The restore performance of a DL3D has two key metrics. First is the restore of data that has been truncated. Data that is being restored by reconstructing its meta-data. Which means that you are restoring from data that has been deduplicated. In this case, the DL3D performs very similarly to a Data Domain box. This is our worst case: the slowest restore speeds on a DL3D come from deduplicated data, and a DL3D 3000 will be every bit as fast as a DD690.

The second key metric is based on the restore of native data. If the native data has not been truncated (and I will go into the circumstances in which that is the case in a follow-up post) then the restore of the data is nearly as fast as the backup. In other words, there is virtually no penalty at all to restoring this data. Put another way: it is 4 or more times faster than any restore from a DD690.

And now we see why we "cache" data even during immediate deduplication: because by doing so, we have the ability to do restores 4 times faster than otherwise. And 4 times faster than the pure in-line deduplication only vendors that do not keep a native copy of the data at all.

Comments

[W. Curtis Preston]

You can say that your product does “immediate” deduplication, or “concurrent” deduplication, but saying it does inline deduplication is misleading. It’s the equivalent of calling an asynchronous replication “synchronous” just because the target system is updated seconds after the source system. You’re not synchronous unless you hold up the original write until it’s replicated, and you’re not inline dedupe unless you dedupe before you write to disk. It’s also the equivalent of Symantec calling Continuous Protection Server continuous data protection (CDP) because it’s “continuously protecting the data.” You can’t recover to any point in time (you can only recover to when you last took a snapshot), so it doesn’t meet the definition of continuous data protection (CDP) – and your product doesn’t meet the definition of inline dedupe.

The definition of inline deduplication (which was decided at least five years before EMC entered the target dedupe market by OEMing Quantum) is dedupe that is done in such a way that the native, non-deduped data is never written to disk – ever. Since your own description of how your dedupe system works is that it, “waits for 250 MB of data … before deduplication begins. Data is written in its "native" format, and deduplicated…” you are not doing inline dedupe.

You are doing post-process deduplication with a very small unit of work and processing it immediately after backups. Where most post-process dedupe systems will wait until a virtual tape is put back on a virtual shelf -- or will wait until a backup file is closed (NAS) – before starting dedupe, you’re starting it as soon as you get 250 MB of data – so you’re starting the process a bit sooner. You’re still processing the system after (post) you write it to disk – hence the term post-process.

I’m pretty sure I know what you’re thinking: post-processing systems wait until the entire backup is done, THEN dedupe it. This is a misconception that was started by inline dedupe sales reps and was never based in fact. Every post-processing vendor I know has always been able to process the backups as they’re coming in – just like you and Quantum can. You also have the choice to wait until all backups are done to start deduping data.

I decided to put the rest of my response in my blog:
http://www.backupcentral.com/content/view/200/47/

[Scott Waterhouse]

W Curtis... I welcome the debate, but... I think there are two levels to this conversation: one is what the user cares about. And honestly I think that just boils down to a performance conversation--in the sense of throughput, time to finish backup, and time to finish replication. The other level is the technical minutiae of what is really going on with the device. The second is also partly semantics in this case.

Let me start with the second level. You say that you don't want the market confused. Fair enough. Neither do we. Having said that, I think you are trying to fit a square peg in to a round hole in order to define something. (By the way, we are fine with the "immediate" definition if you prefer that to in line.)

Why do I say that? You further wrote: "the second reason that the differentiation between inline and post-process is important is that post-processing systems can get “behind” and inline systems cannot." Well, the DL3D can't either. When it is running in "in-line" mode it will not build up a backlog of data on cache. It will deduplicate data as it is received. If you send it data faster than it can deduplicate, it will bottleneck (and slow down the reception of data). Just like any other in line system.

So honestly, the DL3D meets at least one of the tests you proposed for in line deduplication.

Now back to the first level--and the reason I brought up the subject in the first place: why does it matter to users? Only for performance. And in this respect, the DL3D approach to in line (or immediate) deduplication has a huge advantage over competitive approaches. Our approach allows you to restore data up to 6 times faster than you can from an appliance that doesn't employ any sort of disk cache (like Data Domain). So it has every performance characteristic of an in line solution (limit on write speeds, replication is hooked to deduplication and happens simultaneously, restore from truly deduped data is 1/4 the speed of a write to the system, etc.) except that if you are restoring from cached or non-truncated data, you can get up to a six times performance improvement. And it seemed to me that was worth mentioning.

[W. Curtis Preston]

As I just said in my response to your latest comment on my blog, I disagree that the 1/4th or 1/6th restore numbers that you're stating are typical of the industry. I have seen completely different numbers than that with a number of your competitors.

[Scott Waterhouse]

Fair enough. Our experience differs.

Having said that, there are so many components to the performance conversation that it can be tough to make comparisons: how many streams on back, how many on restore, over FC or IP, in-line (immediate) or delayed, how old is the data, how fast is the client, etc.

At the end of the day, it is important for vendors to honestly communicate these factors.

How come *no other vendor* posts *any* meaningful performance data? Do I think we go far enough at EMC? Personally? No. But we do disclose far more than others.