« So They Do Understand... | Main | Fun With Tape, Part 2! »

September 04, 2008

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

W. Curtis Preston

While I understand your reasoning, I need to see the math behind your chart.

The only way the initial backup could finish twice as fast is if you have a device that can ingest data twice as fast.

The DD690 has an ingest rate of 1.4 TB/hr, almost the same as the DL3000's ingest rate of 1.44 TB/hr. The 4000 series has faster numbers, but you didn't mention them. (Are they shipping yet?)

Scott Waterhouse

Fair comment. For the math behind the chart, refer to my post on DL3D performance: http://thebackupblog.typepad.com/thebackupblog/2008/06/dl3d-performance.html

As always, these are real world numbers. They aren't made up, they aren't something imaginary based on total rated throughputs of the busses, or some such nonsense, they are real.

So, that being said, 1.44 TB/hr is the IP in-line ingest capability of a DL3D 3000, 2.5 TB/hr is the FC scheduled ingest capability. Meaning that data is initially written at 2.5 TB/hr (undeduplicated), and then after the fact it gets deduplicated. Coincidentally, the performance of this post process deduplication is also 2.5 TB/hr. And I say coincidentally in the technical sense: it is, but there is no necessary correlation there; with different hardware, processors, etc. it might be different than the initial write speed.

W. Curtis Preston

I really don't mean to sound argumentative, but I've been all over the EMC product pages for the DL3D series, and the only place I can find the numbers you're talking about are on your blog.

If I look here:
http://www.emc.com/collateral/hardware/comparison/emc-disk-library.htm

Or if I look here:
http://www.emc.com/collateral/hardware/specification-sheet/h4411-dl3d1500-3000-ss.pdf

I read 1.44 TB/hr as the ingest speed. I see no reference to the 2.5 TB/hr that you are saying.

There's also a comment I don't understand and I think it's a typo: "For our purposes, I have assumed that deduplication will slow down in-line deduplication modestly to 1.2 TB/hr." Do you mean replication?

You depict replication occuring completely after the backup, but you say that it can (and does) occur during backup. This matches my understanding of all post-process implementations. The question is does doing so affect the dedupe speed? If so, then I'd say your chart is a realistic depiction of time. If replication running at the same time as scheduled dedupe would NOT slow down that dedupe, then I'd say your chart could actually drop off the two hours for replication. If you're deduping at 2.5 TB/hr (again, your number), and you have a 20:1 dedupe ratio, that means you've got 125 TB/hr to replicate, which is only half of the 250 TB/hr number you're quoting. That would put your overall ingest, dedupe, and replication time at 8 hours, not 10.

But something tells me that if you could depict it that way you could. ;) So what's holding you back? Does simultaneous replication slow down dedupe?

Scott Waterhouse

That is OK, because I am not taking it as argumentative.

I haven't looked for additional documentation of the performance metrics I described, but it doesn't surprise me that they are unpublished (externally). Typically EMC will quote worst case performance metrics. There are a variety of reasons for this, but I think it is a sound policy in terms of setting reasonable expectations, and not being forced to commit to a performance metric that requires very specific circumstances to realize. For example, the EDL4106 can achieve more than 15% better performance than we publicly committ to. And it can do so under real world circumstances. However we err on the side of caution.

Having said that, I can tell you the metrics I cite are well founded--and very thoroughly documented internally. And no, I can't publish those studies externally.

So at the end of the day, I will have to ask you to accept that I have very specific supporting evidence, and that the numbers are real. If for whatever reason that makes you uncomfortable, that is OK too. I would just add that I fully understand my professional reputation, and to a lesser extent, that of EMC, is at stake, and that is something I take very seriously.

In short: the numbers are real.

As far as: "a comment I don't understand and I think it's a typo." It is a typo, and I corrected the text of the article for clarity; it should have read "replication" not "deduplication".

Finally, with respect to replication and deduplication. I adjusted the deduplication (in-line) number down, modestly, from 1.44 TB/hr to 1.2 TB/hr to reflect simultaneous replication. For post process, the same thing happens: the data is backed up to disk, but once the deduplication of that data starts, replication also begins simultaneously. I chose to depict one process as two just to illustrate the steps, and depict a *worst case* scenario in terms of ellapsed time. It may run faster, but if I give you the worst case, I know I won't have to backtrack. But in either case (in-line or scheduled/post-process) yes replication will impact deduplication speed because deduplication speed is CPU bound. Replication takes CPU cycles, so it will absolutely impact deduplication. I have adjusted for that in both cases.

For the actual numbers: 10 TB at 20:1 is 500 GB. Yes that could go at the same time as deduplication, and my time might be shorter. Again, I picked the worst case for the sake of illustration and so that nobody (no competitor) can accuse me of creative exaggeration (marketing). So my overall, worst case, is 10 hours. Will replication degrade my deduplication from 4 hours to 6? No. I would also benefit from having as many as 8 GigE ports on a 3000 too. So potentially, post-process deduplication could finish the backup and replication as quickly as in-line deduplication. But I would have way too many caveats to be able to make a reasonable case that this would normally happen.

W. Curtis Preston

If your claim of 2.5 TB/hr is true, then your illustration is indeed an example of how a post-process implementation of dedupe can indeed do the same job as an inline device at the same or greater speeds. It's not about the parts you chose. It's about the results you get. As I've often said, "I DON'T CARE" if you're inline, post-process, forward/revere-referencing, hash-based, delta-based, etc. What I care about is: How big is it? How fast is it? How much does it cost?"

I still have to go back to what's being published vs. what you're saying, and it can't be explained away with "EMC always publishes worst case." I'm not saying your numbers are false. Either EMC has inconsistently written data sheets, or some of the numbers I'm given are wrong.

EMC seems to have no problem publishing even better than best case ingest rates for the 4100. EMC's site says that it can do up to 2200 MB/s (8 TB/hr), which is only possible when dedupe is deferred until after backups. (This would be analogous to your 2.5 TB/hr number for the 3000.) If you plan on deduping everything, it can only support that number for about five hours. That would store 40 TB, which the 4100 will then take about 16 hours to dedupe the 40 TB at 2.5 TB/hr, leaving 3 hours for maintenance.

So... Why is EMC publishing a number the 4100 can only support (if you plan to dedupe everything) for 5 hours a day -- a time period that is less than half most people's backup windows -- but they're (according to you) choosing NOT to publish a number that the 3000 could actually support for a longer period of time?

As to your statement that EMC always quotes worst-case... That's a funny one. I'll have to tell that one sometime. ;)

Scott Waterhouse

For the 4100, the published rate is 1,100 MB/s. I just checked. I think you are confusing it with the rate for the 4400, which is 2,200 MB/s. We actually haven't published numbers at all for the 4000 3D series. So no consistency problem.

Sarcasm aside (ahem) I understand that you are puzzled (to put it charitably) by the 4000 3D series. So, why this architecture? 1) Backup performance. Speed matters. The diagram I provided in this post indicates why backup speed is *sometimes* more important than dedup/replication speed. 2) Restore performance. Restoring data from the 4000 VTL is an order of magnitude faster, or more, than restoring it from a dedup box. I would argue that matters even more than backup speed to some people/applications. 3) Service level. The system performs in a deterministic fashion for backup and recovery so that for the policy/service level you are after, you know what level of response and performance you are going to get for both dedup data and non dedup data.

Is it perfect? Not in the sense of having a system that does 2,200 MB/s of in-line deduplication. I would love to have a system to talk about that dedups at that speed. But it doesn't exist, not from us, not from anybody.

W. Curtis Preston

I wasn't trying to get into the whole DL3D4000 architecture just yet. ;) If I was, I'd probably use the term "bewildered" instead of "confused." I understand it. I just don't get it. ;) I'll save that for my blog on http://www.backupcentral.com.

My point was that EMC's documentation on the *DL* line is not in keeping with what you're saying. It's not that they're not advertising the numbers you are (although they're not), but that sometimes they seem to advertise aggressive numbers and sometimes they don't.

As to my confusion on the numbers for the 4100, it's back to the docs provided by EMC. The spec sheets says nothing about VTL throughput. I think those throughput numbers it gives are individual disk drive numbers (45 MB/s)? What's up with THAT?

So to get numbers, I go to http://www.emc.com/products/detail/hardware/disk-library-dl4100.htm , and click on "Data Sheet." That opens up http://www.emc.com/collateral/hardware/data-sheet/c1156-disk-library-ds.pdf , and THAT document gives the 2,200 MB/s numbers. Now I realize that was talking about the 4000 family. The only place where you see model-by-model numbers is here: http://www.emc.com/collateral/hardware/comparison/emc-disk-library.htm .

And if I go to that page, I see the number I'm discussing (2200 MB/s), with a line that says that dedupe will be/is provided by the DL3D 4000. So I still think my point is valid, although my model number was wrong. EMC is being inconsistent with their advertised numbers.

As to your claim that "restoring data from the 4000 VTL is an order of magnitude faster, or more, than restoring it from a dedup box," I'm going to have to call BS. Since the best case you're advertising is 2200 MB/s, an order of magnitude difference would mean that no one else is going faster than 220 MB/s. And YES, I'm talking about restore speeds.

I'm assuming that you are hinting at the fact that the 4000 will be able to store (and therefore restore) recent data in its non-deduped format, and so it should be able to restore at line speed. The problem with your "order of magnitude" statement is that it's not the only product that can do that, and some of the other products are already GA. Both Quantum and SEPATON store recent data in their original format are already GA. You're joining a trend -- not starting one. ;)

"I would argue that matters even more than backup speed to some people/applications"

I couldn't agree more. No one cares if you can backup. Only if you can restore.

Scott Waterhouse

OK, well perhaps we can agree to disagree. I agree that I have more information here than is available on the EMC website and through .pdfs found there. On the flipside, I disagree: all numbers on the official EMC site are real and conservative measures of performance. If I am reading you correctly, we would just have to provide detailed metrics for the 4000 3D deduplication in addition to the standard 4000?

So lets be blunt: everybody could do a better job at this. I can't think of a single vendor that publishes what I would term adequate performance metrics for their system. Without wanting to toot my own horn, I would say that the metrics I provide are more detailed that you would normally find anywhere else. So I can appreciate your frustration. I would love to see vendors (all of them) come clean on what backup performance is, under various loads, with various numbers of streams, various numbers of disk trays, etc. Likewise for restore.

And speaking purely for myself (and don't in any way shape or form take this for an EMC statement) I would love to see everybody adopt a state of full disclosure here. I could write a new post every day for a month with neat performance information. But I don't set EMC policy, and I find it awfully tough to find fault with those that do given that we are actually just as good (better via this blog) than everybody else. Until that time, I will publish, discuss, analyze and criticize as much of our internal data as I am allowed to disclose, and as much of anybody else's that I can legitimately procure. In my opinion, neither of these are anywhere near "enough", but it is all I have.

Failing total disclosure, a little more honesty would help a lot. Sepaton doesn't publish any performance metric except initial ingest (600 MB/s per SRE) that I am aware of. They make no mention of restore performance, or even deduplication performance for that matter (and no I don't believe they dudup at 600 MB/s). Also Sepaton only restores at full speed for the most recent backups. 2 backups ago? You get a dedup speed--which is what, exactly? Only they know, and they aren't telling. Quantum may store data in (semi) native format until it gets truncated, but I can absolutely assure you that restoring it from that state is not as fast as initial (line speed) ingest. If they want to provide more detail in that respect, then that is their call.

So, lets assume that all the conditions are right to restore at 2,000 MB/s from a DL4406. Meaning, primarily, that you have a sufficiently large number of virtual drives engaged--there is only so fast a single stream will go. Now, how many vendors can honestly claim to be able to restore faster than 200 MB/s? Sepaton, for yesterdays backup. And... That's it. (I guess you could lump HP in there too, as it is a Sepaton derived box.) Data Domain? No way. Quantum? Don't think so. Falconstor? Not a chance. I am running out of alternatives here...

The comments to this entry are closed.

Search The Backup Blog

  • Search

    WWW
    thebackupblog