The Backup Blog

Backup Cost Reduction, Target Deduplication, and TSM

Two posts previous I was discussing the TCO of backup, the differences between hard and soft costs, and the qualities of a good business case. The general conclusion? Most business cases depend on hard costs (and don't recognize soft costs). If hard costs show an ROI of less than 18 months, most CIO/CFO types will love it. If the business case relies on soft costs to get there, it will probably be rejected.

Looking at the particular example from the previous post, the business case Tony Pearson used as an example showed that 81% of the savings were due to soft costs. This is really bad. In most business environments, such a business case would be rejected out of hand.

I thought it might be instructive to have a look at a better business case: a business case based almost entirely on hard cost (savings). Two of them actually. One for source deduplication, one for target deduplication. In both cases, they are real environments, with real costs, real savings, the proposals were made with the costs and savings exactly as described, and they were accepted by the customer and the projects were funded on that basis. I have only removed names and specifics to the extent necessary to anonymize the customer and any technical or business characteristics unique to that customer. And, as I wrote the TCO tool itself EMC uses for target deduplication, I am more than happy to answer any questions on the data, the assumptions that drive it, or the calculations and formulas that we used to determine the benefits and cost savings of the solution.

In this post, I am going to deal with target deduplication. The business case for a source deduplication solution will be the subject of a future post.

Cloud Backup Event

I have been running a poll for a few months now that asks the question: would you back up your business data to the cloud? I have to admit that I find the results a little surprising, but they have been consistent over time, and are relatively consistent across geography. Approximately 50% of respondents have indicated that they are willing to backup up to the cloud.

With the latest release of NetWorker, EMC has introduced cloud backup as native functionality. Right out of the box, NetWorker users can clone backup data to a target in the cloud. EMC is hosting a Live Event that discusses the functionality in greater detail.

EMC NetWorker Fast Start and the new Cloud Backup option offer a secure and affordable alternative to the pain and expense of offsite tape vaulting. Attend this webcast and learn how you can:

• Reduce personnel expenses by automating offsite data protection
• Cut the hard costs associated with tape infrastructure and offsite storage facilities
• Choose a flexible “pay-as-you-go” model for offsite storage

You can register for the event here, or by going to http://info.emc.com/mk/get/DBM5396-1950_RAF_LP?reg_src=PA .

The Cost of Backup

I am going to revisit the issue of backup, cost, and the total cost of ownership of backup in this post. Tony Pearson and I have been discussing this (hopefully in a fairly constructive fashion) over the last little while, and I see he has responded to some of the points that I made in my previous post on the issue. Tony's response can be found here: "The TCO of TSM for Backup".

Tony continues to maintain (I think) that TSM has a "good" TCO and is efficient relative to other backup applications. Although I don't agree with a lot of what he writes, I do think that there are some pretty good lessons to be learned amidst the information he has posted, so I am going to continue to pursue this a bit in this post.

First, Tony claims that I said "only the cost of new TSM servers should be considered in any comparison." Ummm, jeepers Tony, no I didn't! I would never make any such claim! (What I did say is that TSM is unique in the backup world, in that it scales by adding master servers. Once my TSM master server is at capacity, due to I/O requirements, or database scale, or ability to drive disk or tape, I have to buy another master. That stands in sharp contrast to Networker or Netbackup, where I can just buy a Storage Node or Media Server--which tend to be much smaller systems with much less disk than the master servers. So TSM requires many more master servers than competitive solutions--each of which imposes not just a burden of cost, but a management burden at both the server level and the backup application level. Further, based on my experience, the typical TSM customer will end up with more master servers than they would have combined master and media servers if they were using Netbackup or Networker.)

Servers are but one of many cost considerations when it comes to understanding the TCO of a backup environment. Generally speaking, these costs can be divided into hard and soft costs. Hard costs are costs associated with physical, tangible things (servers, network, disk, tape drives, shipping, etc.). Soft costs are those costs associated with stuff that is less tangible. Soft costs can include labor and administration--though truthfully admin costs straddle the line between hard and soft costs, at least in part because they can be difficult to accurately quantify--but also include things like risk, efficiency, and the benefit of systems uptime.

Deduplication Deployments

I have had the chance to talk to a huge number of customers across the country lately. I love to do this stuff, because: a) I actually really like public speaking, and b) it is a really good chance to level set and get feedback on whether the message is actually speaking to customers' needs, or not.

One of the most startling things that I have found is just how low the percentage of organizations that have installed any sort of data deduplication for their data protection environment. At a guess, based on a polling of maybe 400+ individuals in total, less than 10% of people that raised their hand indicated that they had any sort of data deduplication installed.

A couple of qualifiers here: the mix in the crowd was by no means "EMC Customers" only. Obviously a good number were, but at least half, if not more, were just beginning to have a relationship with EMC. Further, there was also a good mix of both enterprise and SMB customers, and a mix of public and private. In short, I am no statistician, but this seemed like a pretty representative sample to me.

So what did I learn from this? I think we, as an industry, need to continue to be clear and concise on the exact value proposition and message for deduplication. And that is: It saves money. It saves bandwidth (for replication). It is secure and reliable (unlike tape). It reduces data centre footprint, compared to tape. It reduces restore times. It can reduce bandwidth requirements for backup. And it can reduce backup times very significantly. In particular, I think we need to continue to explain that deduplication is less expensive, and therefore has a positive ROI, than traditional backup.

The other obvious lesson is that there are still vast opportunities here. For both vendors and customers. For vendors because as many as 90% of our customers have yet to do anything with respect to deduplication for data protection. For customers, most of you still have a huge opportunity to transform the cost and capability of your data protection infrastructure.

All of which makes for interesting and challenging times for us all.

Copies and Backups Revisited

Mark (aka Storagezilla) Twomey wrote in his blog last week about copies and backups, and concluded that point in time copies are backups.

Hrmm. As I have said before, I am not sure that I agree with this.

I have discussed the issue here with W. Curtis Preston too, and I think there is a disconnect happening somewhere. I am fundamentally uneasy with the idea of calling a copy a backup. It seems to me that a copy is a necessary part of a backup, but not sufficient.

Would a tar ball be a backup? A gzip?

Again, I think the answer is, at best, "sort of".

And why do I differentiate between a copy and the tar ball?

Because I think there is not only a difference between copies and backups, but I think there is a difference between a backup and a backup and recovery system.

So lets start with fundamentals.

TSM and the Elephant

OK, fair notice here, but much of what follows is only tangentially related to backup, at best. But there is a point, I think, to my rambling.

First, let me start by reacting to Tony Pearson's continued defence of TSM as a backup application that offers economies of scale. And before I really get going, let me emphasize again the definition of economies of scale: economies of scale offer a declining marginal cost per unit. It is what makes many industries go around: automotive (to the degree that it is actually going around these days!) and chip manufacturing to name just two. It is what makes Intel so successful: the 1,000,000 chip of a given architecture costs pennies (although the first chip might have cost them $2bn or so!).

I have a belief that is somebody can figure out how to achieve real economies of scale in backup and recovery, they are going to achieve two things: they are going to get rich, and they are going to get significant (dominant?) market share. Although I think that EMC has gone further than anybody else in achieving economies of scale with backup, I think there is still a great deal more that we can do. We as an industry, or we as EMC, however you wish to read that.

TSM Scalability (and the pot of gold at the end of the rainbow)

A couple of posts ago I wrote about the need for economies of scale in backup, and lamented (or ranted, to put it bluntly) about the lack of such economies in currently available backup solutions.

Before I go further, let me be clear: the requirement, in my opinion, is for solutions that offer genuine economies of scale. That is, declining marginal costs. This is very different than reducing costs. I think we have done OK as an industry at reducing costs.

If you look at the cost of backup with just tape, and compare that to the cost of backup with source or target deduplication, I don't think there is any doubt at all that you can reduce the cost of backup by employing deduplication--either to augment or complement tape, or to move to a tapeless environment. Your ability to reduce costs is even better if you archive too--every MB that you archive is a MB you don't have to back up.

So, having said all that, I found it interesting that Tony Pearson over at IBM took my post not as an opportunity to discuss backup, recovery and archiving solutions and cost reduction, but to claim that "IBM has Scalable Backup Solutions."

OK. Great.

(But so what? And way to miss the point entirely!)

Something Doesn't Add Up (And Never Will)

When considering what to do to upgrade your backup system, I have always maintained that one of the most important questions you should be asking is: what is wrong with what I got? (Proving nothing more than that the English language will be the first victim of this post.)

What is wrong can have many answers. It is not reliable. It breaks. It is slow. It costs too much. And so on. But one of the very frequent answers is this: my backup takes too long.

As I have said many times before, there are a lot of things that can cause a backup to bottleneck. And there are a lot of things that you can make infinitely fast and your backup will still be no faster overall (yes, LTO3 and LTO4, I am talking about you.)

And until we understand the big picture, and which components in that big picture are contributing to a slow backup, it is a little irresponsible for any of us on the vendor side of things to come out and claim that disk will "fix" your backup and make it faster. It might. Or it might not.

Backup Sucks: Reason #38

"Any customer can have a car painted any color that he wants so long as it is black."

And that was the type of thinking that let Henry Ford achieve enormous economies of scale and sell a lot of cars.

It is also the sort of thinking that will drive the adoption of cloud infrastructure, software, and services. How do you achieve economies of scale? How do you lower the cost of a unit of service so that the next one costs you less to provide than the previous one?

Unfortunately, we have not been able to successfully leverage economies of scale in the world of backup and recovery. If it costs you $5 to backup a given amount of data, it probably costs you $50 to back up 10 times that amount of data, and $500 to back up 100 times that amount of data.

And this is true, more or less irrespective of what backup application, technology, or infrastructure that you use.

Which is not good. Which is the 38th reason that backup sucks.

How Much Is Too Much?

You can never be too thin or too rich. (Although I don't think whoever came up with that had Nicole Richie or the Sultan of Brunei in mind.)

And in the world of backup, you can never be too fast. It is just not possible.

Rich Colbert and Daniel Budiansky have posts regarding the importance of speed to backup. You can read those here and here.

The premise of their approach seems to me two-fold: one, the DD880 is fast enough that it obsoletes post-process deduplication technologies; and two, that this speed is high enough that for the first time in-line deduplication will no longer be a bottleneck in the backup process for the vast majority of customers.

Now I don't entirely disagree with these arguments. For a long time I have been saying of a DL4206 and DL4406 when asked of performance: "They are so fast that I can virtually guarantee that they will not be the bottleneck in your backup process." At up to 2,200 MB/s, we can see that there are many opportunities for backups to degrade before the EDL becomes the problem: data must be read from disk, through a client, across a network, sometimes mediated by an database or application interface, to a backup server, meta-data must be processed, and then it must traverse another network before it is written to a target. There are so many obstacles here (including the presence of an OS, a filesystem, different disk architectures, etc.) that there is very little chance that the EDL will be a bottleneck. Certainly not for a single backup server or storage node or media server.