The Backup Blog

Backup Cost Reduction, Target Deduplication, and TSM

Two posts previous I was discussing the TCO of backup, the differences between hard and soft costs, and the qualities of a good business case. The general conclusion? Most business cases depend on hard costs (and don't recognize soft costs). If hard costs show an ROI of less than 18 months, most CIO/CFO types will love it. If the business case relies on soft costs to get there, it will probably be rejected.

Looking at the particular example from the previous post, the business case Tony Pearson used as an example showed that 81% of the savings were due to soft costs. This is really bad. In most business environments, such a business case would be rejected out of hand.

I thought it might be instructive to have a look at a better business case: a business case based almost entirely on hard cost (savings). Two of them actually. One for source deduplication, one for target deduplication. In both cases, they are real environments, with real costs, real savings, the proposals were made with the costs and savings exactly as described, and they were accepted by the customer and the projects were funded on that basis. I have only removed names and specifics to the extent necessary to anonymize the customer and any technical or business characteristics unique to that customer. And, as I wrote the TCO tool itself EMC uses for target deduplication, I am more than happy to answer any questions on the data, the assumptions that drive it, or the calculations and formulas that we used to determine the benefits and cost savings of the solution.

In this post, I am going to deal with target deduplication. The business case for a source deduplication solution will be the subject of a future post.

The Cost of Backup

I am going to revisit the issue of backup, cost, and the total cost of ownership of backup in this post. Tony Pearson and I have been discussing this (hopefully in a fairly constructive fashion) over the last little while, and I see he has responded to some of the points that I made in my previous post on the issue. Tony's response can be found here: "The TCO of TSM for Backup".

Tony continues to maintain (I think) that TSM has a "good" TCO and is efficient relative to other backup applications. Although I don't agree with a lot of what he writes, I do think that there are some pretty good lessons to be learned amidst the information he has posted, so I am going to continue to pursue this a bit in this post.

First, Tony claims that I said "only the cost of new TSM servers should be considered in any comparison." Ummm, jeepers Tony, no I didn't! I would never make any such claim! (What I did say is that TSM is unique in the backup world, in that it scales by adding master servers. Once my TSM master server is at capacity, due to I/O requirements, or database scale, or ability to drive disk or tape, I have to buy another master. That stands in sharp contrast to Networker or Netbackup, where I can just buy a Storage Node or Media Server--which tend to be much smaller systems with much less disk than the master servers. So TSM requires many more master servers than competitive solutions--each of which imposes not just a burden of cost, but a management burden at both the server level and the backup application level. Further, based on my experience, the typical TSM customer will end up with more master servers than they would have combined master and media servers if they were using Netbackup or Networker.)

Servers are but one of many cost considerations when it comes to understanding the TCO of a backup environment. Generally speaking, these costs can be divided into hard and soft costs. Hard costs are costs associated with physical, tangible things (servers, network, disk, tape drives, shipping, etc.). Soft costs are those costs associated with stuff that is less tangible. Soft costs can include labor and administration--though truthfully admin costs straddle the line between hard and soft costs, at least in part because they can be difficult to accurately quantify--but also include things like risk, efficiency, and the benefit of systems uptime.

TSM and the Elephant

OK, fair notice here, but much of what follows is only tangentially related to backup, at best. But there is a point, I think, to my rambling.

First, let me start by reacting to Tony Pearson's continued defence of TSM as a backup application that offers economies of scale. And before I really get going, let me emphasize again the definition of economies of scale: economies of scale offer a declining marginal cost per unit. It is what makes many industries go around: automotive (to the degree that it is actually going around these days!) and chip manufacturing to name just two. It is what makes Intel so successful: the 1,000,000 chip of a given architecture costs pennies (although the first chip might have cost them $2bn or so!).

I have a belief that is somebody can figure out how to achieve real economies of scale in backup and recovery, they are going to achieve two things: they are going to get rich, and they are going to get significant (dominant?) market share. Although I think that EMC has gone further than anybody else in achieving economies of scale with backup, I think there is still a great deal more that we can do. We as an industry, or we as EMC, however you wish to read that.

TSM Scalability (and the pot of gold at the end of the rainbow)

A couple of posts ago I wrote about the need for economies of scale in backup, and lamented (or ranted, to put it bluntly) about the lack of such economies in currently available backup solutions.

Before I go further, let me be clear: the requirement, in my opinion, is for solutions that offer genuine economies of scale. That is, declining marginal costs. This is very different than reducing costs. I think we have done OK as an industry at reducing costs.

If you look at the cost of backup with just tape, and compare that to the cost of backup with source or target deduplication, I don't think there is any doubt at all that you can reduce the cost of backup by employing deduplication--either to augment or complement tape, or to move to a tapeless environment. Your ability to reduce costs is even better if you archive too--every MB that you archive is a MB you don't have to back up.

So, having said all that, I found it interesting that Tony Pearson over at IBM took my post not as an opportunity to discuss backup, recovery and archiving solutions and cost reduction, but to claim that "IBM has Scalable Backup Solutions."

OK. Great.

(But so what? And way to miss the point entirely!)

TSM, Virtual Tape, Deduplication and Reclamation

Well that is kind of a mouthful.

Quick technical tip for TSM users. When TSM is finished with a tape volume (virtual or physical) the default behaviour is to not relabel it, but instead to reuse it with the existing label (volume ID).

This happens to be not a very good thing if you are using a virtual tape library with deduplication. Why? Because virtual tape libraries don't reclaim space--at the block or segment level that is no longer referenced by valid backup data--on a volume that is still in use. Even if "in use" in this case means sitting in the (virtual) scratch pool doing nothing.

So, the good folks over at IBM figured this out, and have a way to force TSM relabel a volume when it is returned to the scratch pool. In terms, the VTL now sees that the data referenced by this volume is no longer valid, and can be freed up for reclamation, which will free up physical space on your deduplicated VTL. Good stuff.

In order to take advantage of this, you need to set "relabelscratch=yes" in your library definition. You can find additional detail here: http://www-01.ibm.com/support/docview.wss?uid=swg21330242. Note that if you are at TSM 5.5.1.0 you will need to apply a patch for this to work, if you are at 5.5.2.x and higher, it should work without further patching.

Data Protection Advisor

Note: 2nd of 3 updates on new product announcements for this week. The 3rd installment will cover what is new with NetWorker 7.5.

I thought I would take the opportunity in the second of my product update discussion to discuss one of the lesser know, but most relevant and useful products in the lineup: Data Protection Advisor (formerly known as EMC Backup Advisor, and prior to that as WysDM). Data Protection Advisor, or DPA, fills a big gap for most backup and recovery administrators: the reporting, diagnostics, and analysis of their environment in a useful way.

Notice that I say useful: having 400 canned "reports" does nobody any good if the reports are not useful and relevant.

Deduplication and Replication Revisited

After my post two weeks ago, I wanted to come back to this topic and revisit some of the issues, largely thanks to a couple of very insightful comments from Curtis Preston. He raised some good suggestions with respect to replicating deduplicated data sets.

A brief recap of my prior post is this: how do you get the backup application to recognize the deduplicated data set at both the source and the target in such a way that you can use the data at both sites (for recoveries, data portability, data refreshes, etc.)? It isn't easy! I pointed out a couple of work-arounds, and Curtis added two. Because I think they are important, I am going to quote Curtis wholesale here.

Deduplication and Replication

Some time ago I wrote a post Peace, Love and Revolution about the issues that replicating deduplication appliances causes with backup applications. Some six months later, I thought it would be good to update the state of the nation and see what progress has been made.

To quickly refresh, the issue is this: how do you restore data from a deduplication appliance that is the target for replication--when it is the appliances that is managing the replication process and not the backup application. (And to be fair, this is not an issue unique to deduplication: the exact same problems apply if you are just using virtual tape. The issue just happens to be more pressing with deduplication because if it is "just" virtual tape, I can use the backup application to manage replication and not cost myself any bandwidth; with deduplication, if I use the backup application I miss out on the 95% bandwidth savings that deduplication replication promises.)

Tape Drives and TSM

A while back, after reading this post, a reader asked: so how do other backup applications compare to TSM? If TSM runs tape drives at 80% to 90% utilization on average, how busy are tape drives with other applications?

Good question.

Not only is it a good question, it is also one of the primary differences in practice between TSM and everybody else. (And I say everybody else because, for arguments sake, "everybody" else does a traditional grandfather, father, son backup rotation which involves full backups on weekends, and incremental backups throughout the week. Except, as I never tire of pointing out, for databases and email which are probably backed up full everyday by most people anyway. And yes there are a couple other backup applications which do a progressive incremental like TSM, but their market share is so tiny as to render them insignificant.)

I Want A Red Ferrari

Unfortunately, wanting something, and having reality conform to my desires, are two totally different things. So it should come as no surprise that I do not, in fact, have a red Ferrari. Or a Ferrari of any color. Nor is it likely to be the case that I will have one in the future.

And I am OK with that.

When Tony Pearson talks about virtual tape and TSM in his latest post, I get the impression that not only does he want a red Ferrari, he thinks that desire alone is sufficient to ensure that one appears in his driveway.