Posts - Integrating the
Model in the Real

1/13/09: Audit & Risk - Seeing the Forest from the Trees

9/4/08: Security ROI

6/28/08: Boise: A
Terrorist Target?

5/10/01: FFIEC Business
Continuity Planning

4/3/08: SOX 404 Audits

Security ROI

9/4/2008 - The search for a reliable method for determining a security ROI (the return on an investment in a security product or process) has been the Holly Grail for decades. While security and technology managers within the company identify and develop proposals for implementing new security controls, the senior executives on the business side are the ones who have to be convinced to spend the money to implement those controls. Since most decisions on business investments rely heavily on the ROI or return on investment of the product or service (i.e. if the company spends x dollars on developing a product this year, how much will the product return in income over the next five years?), security professionals usually try to justify their requests to senior management for approval to purchase new controls by demonstrating how the control will either generate additional income or reduce future expenses. Typically the security ROI argument boils down to an analysis of the cost to purchase and implement a control (for example, the installation of a network intrusion monitoring tool) against an estimate of the future losses that would be incurred by the company due to undetected harmful activity on the network. Estimating future losses is, at best, an art form (see Bruce Schneier's Blog for an excellent discussion of the inherent problems with Security ROI) for at least two reasons:

  1. There is little or no historical data. If you apply for auto insurance or a credit card, the provider can refer to extensive amounts of historical data to determine the risk in providing you with the product. Nothing close to that amount of data exists in the operational risk area. This leads most practitioners to rely on personal experience and "gut feel" in estimating future losses.
  2. The playing field is constantly changing. If you look at how the tools thieves use to steal cars have evolved over the past ten years, you would likely see a limited number of changes. Compare that over the same period to how the criminals have adapted to changes in technology in order to remotely penetrate corporate and home computer systems. In the 1990's the primary threats were script kiddies blindly running malware and insecure hackers trying to make a name for themselves. If your web site was hacked, it typically was either taken off-line or the hacker posted their own front page to demonstrate their feat. Today organized crime has taken over and their tools change almost daily. Even if you can make a good guess as to future losses for an ROI estimate based on what you know of today's technology, tomorrow will likely prove you wrong.

Schneier provides a good example of the ROI rationalization process:

"The classic methodology is called annualized loss expectancy (ALE), and it's straightforward. Calculate the cost of a security incident in both tangibles like time and money, and intangibles like reputation and competitive advantage. Multiply that by the chance the incident will occur in a year. That tells you how much you should spend to mitigate the risk. So, for example, if your store has a 10 percent chance of getting robbed and the cost of being robbed is $10,000, then you should spend $1,000 a year on security. Spend more than that, and you're wasting money. Spend less than that, and you're also wasting money....

If you're doing an ALE analysis of a security camera at a convenience store, you need to know the crime rate in the store's neighborhood and maybe have some idea of how much cameras improve the odds of convincing criminals to rob another store instead. You need to know how much a robbery costs: in merchandise, in time and annoyance, in lost sales due to spooked patrons, in employee morale. You need to know how much not having the cameras costs in terms of employee morale; maybe you're having trouble hiring salespeople to work the night shift. With all that data, you can figure out if the cost of the camera is cheaper than the loss of revenue if you close the store at night -- assuming that the closed store won't get robbed as well. And then you can decide whether to install one."

Schneier goes on to explain that if determining the ALE for a security camera is difficult, accurately estimating the ALE for a technically sophisticated piece of equipment like a network intrusion monitor is nearly impossible. For all these reasons security ROI estimates are highly suspect.

However, security ROI does have some limited value, especially if you use a tool such as the Simple Risk Model. There are those situations where technology managers fall victim to salespersons and believe that their new piece of ground breaking technology will solve all their problems. It is like the store manager in Schneier's example, just focusing on the security camera. This is a tunnel vision ROI analysis and the results will always be suspect. The better approach is to first focus on the problem (i.e. how do I reduce losses from robbery?) and then assess the ROI of the various options. Coming up with an accurate ROI for a security camera is doubtful. However, if you apply the same methodology, such as the Simple Risk Model, against all of the options, you may not get a 100% accurate prediction of ROI for the individual components, but at least you will have a consistent method of comparing the values of the various options.

For example, the Simple Risk Model centers around the assessment of risk for the core processes in the business, which minimizes the chance that management would begin the assessment process by focusing on individual applications or systems as is typically the case. Case in point - you work for a large corporation that processes the weekly payroll checks on an internally developed payroll system. A vendor may approach management with an encryption solution to protect the payroll records. Developing an ROI for the encryption solution is not going to generate a meaningful result. The real issue, and the one asked under the Simple Risk Model, is what would be the Cost or overall ALE to the company is it were to suffer a breach of confidentiality on the payroll system? Then the vulnerabilties and threats relating to various existing and proposed controls for the system (including encryption) would then be factored in to determine which were the most effective in reducing the risk and at what expense to the company.