1. June 2, 2005Slide 1 Analyzing the Risks of Information Security Investments with Monte- Carlo Simulations WEIS 0 5 Workshop on the Economics of Information Security James R. Conrad, University of Idaho Department of Computer Science conr2286@uidaho.edu

2. June 2, 2005Slide 2 Contents Introduction to the Problem The Monte-Carlo Solution Overview of Monte-Carlo Simulations Example Analysis and Critique Conclusions

3. June 2, 2005Slide 3 Introduction to the Problem An information security investment may need to compete for resources with other business opportunities But many information security models rely upon experts’ estimates And the experts’ estimates may include significant uncertainty How can the analyst communicate an opportunity when so much is uncertain?

4. June 2, 2005Slide 4 Monte-Carlo Solution Apply the Monte-Carlo technique to simulate and express uncertainty in information security models This is not a new model --- this is an enhancement of existing models While less common in the Computer Science discipline, many financial decision makers are already familiar with the Monte-Carlo approach

5. June 2, 2005Slide 5 Monte-Carlo Simulations Specify uncertainty in probability distributions Monte-Carlo engine samples distributions Engine executes the security model once for each of several thousand iterations Monte-Carlo engine captures and collects the result of each iteration distributions enginemodel results

6. June 2, 2005Slide 6 Monte-Carlo Simulations Engine simulates uncertainty in the model parameters Model continues to operate with discrete values Extra complexity largely confined to the Monte-Carlo engine Results can be charted as probability distributions distributions enginemodel results

7. June 2, 2005Slide 7 Monte-Carlo Example Based upon Longstaff et al’s example appearing in “Are we Forgetting the Risks of Information Technology?” of IEEE Computer, December 2000 Simulates the benefit/cost ratio of a proposed infosec investment for a financial enterprise Modeling parameters are similar to Longstaff’s example with an added complication… …The experts don’t agree!

8. June 2, 2005Slide 8 Original (pre-Monte-Carlo) Parameters & Model Intrusion Rate Parameters r12Simulated annual intrusion count w/o investment e5.00E-01Effectiveness of investment r2=r1*eAnnual intrusion count with investment Other Parameters p1=r1/365Daily probability of intrusion w/o investment p2=r2/365Daily probability of intrusion with investment X$20,000,000,000,000 Asset value y1$100,000,000 Cost of software assurance w/o investment y2$200,000,000 Cost of software assurance with investment z11.00%Losses w/o investment z20.50%Losses with investment Model Calculations d1=p1*z1Calc damage w/o investment d2=p2*z2Calc damage with investment D=y2-y1 Calc cost to provide software assurance with investment d=d1-d2Calc percentage of losses prevented by investment b=d*X-DCalc net benefit of investment bcr=b/D Calc benefit/cost ratio for investment (bcr=7.22) intrusion rates other parameters model benefit/cost ratio, bcr

9. June 2, 2005Slide 9 Uncertainty in the Revised Example Consider a case in which the experts don’t agree upon an single value estimate for the annual intrusion rate (fixed at r1=2 events/year in the original problem) The hypothetical disagreement stems from uncertainty in anticipated business practices Experts do agree there exists a 20% chance that business practices will change in a way that will raise the intrusion rate to 20 events/year and an 80% chance that those practices will remain unchanged

10. June 2, 2005Slide 10 Uncertainty in the Revised Parameters Model variability of optimistic intrusion rate as a Poisson process (for purposes of this example), ro=randpoisson(2) Model variability of pessimistic intrusion rate as a Poisson process, rp=randpoisson(20) Model uncertainty of anticipated business conditions by choosing the optimistic rate 80% of the time and the pessimistic rate 20% of the time using randdiscrete(0.80,0.20,ro,rp) Variability refers to a truly random process Uncertainty refers to the experts’ inability to anticipate future business conditions

11. June 2, 2005Slide 11 Revised Params & Model Intrusion Rate Parameters ro=randpoisson(2)Optimistic annual intrusion count w/o investment rp=randpoisson(20)Pessimistic annual intrusion count w/o investment r1=randdiscrete(0.8,0.2,ro,rp)80% Chance of ro. 20% Chance of rp. e5.00E-01Effectiveness of investment r2=r1*eAnnual intrusion count with investment Other Parameters p1=r1/365Daily probability of intrusion w/o investment p2=r2/365Daily probability of intrusion with investment X$20,000,000,000,000 Asset value y1$100,000,000 Cost of software assurance w/o investment y2$200,000,000 Cost of software assurance with investment z11.00%Losses w/o investment z20.50%Losses with investment Model Calculations d1=p1*z1Calc damage w/o investment d2=p2*z2Calc damage with investment D=y2-y1 Calc cost to provide software assurance with investment d=d1-d2Calc percentage of losses prevented by investment b=d*X-DCalc net benefit of investment bcr=b/D Calc benefit/cost ratio for investment intrusion rates other parameters model benefit/cost ratio, bcr

12. June 2, 2005Slide 12 Simulation of Revised Example randpoisson() and randdiscrete() sample the probability distributions in each iteration of the simulation The Monte-Carlo engine recalculates the model for each iteration and captures the results (bcr) The Monte-Carlo engine charts the captured simulation results (next slide)

13. June 2, 2005Slide 13 Simulation Results

14. June 2, 2005Slide 14 Why not use a weighted average of r1 and r2? Why doesn’t the revised model simply compute a weighted average of the two possible intrusion rates? r1 = randpoisson(2)*0.8+randpoisson(20)*0.2 The randdiscrete() simulation preserves the bimodal nature of the experts’ disagreement. Any attempt to “average away” that uncertainty conceals the truth: The experts don’t agree.

15. June 2, 2005Slide 15 Analysis The results reflect the experts’ strong preference for the optimistic intrusion rate in which the benefit/cost ratio remains unchanged at 7.22. Risk-tolerant decision makers might manage to this value. The mean value lies at 22 between the two modes. The results also reflect a second mode at about 81 along with a 10% chance of the benefit/cost ratio exceeding 81. Risk-adverse decision makers might manage to this value to avoid a catastrophe “on their watch.”

16. June 2, 2005Slide 16 Critique But are real experts willing to provide even more estimates? The author’s industry experience with Monte-Carlo models is that many experts are relieved to disclose the uncertainty they know to be in their estimates What real experts truly dislike is being held accountable to an expected value they know is merely representative of the possibilities

17. June 2, 2005Slide 17 Additional Critique Given a tool to express uncertainty as probability distributions, which distributions closely model the empirical evidence? How to extend the Monte-Carlo approach to graphical models?

18. June 2, 2005Slide 18 Conclusions Monte-Carlo techniques offer an approach to simulate uncertainty in expert estimates Enables the use of probability distributions for model parameters and forecast results The Monte-Carlo engine simulates random variables, allowing a security model to continue to manipulate discrete values with only minimal changes May be particularly useful for visualizing the potential of an extreme event, the unlikely possibility of a catastrophic outcome

19. June 2, 2005Slide 19 Questions and Optional Slides

20. June 2, 2005Slide 20 Why Poisson Distribution? The example problem uses a Poisson process to approximate intrusion attempts If and/or when the Poisson process usefully reflects empirical intrusion attempts is an open question Review: Models the number of events occurring during a specified time interval for a Poisson process Review: Continuous opportunity for independent events to occur Review: Long-term rate is constant Review: Used to model lightening strikes in a storm

21. June 2, 2005Slide 21 Correlated Parameters “Every iteration of a… model must be a scenario that could physically occur.” -- Vose. The parameters must “make sense” to the security model! One correlated parameter can usually be expressed as a function (relation) of another. Consider r1 and r2 in the example. These are likely related which is why r2 is calculated as a function of r1. If the relationship (e) between r1 and r2 is also uncertain, this too can be simulated.

22. June 2, 2005Slide 22 Variability and Uncertainty Yes, this example lumped (simulated) variability and uncertainty together for simplicity Vose (Risk Analysis, 2000) offers an excellent treatment of this subject for those who need to keep them separated

23. June 2, 2005Slide 23 Partitioning Yes, partitioning is an alternative technique The Monte-Carlo technique might be viewed as an automated approach to partitioning …and the Monte-Carlo technique avoids the subjective choice of partition boundaries …and the Monte-Carlo technique has commercial tool support for systems-level models.

24. June 2, 2005Slide 24 Commercial Tools Yes, commercial off-the-shelf tools are available They are most useful for systems-level security models. They are less useful for low-level combinatorics security models Search for “monte carlo simulation” and pay particular attention to the “Sponsored Links”

25. June 2, 2005Slide 25 Performance The author’s industry experience includes Monte-Carlo simulations using “hundreds” of random distribution parameters Yes, they required several hours to run… In 1997! My computer is more than 10X faster today. Simulation multiplies model complexity by n, the number of iterations. A simulation of an O(m 2 ) model becomes n*O(m 2 ). Opportunities for parallel approaches when n cannot be ignored.