Copyright Hubbard Decision Research Introduction to Applied Information Economics Catastrophe Avoidance & Decision Optimization by Learning How to Measure Anything

Copyright Hubbard Decision Research Presentation Objectives This is a high-level review of how Applied Information Economics (AIE) is used to measure risk and IT value Overview of the problem and the AIE solution How to measure anything Understanding Risk and Uncertainty Putting the pieces together into AIE Findings from past cases Optional advanced methods

Copyright Hubbard Decision Research The Biggest Decisions What are your biggest, riskiest, decisions? –High uncertainty; chance of being wrong –High cost of error How would you characterize the level of sophistication of analysis for those decisions? –Are measurements empirical, subjective “scores”, or something else? –Does the risk analysis look like something an actuary or statistician might actually conduct? Where does your organization focus sophisticated quantitative analysis (if any)?

Copyright Hubbard Decision Research IT is Risky 0% 10% 20% 30% 40% 50% ,00010,000 Projected Work-Months of Effort US Average Cancellation Rate Source: Derived from data provided by Capers Jones, Applied Software Measurement, McGraw Hill, 1991 Other risks include: Cost Overruns Uncertain Benefits Interference w/Business Even as IT portfolios grow, IT remains one of the riskiest investments a business can make Benchmark: The worst-rated junk bonds default at a rate of 30% to 40%

Copyright Hubbard Decision Research Most quantitative risk analysis is applied to routine operational decisions, not major strategic uncertainties The riskiest decisions (like many large IT projects) get little or no quantitative risk analysis The Risk Paradox

Copyright Hubbard Decision Research The Consequences The risks and uncertainties that really matter (those on the biggest dilemmas) are un-quantified, probably unnoticed, and certainly not mitigated with planning There are probably just a few key measurements that are relevant to a given issue – and it is unlikely they are even identified, much less tracked Failure becomes more likely for the BIG decisions than small decisions

Copyright Hubbard Decision Research “Paradigm Shifts” for IT There are no intangibles - If something is real, it is measurable If something has value (like information), it has a computable economic value Regardless of the level of uncertainty, quantitative methods can be used to make rational decisions Advanced methods are usable by decision makers Even very “politicized” decision making processes can benefit

Copyright Hubbard Decision Research Real Solutions to… …the economics of information …the economics of IT infrastructure …the economics of risk …the economics of labor reduction when headcount is not reduced …the economics of constantly changing technology (avoiding “technology regret”)

Copyright Hubbard Decision Research Modern Portfolio Theory AppliedInformationEconomics Economics Decision/Game Theory Statistics Information Theory OptionsTheory Operations Research Applied Information Economics Applied Information Economics (AIE) is the practical application of scientific and mathematical methods to quantify the value of IT-enabled business investments “Quantifying the risk and comparing its risk/return with other investments sets AIE apart from other methodologies. It can substantially assist in financially justifying a project -- especially projects that promise significant intangible benefits.” The Gartner Group “AIE represents a rigorous, quantitative approach to improving IT investment decision making…..this investment will return multiples by enabling much better decision making. Giga recommends that IT executives learn more about AIE and begin to adopt its tools and methodologies, especially for large IT projects.” Giga Information Group

Copyright Hubbard Decision Research Clients Include: The Axa Group – 6 of the major companies State of North Carolina American Express The Discovery Channel The Banking Industry Technology Secretariat Blue Cross Blue Shield of Illinois BankFirst U.S. Federal Government: –Department of Treasury –Bureau of The Census –Department of Veterans Affairs –General Services Administration –Housing and Urban Development –Environmental Protection Agency –Office of Naval Research

Copyright Hubbard Decision Research Example Measurements Risk of IT The Risk of obsolescence The value of a human life The value of saving an endangered species The value of public health The value of IQ points lost by children exposed to Methyl-Mercury The value of better security Forecasting fuel in battlefield environments The future demand for space tourism The value of better information

Copyright Hubbard Decision Research Example Benefits of AIE OrganizationProblemFindingsBenefit EPA Assess value of “Safe Drinking Water Information System” SDWIS Identified and measured previously unnoticed risks & benefits $15 million improved NPV from reprioritizing functions Optimize desktop upgrade schedule for 20,000+ EPA staff A previously unconsidered option was generated better balanced costs and function $12.8 million improved NPV from minimizing upgrade costs and support costs US Marine Corp Forecast fuel use for the battle field Found factors that better correlated to actual fuel use $50 million savings per year by reducing unnecessary battlefield inventory American Express Analyze ROI of a dedicated test network for new applications Determined key metric to prioritize replacements was downtime reduction $2 million improved net benefit reprioritizing rollout The Discovery Channel Determine if new TV show production systems are needed Found ways to measure previously unmeasured benefits and risks $1 million improved net benefit by changing functionality of proposed systems

Copyright Hubbard Decision Research How to Measure Anything (The Measurement.com approach) Gilb’s Law “Anything can be measured in a way which is superior to not measuring it at all” The perceived impossibility of measurement is an illusion caused by not understanding: –the Concept of measurement –the Object of measurement –the Methods of measurement See my “Everything is Measurable” article in CIO Magazine (got to “articles” link on

Copyright Hubbard Decision Research The Concept Of Measurement Sometimes one believes that a thing is “immeasurable” only because they do not actually understand the concept of measurement The “Measurement Theory” definition of measurement: “A measurement is an observation that results in information (reduction of uncertainty) about a quantity.” Any “reduction of uncertainty” about a quantity can be of value ?

Copyright Hubbard Decision Research Normal Distribution Uniform Distribution Lognormal Distribution Threshold confidence 15%85% Ideal Values: Point Real-world Meas. Real-world Measurements vs. Ideal Values

Copyright Hubbard Decision Research The Object of Measurement If a thing seems like an immeasurable “intangible” it may just be ill-defined Often, if we can define what we mean by a certain “intangible” we find ways to measure it ?

Copyright Hubbard Decision Research The Methods of Measurement Even if we understand the object and concept of measurement, we may think something is “immeasurable” because we are unaware of methods of measurement Ask yourself this: Is it really possible that you have a measurement problem that has never been attempted before?

Copyright Hubbard Decision Research Basic Measurement Methods Primary Research: –Controlled Experiments –Random Samples Secondary Research –Industry Publications –The Internet –Database Queries Subjective Assessments of Probabilities How much empirical analysis effort do you apply to investments of over $1 million? Chances are, you under- analyze. IT spends far less on measurement than other equally risky ventures. How much empirical analysis effort do you apply to investments of over $1 million? Chances are, you under- analyze. IT spends far less on measurement than other equally risky ventures.

Copyright Hubbard Decision Research The “Math-less” Statistics Table Measurement is based on observation and most observations are just samples Reducing your uncertainty with random samples is not made intuitive in most statistics texts This table makes computing a 90% confidence interval easy

Copyright Hubbard Decision Research Understanding Risk Uncertainty and Risk are the facts of business decisions in general and IT decisions in particular We have to understand how to make rational decisions even under such conditions

Copyright Hubbard Decision Research Uncertainty, Risk and Reward We define risk the same way as actuaries: The probability of a specific, undesirable event Technically, risk has uncertainty, but not all uncertainty has risk Risk is something that most people are willing to accept for a potential reward Most people are willing to accept more risk for a higher potential reward

Copyright Hubbard Decision Research The Monte Carlo Method Monte Carlo analysis is the random generation of many (usually thousands) of possible scenarios based on the probability distributions of input variables It is an approximation and can be a little time consuming, but it is simpler than some mathematical methods With current PC power, this can be done by virtually any person in a position to make IT investment decisions

Copyright Hubbard Decision Research Distribution-based ROI Administrative Cost Reduction Total Project Cost % Improvement in Customer Retention 5%10%15% 10%20%30% $2 million$4 million$6 million ROI -50%50%100%0%

Copyright Hubbard Decision Research 2006 Analyzing the Distribution 25%50%75%100%125%-25%0% Risk of Negative ROI “Expected” ROI ROI = 0% Probability of Positive ROI Return on Investment (ROI) The “cancellation hump”

Copyright Hubbard Decision Research Various Risks & Returns 0%100%200%-100%0%100%200%-100% 0%100%200%-100%0%100%200%-100% High Risk Low Risk Low Expected ReturnHigh Expected Return

Copyright Hubbard Decision Research Return Risk 10% 20% 30% 40% 10%20%30%40%50%60%Probability of a negative ROI Acceptable Risk/Return Boundary Investment Region Quantifying Risk Aversion

Copyright Hubbard Decision Research Risk Articles Some of the articles I’ve written also address risk in different ways Under my “articles” link on at go to: –“Hurdling Risk” CIO Magazine –“Risk vs. Return” InformationWeek –There are several columns called “Expert Analysis” in CIO Magazine where risk and proper measurement are common themes

Copyright Hubbard Decision Research The AIE Process Put all this together, and you almost get the AIE process We need to introduce how to compute the value of information We need to show how to decide how much analysis is needed for a given investment

Copyright Hubbard Decision Research The AIE Process 1.Figure out which decisions need detailed risk/return analysis 2.Identify the decision variables 3.“Calibrate” the estimators 4.Populate the model with calibrated estimates 5.Conduct “value of information analysis” 6.Design and conduct additional measurements 7.Run final optimization of the decision

Copyright Hubbard Decision Research How Much Analysis? No Classification Required Abbreviated Deliverable Accept w/o Further Analysis Reject w/o Further Analysis Proceed with Risk/Return Calculation Expected Investment Size ($) Confidence Index k 100k1M10M100M Not all investments require an in-depth analysis If you can quickly assess the size and risk of an investment you can decide if you need more detailed analysis or can make a decision now

Copyright Hubbard Decision Research The Lens Model One method that is proven to improve on expert judgments: build a “regression model” based on nothing more than the judgments of experts Identify Experts Develop Hierarchy of Leading Indicators Develop Sample Set Of Hypothetical Scenarios Evaluate Expert Judgments with Lens Method The Lens Model Give Experts Calibration Training

Copyright Hubbard Decision Research Lens Model Track Record This chart shows the percentage reduction in error of intuitive estimates compared to bootstrapped estimates In every case, this method equaled or bettered the judgment of experts – AIE is better than average 0%5%10%15%20%25%30%35%40% Cancer patient life-expectancy Life-insurance salesrep performance Graduate students grades Changes in stock prices Mental illness using personality tests Student ratings of teaching effectiveness IQ scores using Rorschach tests Psychology course grades Business failures using financial ratios Reduction in Errors AIE

Copyright Hubbard Decision Research Example Classification ,00010, Expected Investment Size ($000) Confidence Index No Classification Needed Do Abbreviated Risk-Return Analysis: 6. DLSW Router Network Redesign 9. Extended Hours 18. Doc. Access Strategy Do Abbreviated Risk-Return Analysis: 6. DLSW Router Network Redesign 9. Extended Hours 18. Doc. Access Strategy Do Full Risk-Return Analysis: 8. Pearl Indicator and Pearl I/O interface 11. Richardson Data Center Consolidation 15. MVS DB2 Tools Do Full Risk-Return Analysis: 8. Pearl Indicator and Pearl I/O interface 11. Richardson Data Center Consolidation 15. MVS DB2 Tools Reject; Consider Other Options: 1. Data Strategy 2. Enterprise Security Strategy 3. Remote Server Redundancy 12. MQ Series: Base 13. Development Environment 2000 (mf) 14. “Source Control” Source Code Mgmt 16. Enterprise InterNet Reject; Consider Other Options: 1. Data Strategy 2. Enterprise Security Strategy 3. Remote Server Redundancy 12. MQ Series: Base 13. Development Environment 2000 (mf) 14. “Source Control” Source Code Mgmt 16. Enterprise InterNet Success Factor Adjustments: 4. Network OS migration to Novell 5.x 10. Optimize Single Code Base Success Factor Adjustments: 4. Network OS migration to Novell 5.x 10. Optimize Single Code Base Accept without Further Analysis: 5. Lucent switch upgrade 7. Image Server Relocation 17. Enterprise IntraNet to all sites Accept without Further Analysis: 5. Lucent switch upgrade 7. Image Server Relocation 17. Enterprise IntraNet to all sites

Copyright Hubbard Decision Research “Full” RRA Overview -100% 0%100%200%300% 400% ROI Run Monte Carlo Optimize Investment, Make Recommendations Conduct the Value of Information Analysis Average ROI Risk 0% 10% 20% 30% 40% 50% 0%50%100%150% 200% Assess risk/return Put ranges and probabilities into model “Calibrate” workshop participants Model the costs, benefits, “intangibles” Conduct empirical measurements

Copyright Hubbard Decision Research 2006 Calibrated Estimates A small amount of training (3 hours) can significantly improve calibration of estimates Comparisons of actual measures to original calibrated estimates show calibration works in most cases In those areas where estimators are still un-calibrated (usually statistical overconfidence), historical variances are available and more reliable: –Business cycles –Failure rates of IT projects Actual 90% Confidence Interval Perceived 90% Confidence Interval by non-calibrated estimators

Copyright Hubbard Decision Research Benchmarking Your Calibration For the initial calibration test, you have two types of questions: For the questions that ask for a range, provide an upper and lower bound that you are 90% certain contains the correct answer For the true/false questions, circle true or false and then circle the percentage that best represents your confidence in your response

Copyright Hubbard Decision Research % Confidence Interval Confidence Levels Most people are significantly overconfident about their estimates, especially educated professionals

Copyright Hubbard Decision Research Giga Analysts Giga Clients Statistical Error “Ideal” Confidence 30% 40% 50% 60% 70% 80% 90% 100% 50%60%80%90%100% % Assessed Chance Of Being Correct Percent Correct 99 # of Responses The Giga Experiment

Copyright Hubbard Decision Research The Equivalent Bet For 90% Confidence Interval questions, which would you rather have? –A: Win $1,000 if your interval contains the correct answer –B: A 90% chance to win $1,000 For the Binary Confidence questions, which would you rather have? –A: Win $1,000 if your answer is correct –B: A chance to win $1,000 equal to your stated confidence? Win 0

Copyright Hubbard Decision Research Other Checks Considering all the ways you can be wrong Thinking of this as a test of trivia knowledge instead of a test of assessing your uncertainty (i.e. are you focusing on being “right” instead of realistically representing your uncertainty?) Hanging on to traditional expectations of “+/- 10%” or similar narrow ranges (i.e. are you resisting wider ranges because you think they are “too wide”?) Actively considering how to adjust given your previous feedback from calibration tests

Copyright Hubbard Decision Research 2006 The Decision Theory Formula: What it means:  Information reduces uncertainty  Reduced uncertainty improves decisions  Improved decisions satisfy business objectives (by definition) The Economic Value of Information

Copyright Hubbard Decision Research Computing EVI “Worst Bound” of the 90% CI; this is the undesirable end of the range “Best Bound” of the 90% CI; this is the desirable end of the range Threshold: Below this point is losing money A B Relative Threshold (RT)=B/A Usually, an Excel macro is recommended, but for some problems, a simpler estimate will suffice As a rule of thumb, the value of information is simply the cost of being wrong times the chance of being wrong I constructed this chart for quick estimates of information value of a range

Copyright Hubbard Decision Research An Information Value Chart Compute the Relative Threshold (RT)* 2. Find the RT on the vertical axis Expected Opportunity Loss Factor (EOLF) EOLF Curves for Normal Distributions EOLF Curves for Uniform Distributions 3. Look directly to the right of the RT value, until you get to the appropriate curve (normal or uniform, depending on the probability distribution you are using). This is the EOLF 4. Compute the Expected Value of Perfect Information (EVPI) = EOLF/1000*units in range*loss per unit

Copyright Hubbard Decision Research Typical Follow-up Measures Creating more detailed models Focus group/interview used to update calibrated estimate Surveys –As small as 5 responses, as many as 100+ –By phone or by Controlled experiments

Copyright Hubbard Decision Research Measuring to the Threshold Measurements have value usually because there is some point where the quantity makes a difference Its often much harder to ask “How much is X” than “Is X enough” Samples Below Threshold 20% 30% 40% 50% 0.1% 1% 10% Number Sampled Chance the Median is Below the Threshold % 5% 0.2% 0.5% 0 1. Find the curve beneath the number of samples taken 3. Follow the curve identified in step 1 until it intersects the vertical dashed line identified in step Identify the dashed line marked by the number of samples that fell below the threshold 4. Find the value on the vertical axis directly left of the point identified in step 3; this value is the chance the median of the population is below the threshold

Copyright Hubbard Decision Research Review -100% 0%100%200%300% 400% ROI Run Monte Carlo Optimize Investment, Make Recommendations Conduct the Value of Information Analysis Average ROI Risk 0% 10% 20% 30% 40% 50% 0%50%100%150% 200% Assess risk/return Put ranges and probabilities into model “Calibrate” workshop participants Model the costs, benefits, “intangibles” Conduct empirical measurements

Copyright Hubbard Decision Research AIE Results for R/R Analysis AIE projects are probably not a random sample of investments, since they tend to be larger and more controversial than average Whether an investment was accepted, rejected, or modified, it can contradict the “default” decision 45% accepted as originally defined, some project mgt, risk mitigation recommendations made 20% Rejected, deferred indefinitely or until other conditions are met 35% Significantly modified, then accepted

Copyright Hubbard Decision Research Using Risk Analysis to Improve Decisions If the Risk is significant (it usually is), consider doing the following: Reduce the size and functionality of the proposed system - focus on fewer high-return features Wait until specific uncertainties in the environment subside - e.g. major mergers, reengineering, etc. Wait to tackle big projects until proper skills are developed and methods are in place Consider purchased packages that aren’t a perfect fit but close enough - they may look more attractive now Invest more on a proper economic analysis of the largest IT investments - this should reduce uncertainty about critical quantities Include deferred benefits in any estimate of scope creep costs

Copyright Hubbard Decision Research AIE vs. Basic Monte Carlo AIE is based on a Monte Carlo analysis of an investment decision – but adds some components Any subjective probabilities and ranges are calibrated probabilities The economic value of additional measurements is computed for each variable in the model using standard decision theory methods Other components of AIE utilize Modern Portfolio Theory, Real Options and other optimization methods

Copyright Hubbard Decision Research Organizational Issues Use separate individuals for estimating and “auditing” These individuals must not report to (or be the same as) the project sponsor The potential for conflicts of interests must be monitored The use of certified individuals and a formal QA process is critical

Copyright Hubbard Decision Research Key Revelations Since 1996, I’ve conduced 55 major risk/return analysis, 6 major portfolio prioritizations and numerous related studies. The biggest findings were: The magnitude and frequency of avoidable errors in traditional business cases The effect of quantifying risk on IT investment priorities Current measurement priorities vs. optimal The true cost of scope creep The predictors of success

Copyright Hubbard Decision Research A Survey of Business Cases HDR has performed a survey of business cases in 20 business units in multiple companies and agencies Over 100 individuals were surveyed and over a 200 business cases were reviewed A look at the business cases themselves revealed some striking and frequent spreadsheet errors that had significant effects on the calculated ROI’s (errors in spreadsheets slightly exceeds error rates in software code, but does not get the same level of auditing) Most (70%) responded that current business cases were either unrealistic or had no effect on the decision Some felt that the decision was already made before they were asked to make the business case Certain quantities are traditionally left out that should be included in a complete probabilistic model: -Cancellation or other “catastrophic events” -Adoption Rate -Productivity Realization Rate -Changes in staff size/volumes/budget -Cost of infrastructure impact -Cost of training & long term maintenance

Copyright Hubbard Decision Research % 100% 1000% 20%40%60%80%100% 20% 30% 50% 200% 300% 500% Size of the Project Relative to the Entire IT Portfolio (i.e. 50% = project makes up half the work in the entire portfolio) Required Minimum Return (IRR over 5 years) Most Risk Averse Approximate Median Most Risk Tolerant Range of Typical “Hurdle Rates” Risk Increases Required ROI’s Adjusting for risk causes some previously-acceptable projects to be rejected Also, some low return but low risk projects would now be acceptable More projects with “intangible” benefits are now economically justified The net result: A completely reshuffled deck of IT project approvals

Copyright Hubbard Decision Research Example of Risk Effects 50% 40% 30% 20% 10% 0% 50%100%150%200% Expected IRR over 5 years Chance of a negative IRR These are real IT investments of $2M-$3M plotted against a client’s investment boundary The 27% ROI investment is actually preferred to the 83% ROI investment Region of Unacceptable Investments Region of Acceptable Investments Articles by Hubbard: “Hurdling Risk” CIO Magazine and “Risk vs. Return” InformationWeek (articles linked at

Copyright Hubbard Decision Research The IT Measurement Inversion Typical Attention Economic Relevance Receives Most Attention Least Relevant to Approval Decisions Receives Least Attention Most Relevant to Approval Decisions Costs –Initial Development Costs –Ongoing Maintenance/Training Costs Benefits –A specific benefit (productivity, sales, etc.) –Utilization (when usage starts and how quickly usage grows) Chance of Cancellation See my article “The IT Measurement Inversion” in CIO Magazine (its also on my website at under the “articles” link)

Copyright Hubbard Decision Research EPA Examples of VIA Count ProjectEstimated Variables Variables with Significant Expected Value of Perfect Information (EVPI) Desktop Refresh483 ERDMS724 Remote Access620 National Infrastructure (Wiring/Networks) 473 SDWIS Modernization992 Thin Client532 CERCLIS 3863

Copyright Hubbard Decision Research “Scope Creep” Cancellation Risk 2% Long Term Support 26% Deferred Benefits 48% Initial Development 24% One reason for scope creep may be that the true cost of adding additional features to software in development is greatly underestimated If costs are computed at all, they usually consider only initial development

Copyright Hubbard Decision Research Best Predictors of Success Success FactorIncrease in Chance of Success Level of Sponsor/Champion meets or exceeds requirements for project scope 5% to 30% Duration less than one budget cycleUp to 22% Technology related: Age of technology, Used by competitors, etc. 2% to 10% Vendor related: Currently used & proven vendors, track record with vendor 3 to 8%

Copyright Hubbard Decision Research Advanced Methods AIE builds on the basic Risk Adjusted Valuation of Investments Much more complicated problems can be addressed with the tools of AIE More value is found beyond accept/reject analysis and beyond prioritization - we need to optimize the IT investments

Copyright Hubbard Decision Research 2006 Anticipating Change A Critical Technology Measure Time Application Feasibility ? Now Even though technology change is inevitable it is often not anticipated “Anticipatory Development” allows maximum capitalization on technology

Copyright Hubbard Decision Research 2006 Combinations Of Features An Application Features Dependent Features 10 independent features = 108 million combinations of “Versions”

Copyright Hubbard Decision Research ? Consolidation Optimization My European client is doing a world-wide data center consolidation Of the 11 data centers in Europe there are over possible consolidations We formulated the problem as a type of “optimal spanning tree” with a dynamic programming algorithm - we narrowed the field down to the few most economically optimal Belgium France Germany Italy England Spain Switzerland Portugal Austria

Copyright Hubbard Decision Research RRA Portfolio Management With risk/return calculation as a foundation, portfolio management methods can be applied for: –Project prioritization –Justifying investments that only have value through their effect on the portfolio (infrastructure, etc.) Techniques will include Modern Portfolio Theory and various optimization formulas

Copyright Hubbard Decision Research Prediction Markets Simulated trading markets are a proven method of generating probabilities for uncertain events Research by HDR and others shows that it works even without purely monetary reward systems Source: Servan-Schreiber, et. al. Electronic Markets, v 14-3, September 2004

Copyright Hubbard Decision Research Cost vs. Value of AIE The cost of RRA of IT investments has ranged from $40k to $100k The cost of analysis routinely comes in below 1% and has always been under 2% of the investment size - including initial training - still less than non-IT investments of similar size and risk It is also sometimes less time-consuming than the previous non- quantitative analysis techniques used by the firm Using the standard VIA calculation for the value of AIE analysis, AIE itself was the best investment of all the IT investments we analyzed - very conservative measures of payoffs put $20 to every $1 spent on AIE –The VA credits AIE with a $30 million savings – a 300:1 payback

Copyright Hubbard Decision Research In Conclusion... AIE is about getting the most out of the IT investment Decisions would be very different - and much better - if intangibles, risk, redefined roles, and optimization methods were used AIE is theoretically well-founded and has a measurable benefit The problems AIE solves will only become more prevalent

Copyright Hubbard Decision Research Questions? Doug Hubbard Hubbard Decision Research (630)