1. Quantitative Methods Make A Difference

2. Overview
Quantitative methods (probabilistic analysis, operations research, etc.) are widely used in other industries, but mostly lacking in IT investment analysis
Over the past 7 years, we have been focusing specifically on the application of more advanced quantitative methods to IT
This presentation will review the key findings from the application of quantitative methods to over 30 projects

3. Quantitative Methods Include:
Computing uncertainties and risks
Computing the economic value of information
Measurement methods for many items usually considered intangible
Optimizing solutions when there are huge combinations of options for:
Roll-out priorities of systems
Selection of a portfolio of functions
Any other problem where different alternatives about different aspects of the investment generate lots of possibilities

4. Empirical Decision Theory
Method: AIE
Applied Information Economics is the practical application of scientific and mathematical methods to quantify the value of IT-enabled business investments
Economics
Decision/Game Theory
Empirical Decision Theory
Operations Research
Applied
Information
Economics
Modern Portfolio
Theory
Statistics
Information
Engineering
Information Theory
Software
Metrics

5. Some HDR Clients Booz-Allen & Hamilton
Accenture w/ the State of North Carolina
Giga Information Group
American Express
The Discovery Channel
U.S. Federal Government:
Department of Treasury
Bureau of The Census
Department of Veterans Affairs
General Services Administration
Housing and Urban Development
The Axa Group – 6 major companies
The Banking Industry Technology Secretariat
Blue Cross Blue Shield of Illinois

6. What Do the Critics Say?
“Quantifying the risk and comparing its risk/return with other investments sets AIE apart from other methodologies. It can substantially assist in financially justifying a project -- especially projects that promise significant intangible benefits.” The Gartner Group
“AIE represents a rigorous, quantitative approach to improving IT investment decision making…..this investment will return multiples by enabling much better decision making. Giga recommends that IT executives learn more about AIE and begin to adopt its tools and methodologies, especially for large IT projects.” Giga Information Group
“AIE-like methods must become the standard way to make (IT) investment decisions.” Forrester Research, Inc.

7. Five Key Revelations
Quantifying risk radically changes IT investment priorities
Current measurement priorities are “upside-down” when compared to priorities based on economic value of information
“Technology regret” is a significant and overlooked factor in the the value of IT investments
The true cost of “scope creep” is much higher than most would think
The value of quantitative analysis would make it the best investment in most IT portfolios

8. Finding 1: Risk Analysis
When IT computes risk in the same way that an actuary would, many IT investments will look completely different
We define risk a “The probability of a quantified loss”
Risk analysis of IT investments involves a probabilistic analysis of all uncertain variables

9. Real-world Measurements vs. Ideal Values
Ideal Values: Point
Real-world Meas.
Normal Distribution
Uniform Distribution
Lognormal Distribution
Hybrid
Threshold confidence
15%
85%

10. Calibrated Estimates
Measuring your own uncertainty about a quantity is a general skill that can be taught with a measurable improvement
Studies show that most managers are statistically “overconfident” when assessing their own uncertainty
Training can “calibrate” people so that when they provide a 90% confidence interval, it still has a 90% chance of being right (even though it is subjective)
When asked to provide a subjective 90%
confidence interval, most managers provide
a range that only has about a 40%-50%
chance of being right
Perceived 90% Confidence Interval
Actual 90% Confidence Interval

11. Distribution-based ROI
Administrative Cost Reduction
Total Project Cost
% Improvement in Customer Retention 5%
10%
15%
20%
30%
$2 million
$4 million
$6 million
ROI
-50%
50%
100% 0%
Inputs

12. Analyzing the Distribution
ROI = 0%
“Expected” ROI
Risk of Negative ROI
Probability of Positive ROI
The “cancellation hump”
-25% 0%
25%
50%
75%
100%
125%
Return on Investment (ROI)

13. Plotting the Risk and Return
40%
A proposed IT investment with a 15% risk and 54% return
30%
Probability of less than a
risk-free return
20% X
10%
Return
10%
20%
30%
40%
50%
60%

14. Example of Risk Effects
50%
These are real IT investments of $2M-$3M plotted against a client’s investment boundary
The 27% ROI investment is actually preferred to the 83% ROI investment
Region of Unacceptable Investments
40%
30%
Chance of a negative IRR
20%
Region of Acceptable Investments
10% 0% 0%
50%
100%
150%
200%
Expected IRR over 5 years

15. Risk Increases Required ROI’s
Adjusting for risk causes some previously-acceptable projects to be rejected
Also, some low return but low risk projects would now be acceptable
More projects with “intangible” benefits are now economically justified
The net result: A completely reshuffled deck of IT project approvals
10%
100%
1000%
20%
40%
60%
80%
30%
50%
200%
300%
500%
Size of the Project Relative to the Entire IT Portfolio
(i.e. 50% = project makes up half the work in the entire portfolio)
Required Minimum Return (IRR over 5 years)
Most Risk Averse
Approximate Median
Most Risk Tolerant
Range of Typical “Hurdle Rates”

16. Using Risk Analysis to Improve Decisions
If the Risk is significant (it usually is), consider doing the following:
Reduce the size and functionality of the proposed system - focus on fewer high-return features
Wait until specific uncertainties in the environment subside - e.g. major mergers, reengineering, etc.
Wait to tackle big projects until proper skills are developed and methods are in place
Consider purchased packages that aren’t a perfect fit but close enough - they may look more attractive now
Invest more on a proper economic analysis of the largest IT investments - this should reduce uncertainty about critical quantities
Include deferred benefits in any estimate of scope creep costs

17. Finding 2: Measurements
Information has a value that can be calculated with a formula known for 50 years
Computing the value of additional information on uncertain variables causes some surprising changes in what gets measures

18. The Economic Value of Information
The Decision Theory Formula:
What it means:
Information reduces uncertainty
Reduced uncertainty improves decisions
Improved decisions satisfy business objectives (by definition)

19. The IT Measurement Inversion
Receives Most Attention
Least Relevant to Approval Decisions
Costs
Initial Development Costs
Ongoing Maintenance/Training Costs
Benefits
A specific benefit (productivity, sales, etc.)
Utilization (when usage starts and how quickly usage grows)
Chance of Cancellation
Economic Relevance
Typical Attention
Receives Least Attention
Most Relevant to Approval Decisions
See my article “The IT Measurement Inversion” in CIO Magazine
(its also on my website at under the “articles” link)

20. Finding 3: Technology Regret
Most business cases treat IT investments implicitly as a “now or never” choice
Technology regret is an economic quantity associated with committing to a technology after which, for whatever reason, becomes regrettable
The equivalent of “buyers remorse”
Technology regret becomes and important consideration in any environment where changing technology is a constant
The issue becomes balancing technology regret (which tends to defer IT investments) vs. the opportunity loss of deferring the benefits of making the investment now

21. Relative Computing Power
Changing Technology
32% Annual Growth Rate
350
Some Areas of Growth:
Processors & Memory
(Moore’s Law)
Storage
Communications (Payne’s Law)
Internet Users
Sensory devices
300
250
Relative Computing Power
Per $ (1980=1)
200
150
100 50
1980
1985
1990
1995
2000
Year
Competition makes capitalizing on new technology more critical to survival

22. Changing With Technology
How often should you change with technology? Avoiding “technology regret” is often a major driver in IT decisions.
Upgrade 2
A Critical Technology Measure
Upgrade 1
Time

23. “Real Option” Theory - +
The option value tells us when an investment, even if it looks positive now, should be deferred until the opportunity is better
In the case of IT, waiting for the possibility of better technology around the corner should be considered -
Re-evaluate in the next decision cycle
Reject the investment
Invest this cycle, low priority, may be deferred if resources are strained
Single Period Option Value
(Value of Waiting one period)
Invest in this cycle, high priority +
Net Value of the Investment

24. The Effects of Tech Regret
Very long duration IT projects that commit to a proprietary solution tend to look much less favorable
Short turnaround projects based on non-proprietary standards tend to look better
Large scale commitments to the fastest improving technologies (like data storage, bandwidth) tend not to be favorable

25. Finding 4: Scope Creep
The cost of adding one additional function to an software development project is rarely addressed properly
If anything, the only cost of new features considered is development cost
Long term maintenance, increased risk of cancellation plus deferred benefits is much more significant

26. True Scope Creep Costs 24%: Initial development costs
24%: Future maintenance costs (computed over 5 years)
1%: Incremental contribution to probability of total project cancellation
51%: Deferred benefits of the other functions delayed by the proposed new function
24%
51%
24% 1%

27. Finding 5: Value of Quant.
Organizations have successfully adopted more advanced quantitative methods for evaluating IT investments
The cost of analysis routinely comes in below 1% and has always been under 2% of the investment size - including initial training
This is still less than non-IT investments of similar size and risk
It is also sometimes less time-consuming than the previous non-quantitative analysis techniques used by the firm (One of the reasons this analysis is efficient is we conduct a Value of Information Analysis - we only measure what is economically justified)
Using the standard VIA calculation for the value of AIE analysis, AIE itself was the best investment of all the IT investments we analyzed - very conservative measures of payoffs put $20 to every $1 spent on AIE

28. Overview of RRA Analysis
Classification 5%
10%
15%
20%
30%
$2 mill
$4 mill
$6 mill
Measurements
Value of Info. $
Intangibles
“Customer Satisfaction”
“Strategic Alignment”
“Technology Risk”
“Information Quality” etc.
$$$ $$
Measurables
Errors in Decision X
Change to Strategic Measure M
Productivity in Activity Y
Chance of cancellation, etc.
Each company will decide on its own what “compliance”, “strategic” and “economic” mean precisely. Also, they will determine their own boundaries on the classification chart and the specific analysis requirements in each “zone” of the chart.
Risk
Calculate Risk/Return Position
"expected" ROI
50%
100%
150%
200%
250%
-50% 0%
Probability
of a negative ROI
Probability of a positive ROI
Organization's investment limit
Acceptable region of investment
Return

29. In Conclusion…
Quantitative methods like AIE cause major IT decisions to be very different – and better
Advanced methods can and have been learned and adopted by IT organizations
More quantitative analysis can be the highest return investment in your IT portfolio