1. Classification Overview

2. Hubbard Decision Research The Applied Information Economics Company Overview  A classification chart is one type of bootstrapping that produces a “solution space” of two variables  The vertical axis is a bootstrapped variable called “Confidence Index” – it is the probability of success (defined in a particular way) of a project  The horizontal axis is the estimated size of the investment  For small and low-risk investments, the decision to accept should be made without a full RRA assessment  Larger and riskier investments will tend to require a full RRA assessment

3. Hubbard Decision Research The Applied Information Economics Company Classification Chart No Classification Required RRA Light Accept w/o Further Analysis Reject w/o Further Analysis Proceed with Full RRA Expected Investment Size Confidence Index (confidence about value).2.4.6.8 1.0 0 10k 100k1M10M100M

4. Hubbard Decision Research The Applied Information Economics Company Defining the Confidence Index  The confidence index is meant to be an indicator of the chance of success of an investment  “Success” must be defined by the participants in the workshops  Then a bootstrap model is built (see bootstrapping procedure) and evaluations are given on the following two questions: What is the chance of success of this investment? (0% to 100%) How much analysis should be required? (accept w/o further analysis, reject w/further analysis, continue RRA)

5. Hubbard Decision Research The Applied Information Economics Company Placing the Boundaries  Three methods are used in concert Decision maker interviews Checking against classification boundary constraints Checking responses to the “How much analysis is required for this investment” question from the bootstrap list

6. Hubbard Decision Research The Applied Information Economics Company Example Questions for Building the Chart No Classification Required Abbreviated Deliverable Accept w/o Further Analysis Reject w/o Further Analysis.2.4.6.8 1.0 0 10k 100k1M10M100M “Even if the confidence index were 100%, how big would an investment need to be to proceed with a RRA assessment?”  “At the minimum size required for classification, how much does the confidence index have to be for you to accept the investment without further analysis?” “What is the minimum size of an investment before classification is required?”  

7. Hubbard Decision Research The Applied Information Economics Company.2.4.6.8 1.0 0 10k 100k1M10M 100M Classification Boundaries Constraints.2.4.6.8 1.0 0 10k 100k1M10M 100M Must be a vertical line These areas should not be touched by the classification boundaries No point on the boundaries of the “Risk/Return Analysis” area can be to the left of the Triple Point Lower bound of “Risk/Return Analysis” area must touch this range The Triple Point should be within this zone Must be flat or slope up to the Triple Point Upper bound of “Risk/Return Analysis” area must touch this range

8. Hubbard Decision Research The Applied Information Economics Company Check Boundaries with Bootstrap  If we ask the question “What action would you take with this investment” we may find that our boundaries need adjustment  Plot the various responses with color-coding so that we can check boundaries against bootstrapped preferences No Classification Required.2.4.6.8 1.0 0 10k 100k1M10M100M Proceed w/RRA Accept Reject

9. Hubbard Decision Research The Applied Information Economics Company Plotting the Investment  When an individual project is actually classified the investment size and the confidence index have error  The two ranges produce the shape of an ellipse in two dimensions Expected Investment Size Confidence Index No Classification Required.2.4.6.8 1.0 0 10k 100k1M10M100M { {

10. Hubbard Decision Research The Applied Information Economics Company Optional Zones  Optionally, additional zones may be added if there is a dilemma about how to proceed  Sometimes simply changing those success factors that are controllable can make the investment acceptable – this may indicate another zone  If RRA Light should be used for investments under a certain size, then a zone can be added for that. 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 101001,00010,000 Expected Investment Size ($000) Confidence Index Reject & Consider Options SF Adj. RRA Light: Just a spreadsheet and a page or two of explanatory material RRA Light: Just a spreadsheet and a page or two of explanatory material Accept RRA Light No Classification Needed RRA Standard Success Factor Adjustments: a better technology record, single sponsor, acquiring capability for ITG, Sponsor w/better track record could make the difference Success Factor Adjustments: a better technology record, single sponsor, acquiring capability for ITG, Sponsor w/better track record could make the difference

11. Hubbard Decision Research The Applied Information Economics Company Confirm Results  To confirm results show each of the following: Plot of the original estimates vs. the model The test classification chart Plot actual projects on classification chart and discuss discrepancies  Determine volumes in each zone to check if support is realistic  Present results to group

12. Hubbard Decision Research The Applied Information Economics Company Actual Classification Plots  An Illinois insurance company created a classification chart to help prioritize the current list of proposed investments  They wanted to determine which investments could be accepted without more analysis and which need more analysis  18 investments were plotted on the classification chart  The results had a profound effect on investment priorities  Some investments that were assumed to be beneficial now required analysis and some that required analysis could now be approved immediately

13. Hubbard Decision Research The Applied Information Economics Company Regression Example  Input to the model was based on average VP “calibrated” estimates of the probability of success of 42 hypothetical investments.  Each investment was described by 13 variables like project duration, # of ITG units involved, sponsorship, etc.  To test consistency, 3 investments were duplicates of 3 others.  Disagreement among VPs on the same investment was 30% on average. 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 00.10.20.30.40.50.60.70.80.91 Average of VPs’ calibrated estimates Computed Confidence Index Comparison of estimates to model

14. Hubbard Decision Research The Applied Information Economics Company Classification of Example Projects 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 101001,00010,000 1 2 Expected Investment Size ($000) Confidence Index No Classification Needed Do Abbreviated Risk-Return Analysis: 6. DLSW Router Network Redesign 9. Extended Hours 18. Doc. Access Strategy Do Abbreviated Risk-Return Analysis: 6. DLSW Router Network Redesign 9. Extended Hours 18. Doc. Access Strategy Do Full Risk- Return Analysis: 8. Pearl Indicator and Pearl I/O interface 11. Richardson Data Center Consolidation 15. MVS DB2 Tools Do Full Risk- Return Analysis: 8. Pearl Indicator and Pearl I/O interface 11. Richardson Data Center Consolidation 15. MVS DB2 Tools Reject; Consider Other Options: 1. Data Strategy 2. Enterprise Security Strategy 3. Remote Server Redundancy 12. MQ Series: Base 13. Development Environment 2000 (mf) 14. “Source Control” Source Code Mgmt 16. Enterprise InterNet Reject; Consider Other Options: 1. Data Strategy 2. Enterprise Security Strategy 3. Remote Server Redundancy 12. MQ Series: Base 13. Development Environment 2000 (mf) 14. “Source Control” Source Code Mgmt 16. Enterprise InterNet Success Factor Adjustments: 4. Network OS migration to Novell 5.x 10. Optimize Single Code Base Success Factor Adjustments: 4. Network OS migration to Novell 5.x 10. Optimize Single Code Base Accept without Further Analysis: 5. Lucent switch upgrade 7. Image Server Relocation 17. Enterprise IntraNet to all sites Accept without Further Analysis: 5. Lucent switch upgrade 7. Image Server Relocation 17. Enterprise IntraNet to all sites

15. Hubbard Decision Research The Applied Information Economics Company Impact of Classification  Although it was a non-standard application of classification, the exercise had a significant impact on the IT priorities  3 investments plotted in the “Accept without further analysis” area – each of these were accepted and unnecessary analysis effort was avoided  Some of the more popular projects plotted very poorly, causing them to rethink the approach and scope of these projects  Risk return assessments were required for some that were assumed to be low risk

16. Hubbard Decision Research The Applied Information Economics Company Proportion of Investments Analyzed  When classification is applied we find that larger companies will do RRA Standard on a larger percentage (by budget) of their portfolios  Even though they are a small percentage of the budget, a very large number of smaller investments are accepted or rejected on the classification chart alone Belgian HDR clientAustralian HDR client RRA Standard RRA Standard RRA Light Decide by Classification Index alone None Decide by Classification Index alone None