2. Mythbusting Software Estimation Todd Little VP Product Development IHS

3. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Test First

4. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #1: Estimation challenges are well understood by General Management, Project Management, and Teams and it is normal to be able to estimate projects within 25% accuracy.

5. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #2: Estimation accuracy significantly improves as the project progresses

6. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #3: Estimations are frequently impacted by biases and these biases can be significant.

7. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #4: We’re pretty good at estimating things relatively

8. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #5: Velocity/Throughput is a good tool for adjusting estimates.

9. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #6: We’re a bit behind, but we’ll make it up in testing since most of our uncertainty was in the features.

10. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #7: Scope Creep is a major source of estimation error.

11. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #8: Having more estimators, even if they are not experts, improves estimation accuracy

12. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #9: Project success is determined by on-time delivery

13. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #10: Estimation is waste

14. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #1: Estimation challenges are well understood by General Management, Project Management, and Teams and it is normal to be able to estimate projects within 25% accuracy.

15. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Managing the Coming Storm Inside the Cyclone When will we get the requirements? All in good time, my little pretty, all in good time But I guess it doesn't matter anyway Doesn't anybody believe me? You're a very bad man! Just give me your estimates by this afternoon No, we need something today! I already promised the customer it will be out in 6 months No, we need it sooner. Not so fast! Not so fast!... I'll have to give the matter a little thought. Go away and come back tomorrow Ok then, it will take 2 years. Team Unity Project Kickoff

16. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. We’re not in Kansas Anymore My! People come and go so quickly here! I may not come out alive, but I'm goin' in there! The Great and Powerful Oz has got matters well in hand. "Hee hee hee ha ha! Going so soon? I wouldn't hear of it! Why, my little party's just beginning! Developer Hero Reorg Testing

17. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Why is Software Late? Genuchten 1991 IEEE General Manager Project ManagerItem 110Insufficient front end planning 23Unrealistic project plan 38Project scope underestimated 41Customer/management changes 514Insufficient contingency planning 613Inability to track progress 75Inability to track problems early 89Insufficient Number of checkpoints 94Staffing problems 102Technical complexity 116Priority Shifts 1211No commitment by personnel to plan 1312Uncooperative support groups 147Sinking team spirit 15 Unqualified project personnel

18. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. The Context of Feedback

19. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Why is Software Late? Genuchten 1991 IEEE General Manager Project ManagerItem HHCustomer/management changes HHUnrealistic project plan MHStaffing problems LHOverall complexity HLInsufficient front end planning

20. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Negotiation Bias "It is difficult to get a man to understand something when his salary depends upon his not understanding it.“ Upton Sinclair:

21. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Space Shuttle Challenger EngineersManagement Probability of loss of life1 in 1001 in 100,000 135 Flights 2 Disasters 14 Deaths

22. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. IEEE Software, May/June 2006

23. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Accuracy of Initial Estimate

24. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Data From Steve McConnell

25. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Uncertainty Percentage of Projects 10-20% Less than or equal to original estimate 50% Less than 2X original estimate 80-90% Less than 4X original estimate

26. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Actual vs. Original Estimate

27. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. 5 John Helm Agile 2013

28. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Jørgensen 2013 Put software development project for bid on online marketplace vWorker.com Received 16 bids. Reduced down to 6 bids from vendors that had high (9.5) client satisfaction. All 6 bidders went ahead and built the software

29. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Jørgensen 2013 Highest Estimate 8x the Lowest Actual/Estimate Range: 0.7 – 2.9 (4x) Actual Performance Range: Worst took 18X the effort of the best

30. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #1: Estimation challenges are well understood by General Management, Project Management, and Teams and it is normal to be able to estimate projects within 25% accuracy.

31. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #2: Estimation accuracy significantly improves as the project progresses

32. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. How does Estimation Accuracy Improve Over Time?

33. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Landmark Cone of Uncertainty

34. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. But is Uncertainty Really Reduced? “Take away an ordinary person’s illusions and you take away happiness at the same time.” Henrik Ibsen--Villanden

35. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. The Real Business Question How much work do we have left to do and when will we ship?

36. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Remaining Uncertainty

37. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Remaining Uncertainty Story Estimate

38. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #2: Estimation accuracy significantly improves as the project progresses

39. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #3: Estimations are frequently impacted by biases and these biases can be significant.

40. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Optimism Bias

41. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Test 1 (Jørgensen IEEE Software 2008) GroupGuidanceResult A800 B40 C4 DNone160

42. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Test 1 GroupGuidanceResult A800300 B40100 C460 DNone160

43. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Test 2 GroupGuidanceResult AMinor Extension BNew Functionality CExtension50

44. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Test 2 GroupGuidanceResult AMinor Extension 40 BNew Functionality 80 CExtension50

45. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Test 3 GroupGuidanceResult AFuture work at stake, efficiency will be measured BControl100

46. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Test 3 GroupGuidanceResult AFuture work at stake, efficiency will be measured 40 BControl100

47. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Understand Bias "What gets us into trouble is not what we don't know. It's what we know for sure that just ain't so.“ Mark Twain

48. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #3: Estimations are frequently impacted by biases and these biases can be significant.

49. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #4: We’re pretty good at estimating things relatively

50. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Anchoring

51. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Relative Anchoring “A” relative to “B” is not symmetric with “B” relative to “A” Jørgensen IEEE Software March 2013 Austria’s population is 70% of Hungary’s (Austria relative to Hungary), while Hungary’s population is 80% of Austria’s (Hungary relative to Austria).

52. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Relative Estimation Studies? An evaluation of the paired comparisons method for software sizing Eduardo Miranda

53. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Relative Sizing - Dimensionality

54. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #4: We’re pretty good at estimating things relatively

55. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #5: Velocity/Throughput is a good tool for adjusting estimates.

56. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Velocity Scope Creep Burnup Chart

57. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Velocity Helps Remove Bias

58. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. But Velocity is not a Silver Bullet Story Estimate

59. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #5: Velocity is a good tool for adjusting estimates.

60. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #6: We’re a bit behind, but we’ll make it up in testing since most of our uncertainty was in the features.

61. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Lan Cao - Estimating Agile Software Project Effort: An Empirical Study

62. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #6: We’re a bit behind, but we’ll make it up in testing since most of our uncertainty was in the features.

63. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #7: Scope Creep is a major source of estimation error.

64. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. We want this

65. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Scope Creep Capers Jones  2% per month  27% per year Velocity Scope Creep

66. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Estimate Velocity Net of Scope Creep

67. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Success vs. Project Duration -- Larman / Standish

68. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #7: Scope Creep is a major source of estimation error.

69. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #8: Having more estimators, even if they are not experts, improves estimation accuracy

70. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Group Estimation Exercise Number of Jellybeans in the jar

71. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Actual vs. Original Estimate Range (p90/p10)

72. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Wisdom of Crowds Jelly Beans “Who Wants To Be a Millionaire?” audience correct 91% Dutch Tulip Mania 1637

73. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Ask the Team

74. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #8: Having more estimators, even if they are not experts, improves estimation accuracy

75. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #9: Project success is determined by on-time delivery

76. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Delivery Challenges/Failures Standish Group 2006, reported by CEO Jim Johnson, CIO.com, ‘How to Spot a Failing Project’

77. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Why do we care about on-time delivery?

78. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Cost of Delay

79. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Wrong Priorities

80. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Poker Metric: Percent of Hands Won

81. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Software Metric – On Time%

82. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Value Metric Word Processing Market Share

83. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. The Measurement Inversion 82 Lowest Information Value Highest Information Value Most Measured Least Measured Cost & Time Value Delivery

84. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. OntimeNoYesNoYes OutcomeBadGood Bad Outputs and Outcomes

85. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Outputs and Outcomes Outcome GoodBad Output Late Ontime

86. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #9: Project success is determined by on-time delivery

87. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #10: Estimation is waste

88. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. The Real Business Questions Is it worth doing? What is the priority? When is the target time to ship? What is the critical scope? Do we have the right investment? What is the cost of delay?

89. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. #10: Estimation is waste

90. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Now What?

91. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Estimation and Prioritization XL L M S SML Cost Value Priority

92. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. The A/B/C List sets proper expectations (similar to MoSCoW) A MUST be completed in order to ship the product and the schedule will be slipped if necessary to make this commitment. B Is WISHED to be completed in order to ship the product, but may be dropped without consequence. C Is NOT TARGETED to be completed prior to shipping, but might make it if time allows. Only “A” features may be committed to customers. If more than 50% of the planned effort is allocated to “A” items the project is at risk.

93. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Sizing for Scope Creep 500 Point release backlog Velocity of 25 points per 2 week iteration 2%/mo = 1% scope creep per iteration = 5 pts. Net Planned Velocity = 20 pts/iteration

94. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. A A/B/C List 50%100% Backlog Plan Typical Delivery 25% AB C B C D 50% 25% Target Delivery Date

95. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. A/B/C List 50%100% Backlog Plan Uncertainty Risk 25% AB C B C D 50% 25% Target Delivery Date A

96. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Metrics to Track

97. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Velocity Scope Creep Burnup Chart Monitor Quality

98. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Ask the Team

99. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Cost of Delay

100. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Contact Todd Little todd.little@ihs.com toddelittle@gmail.com www.toddlittleweb.com www.accelinnova.com www.linkedin.com/in/toddelittle/

101. © 2012 IHS No portion of this presentation may be reproduced, reused, or otherwise distributed in any form without prior written consent. Overconfidence of Success Matthew G. Miller, Ray J. Dawson, Kieran B. Miller, Malcolm Bradley (2008). New Insights into IT Project Failure & How to Avoid It. Presented at 22nd IPMA World Congress -­ ‐ Rome (Italy) November 9-­ ‐ 11, 2008, in Stream 6. As of May 2013, self published at http://www.mgmiller.co.uk/files/paper.pdfhttp://www.mgmiller.co.uk/files/paper.pdf