1. Copyright QSM Associates, Inc. 1 Michael C. Mah Managing Partner QSM Associates, Inc. 75 South Church Street Pittsfield, MA 01201 413-499-0988 Fax 413-447-7322 e-mail: michaelm@qsma.com Presentation for Chicago SPIN January, 2002 Deadline-Driven Software Project Estimation Negotiating Trade-offs and Risks Web Site: www.qsma.com

2. Copyright QSM Associates, Inc. 2 Background: QSM Associates  “The Metrics Company”  QSM Software Lifecycle Management Tools (SLIM Suite) Used Worldwide by Fortune 100 Clients to Measure, Estimate, and Control Software Development  “Management By The Numbers” - “Numbers into Pictures”

3. Copyright QSM Associates, Inc. 3 Partial List of QSM Clients v British Telecom v EDS v Rockwell v Intel v BellSouth v IBM Global Services v Sprint v Honeywell v Computer Sciences Corp v United Technologies v GTE v Compaq v Keane v Boeing v Royal Bank of Canada v Lockheed Martin v Alcatel

4. Copyright QSM Associates, Inc. 4 Worldwide Software Productivity Trend v The Good News: Over the last decade, sw dev’t productivity has increased fairly consistently. 10% faster speed, 25% less cost, about every 2.5 years. v The Bad News: It’s Not Enough. Demand continues to outstrip capacity. v Companies are reporting growing backlogs ranging from 7 months to over 2 years. v The Badder News: Continued labor shortages. v The More Badder News: Don’t expect any schedule relief in an “Internet Speed” economy.

5. Copyright QSM Associates, Inc. 5 Industry Overrun Statistics v $250 Billion + Spent on IT Application Development v 31% of Projects Will Be Cancelled, Representing $81 Billion in Losses v 52.7% of Projects Will Overrun by >189% v Only 16.2% On-time, Under Budget v But with only 42% Original Functionality! * Source: Standish Group, Dennis MA,

6. Copyright QSM Associates, Inc. 6 Anyone Know What This Is?

7. Copyright QSM Associates, Inc. 7 Reality Bites “We don’t have the luxury of determining our schedules. They’re told to us. Then, given the time frame, we try to tell the client what we can build. In the end, we usually wind up working lots of overtime, because they want everything.” - Manager of Development Wall St. Financial Firm

8. Copyright QSM Associates, Inc. 8 Size Probability on Time Scenario: Size Growth, or “Feature Creep” 50% 75% 25% A B C

9. Copyright QSM Associates, Inc. 9 Good to Know... " As impressive as growth of the software industry has been, it is outpaced by growth of software- related litigation. It is not unusual for a large software development organization today to have upwards of 50 active cases on its hands." Tom DeMarco, Cutter IT Journal

10. Copyright QSM Associates, Inc. 10 “Most litigation ends up focused on [lack of] measurement, management, requirements practice, or some combination thereof.” “Organizations that can’t or don’t measure themselves in a fairly systematic way are at a huge disadvantage in litigation. If you are deficient at measurement and the other side is on top of it, then the jig is up for you.” Tim Lister Cutter IT Journal Good to Know...

11. Copyright QSM Associates, Inc. 11 Real Time Engineering Info Systems Impossible Zone Are Deadlines/Plans in the “Impossible Zone”?

12. Copyright QSM Associates, Inc. 12 Building Your Own Benchmarks All Projects in SampleAvg.1 Sigma

13. Copyright QSM Associates, Inc. 13 579111315171921232527293133353739 Number of Projects 0 2 4 6 8 2001 BASELINE PI All Projects in SampleAvg.1 Sigma Building Your Own Benchmarks

14. Copyright QSM Associates, Inc. 14 Ed Yourdon on “Sizing”.. “Studies by the Carnegie Mellon SEI indicate that the most common failing of Level 1 (Ad-hoc) software organizations is an inability to make size estimates accurately.”

15. Copyright QSM Associates, Inc. 15 Ed Yourdon on “Sizing”.. “If you underestimate the size of your next project, common sense says that it doesn’t matter which methodology you use, what tools you buy, or even what programmers you assign to the job.”

16. Copyright QSM Associates, Inc. 16 Many Functional Metrics Can be used to Represent S/W Size v Number of subsystems v Number of entities v Number of function points v Number of modules v Number of objects v Number of programs v Number of SLOC v Number of object instructions

17. Copyright QSM Associates, Inc. 17 Breaking things Down to Size Existing Code, Database 4GL, Forms, PL/SQL, Pro*C, Reports, Utilities, etc. (Item C) New Code (Item A) Modified Code (Item B) Existing, or “Base Code”

18. Copyright QSM Associates, Inc. 18 When Time & Effort are “Fixed” Something’s Got to Give... Size Time Effort Defects

19. Copyright QSM Associates, Inc. 19 “Feature Creep” - Impact on Quality with Fixed Schedule Schedule Fixed at 14 Months Size Increases in 5KSLOC Increments from 72K to 92K Quality Levels Cut in Half - MTTD 3.5 Days to 1.75 Days

20. Copyright QSM Associates, Inc. 20 Overall Project Risk: Green (Minimal Risk) Schedule, Cost, and Quality Targets all at 80% Probability or Better Risk Analysis - Determine the Probability of Success Deadline

21. Copyright QSM Associates, Inc. 21 When Schedules/Resources are Fixed - Assess Functionality Target Schedule Target Cost Target Quality Size Range to Test

22. Copyright QSM Associates, Inc. 22 MB 8.82 65.33 599 10.00 2.05 8/9/97 Life Cycle 18.29 104.34 956 10.00 8.88 1/1/97 Size 46473 ESLOC MBI3.3 PI16.0 Risk Analysis - Determine the Probability of Success Overall Project Risk: Red (High Risk) Only 45% Probability of Meeting Target Schedule Deadline Cost and Quality - High Probability

23. Copyright QSM Associates, Inc. Defects Discovered Each Month High defect rate Low Mean Time to Defect Poor Quality Low defect rate High Mean Time to Defect Good Quality Reliability Modeling Mean Time to Defect

24. Copyright QSM Associates, Inc. 24 How Development Lifecycles Behave A Pop Quiz: “If you tried to shorten the schedule on an application development project by adding staff to say, double (20 people versus 10), how much will you able to compress it? Will defects go up or down, and by how much?”

25. Copyright QSM Associates, Inc. 25 How Software Lifecycles Behave Answer: Schedules will only compress (nominally) by about 20 percent. Defects typically rise by about 6 fold.* Rule of Thumb: “20/200/6x” *Source: QSM Industry Database Statistics

26. Copyright QSM Associates, Inc. Reliability Modeling Defect Severity Categories 5 Severity Categories

27. Copyright QSM Associates, Inc. Total Defect Rate 0 100 200 300 400 500 S754321 Defects 258111417202326293235 * Jan '95 AprJulOctJan '96 AprJulOctJan '97 AprJulOctJan '98 Actual Interpolated Plan Green CB Yellow CB S = Start 1 = PDR 2 = Bld_1 3 = CDR 4 = Bld_2 5 = TRR 7 = Bld_3 Reliability Modeling Real Data - Actual Vs. Planned Defects Starting from Design

28. Copyright QSM Associates, Inc. Reliability Modeling Real Data - Actual Vs. Planned Defects Starting from Code Total Defect Rate 0 2 4 6 8 10 12 14 S75421 S421 Defects 15913172125293337 * 2/4 '95 3/44/14/295/276/247/228/199/1610/14 Actual Interpolated Plan Green CB Yellow CB S = Start 1 = RB 2 = DD 4 = SIT 5 = UOST 7 = FOC

29. Copyright QSM Associates, Inc. v These reliability drivers are able to be controlled! Reliability and availability is something that we can INFLUENCE! Reliability Modeling The Moral of the Story Size Staffing Productivity Variables Defects

30. Copyright QSM Associates, Inc. 30 Core Metrics Provide an “Early Warning Indicator” Schedule/Milestones Appear on Target Cost is Close to the Plan Product is Being Constructed at a Slower Rate than Planned Defect Rates are Higher than Planned Example Yellow Green

31. Copyright QSM Associates, Inc. 31 Traffic Lights - A Visual Trigger for Course Correction Data are Consistently in the Amber Region Example

32. Copyright QSM Associates, Inc. 32

33. Copyright QSM Associates, Inc. 33 Adaptive Forecasting: What’s the Remaining “Trajectory”? Forecasted Schedule: +5.5 Months Forecasted Cost: +$620K Forecasted Code Production: Planned PI 10.8, Actual PI 9.1 40 Total Defects Will Be Remaining at the Initial Production Date

34. Copyright QSM Associates, Inc. 34 Case Study: Original Plan vs. Actual Data Actual Data (black squares) Plan Data Bounds Forecast Data (white squares) Example

35. Copyright QSM Associates, Inc. 35 Case Study: Re-Plan vs. Actual Data Defects currently tracking in high range; probably due to early aggressive schedule Forecast is on target with plan Example

36. Copyright QSM Associates, Inc. 36 Project Office Warboard (Currently in Use by QSM Clients) * * underrun ** no data **

37. Copyright QSM Associates, Inc. 37 In Summary... v It Comes Down to Promises, Commitments, and Expectations! v Understand that SW Dev’t is R&D, and It Behaves That Way! Non-linear interdependencies. v Apply “Negotiation on the Merits” v Generate Multiple Options v Test Each Option for Legitimacy, Reasonableness, & Risk v “Know Your Capability” – Estimates Based on History

38. Copyright QSM Associates, Inc. 38 The Role of Measurement, and Commitments Estimation & Planning Control & Forecasting Support Future Commitments Manage Commitment CommitmentAnalyze Performance on Commitment History Repository Assess Viable Strategies Monitor Status & Replan Post Project Analysis Make Commitment

39. Copyright QSM Associates, Inc. 39 Info Sources on the Web v Software Measurement, Estimation, Control QSM Associates - www.qsma.com v Information Technology Research Pubs Cutter Consortium - www.cutter.com/consortium v Negotiation Program on Negotiation at Harvard Law - www.pon.harvard.edu Workshops from QSM Associates/Triad Consulting - www.qsma.com/education.html

40. Copyright QSM Associates, Inc. 40 Recommended Reading - Negotiation v Fisher, Roger and Alan Sharp, “Getting It Done, How to Lead When You’re Not in Charge” HarperCollins 1998. v Fisher, Roger, William Ury and Bruce Patton, “Getting to YES, Negotiating Agreement Without Giving In” Penguin 1981. v Heen, Sheila, Doug Stone and Bruce Patton “Difficult Conversations - How to Discuss What Matters Most” Viking/Penguin 1999.

41. Copyright QSM Associates, Inc. 41 Recommended Reading - Metrics  Carleton, Anita, Park, Robert, and Goethert, Wolfhart, “The SEI Core Measures: Background Information and Recommendations for Use and Implementation” © 1994 The Journal of the Quality Assurance Institute.  Mah, Michael C., “Software Estimation Tricks of the Trade; Secrets They Never Told Me” IT Metrics Strategies © June 2000 Cutter Information Corp.  Putnam, Lawrence H., and Myers, Ware, “Executive Briefing: Controlling Software Development” © 1996 IEEE Computer Society Press.  Tufte, Edward, “Visual Explanations, Images and Quantities, Evidence and Narrative” © 1997 Graphics Press.