Copyright QSM Associates, Inc. 1 Michael C. Mah Managing Partner QSM Associates, Inc. 75 South Church Street Pittsfield, MA Fax Presentation for Chicago SPIN January, 2002 Deadline-Driven Software Project Estimation Negotiating Trade-offs and Risks Web Site:

Copyright QSM Associates, Inc. 2 Background: QSM Associates  “The Metrics Company”  QSM Software Lifecycle Management Tools (SLIM Suite) Used Worldwide by Fortune 100 Clients to Measure, Estimate, and Control Software Development  “Management By The Numbers” - “Numbers into Pictures”

Copyright QSM Associates, Inc. 3 Partial List of QSM Clients v British Telecom v EDS v Rockwell v Intel v BellSouth v IBM Global Services v Sprint v Honeywell v Computer Sciences Corp v United Technologies v GTE v Compaq v Keane v Boeing v Royal Bank of Canada v Lockheed Martin v Alcatel

Copyright QSM Associates, Inc. 4 Worldwide Software Productivity Trend v The Good News: Over the last decade, sw dev’t productivity has increased fairly consistently. 10% faster speed, 25% less cost, about every 2.5 years. v The Bad News: It’s Not Enough. Demand continues to outstrip capacity. v Companies are reporting growing backlogs ranging from 7 months to over 2 years. v The Badder News: Continued labor shortages. v The More Badder News: Don’t expect any schedule relief in an “Internet Speed” economy.

Copyright QSM Associates, Inc. 5 Industry Overrun Statistics v $250 Billion + Spent on IT Application Development v 31% of Projects Will Be Cancelled, Representing $81 Billion in Losses v 52.7% of Projects Will Overrun by >189% v Only 16.2% On-time, Under Budget v But with only 42% Original Functionality! * Source: Standish Group, Dennis MA,

Copyright QSM Associates, Inc. 6 Anyone Know What This Is?

Copyright QSM Associates, Inc. 7 Reality Bites “We don’t have the luxury of determining our schedules. They’re told to us. Then, given the time frame, we try to tell the client what we can build. In the end, we usually wind up working lots of overtime, because they want everything.” - Manager of Development Wall St. Financial Firm

Copyright QSM Associates, Inc. 8 Size Probability on Time Scenario: Size Growth, or “Feature Creep” 50% 75% 25% A B C

Copyright QSM Associates, Inc. 9 Good to Know... " As impressive as growth of the software industry has been, it is outpaced by growth of software- related litigation. It is not unusual for a large software development organization today to have upwards of 50 active cases on its hands." Tom DeMarco, Cutter IT Journal

Copyright QSM Associates, Inc. 10 “Most litigation ends up focused on [lack of] measurement, management, requirements practice, or some combination thereof.” “Organizations that can’t or don’t measure themselves in a fairly systematic way are at a huge disadvantage in litigation. If you are deficient at measurement and the other side is on top of it, then the jig is up for you.” Tim Lister Cutter IT Journal Good to Know...

Copyright QSM Associates, Inc. 11 Real Time Engineering Info Systems Impossible Zone Are Deadlines/Plans in the “Impossible Zone”?

Copyright QSM Associates, Inc. 12 Building Your Own Benchmarks All Projects in SampleAvg.1 Sigma

Copyright QSM Associates, Inc Number of Projects BASELINE PI All Projects in SampleAvg.1 Sigma Building Your Own Benchmarks

Copyright QSM Associates, Inc. 14 Ed Yourdon on “Sizing”.. “Studies by the Carnegie Mellon SEI indicate that the most common failing of Level 1 (Ad-hoc) software organizations is an inability to make size estimates accurately.”

Copyright QSM Associates, Inc. 15 Ed Yourdon on “Sizing”.. “If you underestimate the size of your next project, common sense says that it doesn’t matter which methodology you use, what tools you buy, or even what programmers you assign to the job.”

Copyright QSM Associates, Inc. 16 Many Functional Metrics Can be used to Represent S/W Size v Number of subsystems v Number of entities v Number of function points v Number of modules v Number of objects v Number of programs v Number of SLOC v Number of object instructions

Copyright QSM Associates, Inc. 17 Breaking things Down to Size Existing Code, Database 4GL, Forms, PL/SQL, Pro*C, Reports, Utilities, etc. (Item C) New Code (Item A) Modified Code (Item B) Existing, or “Base Code”

Copyright QSM Associates, Inc. 18 When Time & Effort are “Fixed” Something’s Got to Give... Size Time Effort Defects

Copyright QSM Associates, Inc. 19 “Feature Creep” - Impact on Quality with Fixed Schedule Schedule Fixed at 14 Months Size Increases in 5KSLOC Increments from 72K to 92K Quality Levels Cut in Half - MTTD 3.5 Days to 1.75 Days

Copyright QSM Associates, Inc. 20 Overall Project Risk: Green (Minimal Risk) Schedule, Cost, and Quality Targets all at 80% Probability or Better Risk Analysis - Determine the Probability of Success Deadline

Copyright QSM Associates, Inc. 21 When Schedules/Resources are Fixed - Assess Functionality Target Schedule Target Cost Target Quality Size Range to Test

Copyright QSM Associates, Inc. 22 MB /9/97 Life Cycle /1/97 Size ESLOC MBI3.3 PI16.0 Risk Analysis - Determine the Probability of Success Overall Project Risk: Red (High Risk) Only 45% Probability of Meeting Target Schedule Deadline Cost and Quality - High Probability

Copyright QSM Associates, Inc. Defects Discovered Each Month High defect rate Low Mean Time to Defect Poor Quality Low defect rate High Mean Time to Defect Good Quality Reliability Modeling Mean Time to Defect

Copyright QSM Associates, Inc. 24 How Development Lifecycles Behave A Pop Quiz: “If you tried to shorten the schedule on an application development project by adding staff to say, double (20 people versus 10), how much will you able to compress it? Will defects go up or down, and by how much?”

Copyright QSM Associates, Inc. 25 How Software Lifecycles Behave Answer: Schedules will only compress (nominally) by about 20 percent. Defects typically rise by about 6 fold.* Rule of Thumb: “20/200/6x” *Source: QSM Industry Database Statistics

Copyright QSM Associates, Inc. Reliability Modeling Defect Severity Categories 5 Severity Categories

Copyright QSM Associates, Inc. Total Defect Rate S Defects * Jan '95 AprJulOctJan '96 AprJulOctJan '97 AprJulOctJan '98 Actual Interpolated Plan Green CB Yellow CB S = Start 1 = PDR 2 = Bld_1 3 = CDR 4 = Bld_2 5 = TRR 7 = Bld_3 Reliability Modeling Real Data - Actual Vs. Planned Defects Starting from Design

Copyright QSM Associates, Inc. Reliability Modeling Real Data - Actual Vs. Planned Defects Starting from Code Total Defect Rate S75421 S421 Defects * 2/4 '95 3/44/14/295/276/247/228/199/1610/14 Actual Interpolated Plan Green CB Yellow CB S = Start 1 = RB 2 = DD 4 = SIT 5 = UOST 7 = FOC

Copyright QSM Associates, Inc. v These reliability drivers are able to be controlled! Reliability and availability is something that we can INFLUENCE! Reliability Modeling The Moral of the Story Size Staffing Productivity Variables Defects

Copyright QSM Associates, Inc. 30 Core Metrics Provide an “Early Warning Indicator” Schedule/Milestones Appear on Target Cost is Close to the Plan Product is Being Constructed at a Slower Rate than Planned Defect Rates are Higher than Planned Example Yellow Green

Copyright QSM Associates, Inc. 31 Traffic Lights - A Visual Trigger for Course Correction Data are Consistently in the Amber Region Example

Copyright QSM Associates, Inc. 32

Copyright QSM Associates, Inc. 33 Adaptive Forecasting: What’s the Remaining “Trajectory”? Forecasted Schedule: +5.5 Months Forecasted Cost: +$620K Forecasted Code Production: Planned PI 10.8, Actual PI Total Defects Will Be Remaining at the Initial Production Date

Copyright QSM Associates, Inc. 34 Case Study: Original Plan vs. Actual Data Actual Data (black squares) Plan Data Bounds Forecast Data (white squares) Example

Copyright QSM Associates, Inc. 35 Case Study: Re-Plan vs. Actual Data Defects currently tracking in high range; probably due to early aggressive schedule Forecast is on target with plan Example

Copyright QSM Associates, Inc. 36 Project Office Warboard (Currently in Use by QSM Clients) * * underrun ** no data **

Copyright QSM Associates, Inc. 37 In Summary... v It Comes Down to Promises, Commitments, and Expectations! v Understand that SW Dev’t is R&D, and It Behaves That Way! Non-linear interdependencies. v Apply “Negotiation on the Merits” v Generate Multiple Options v Test Each Option for Legitimacy, Reasonableness, & Risk v “Know Your Capability” – Estimates Based on History

Copyright QSM Associates, Inc. 38 The Role of Measurement, and Commitments Estimation & Planning Control & Forecasting Support Future Commitments Manage Commitment CommitmentAnalyze Performance on Commitment History Repository Assess Viable Strategies Monitor Status & Replan Post Project Analysis Make Commitment

Copyright QSM Associates, Inc. 39 Info Sources on the Web v Software Measurement, Estimation, Control QSM Associates - v Information Technology Research Pubs Cutter Consortium - v Negotiation Program on Negotiation at Harvard Law - Workshops from QSM Associates/Triad Consulting -

Copyright QSM Associates, Inc. 40 Recommended Reading - Negotiation v Fisher, Roger and Alan Sharp, “Getting It Done, How to Lead When You’re Not in Charge” HarperCollins v Fisher, Roger, William Ury and Bruce Patton, “Getting to YES, Negotiating Agreement Without Giving In” Penguin v Heen, Sheila, Doug Stone and Bruce Patton “Difficult Conversations - How to Discuss What Matters Most” Viking/Penguin 1999.

Copyright QSM Associates, Inc. 41 Recommended Reading - Metrics  Carleton, Anita, Park, Robert, and Goethert, Wolfhart, “The SEI Core Measures: Background Information and Recommendations for Use and Implementation” © 1994 The Journal of the Quality Assurance Institute.  Mah, Michael C., “Software Estimation Tricks of the Trade; Secrets They Never Told Me” IT Metrics Strategies © June 2000 Cutter Information Corp.  Putnam, Lawrence H., and Myers, Ware, “Executive Briefing: Controlling Software Development” © 1996 IEEE Computer Society Press.  Tufte, Edward, “Visual Explanations, Images and Quantities, Evidence and Narrative” © 1997 Graphics Press.