CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 1 SMU CSE 8314 / NTU SE 762-N Software Metrics and Quality Engineering Module 32 Collecting and Storing Data (Metrics Data Bases)

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 2 Outline Database Issues A Database Case Study – Implementing a database – Using data at various organizational levels Data Collection Summary

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 3 Database Issues

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 4 Metrics Database Concept Do Something Do it Better Do Something Define and Collect Data to Record Experience Knowledge Database Analyze Utilize The database plays a central role

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 5 Experience vs. Knowledge Experience CM takes a lot of time But Lack of CM results in havoc The hardware changed since the software spec was written We had to work overtime to complete I&T Knowledge An efficient approach to CM can and must be devised I&T goes better if you document and control all changes to specifications and designs

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 6 Issues of Metrics Database Design What to put in? What to leave out? How to collect the data? How to motivate cooperation? How to validate the data? How to assure accuracy? How to store the data? How to use the data? These are not easy issues to solve

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 7 A Case Study to Illustrate Some of the Issues Experience at Hewlett Packard, as reported by Grady & Caswell (see references)

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 8 First Attempt Defined common, standard metrics Defined standard forms (electronic) for recording data Created a database (old-style network architecture)

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 9 On-Line, Interactive Database Electronic Forms Users Common Set of Measures Users Actions Fill out electronic forms with data Access data as needed Database

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 10 Original Database Design For Each Project: Language #2 Size of Code in That Language Language #1 Size of Code in That Language Language #3 Size of Code in That Language etc. (many items) etc.

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 11 Problems Encountered by HP’s First Database Effort Inflexibility – They kept needing to change the database schema each time the metrics changed – Too many alternatives Language (6) Size of Code (arbitrary) Too much Unused Space in the database – Not enough alternatives Too many “others” Lesson Learned: Use a relational database

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 12 Additional Problems with On Line Database Not always accessible Perceived as hard to use Many calculations were manual Required special training Hard to modify user interface Never achieved wide use -- Abandoned

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 13 Attempt 2 Standard Spreadsheet Project Name Principal Language Total SLOC Other Info Recorded SW Class Calendar Months Staff Months of Effort Size (bytes) Total Defects Introduced Total Defects Found before Release Total Defects Corrected Manager’s Name etc.

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 14 Advantages of Spreadsheet Software widely available Easy to learn / minimal training required Easy to use Easy to modify Automatic calculations

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 15 Drawbacks of Spreadsheet Integration with other tools Data distribution was manual – Later corrected Capacity problems But did achieve wide use

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 16 Experiences After Use - I Data distribution -- Who gets it? Concerns about anonymity – 5% unwilling to provide project name – 15% unwilling to provide manager’s name – HP made this information optional to pacify users Inconsistencies in data – People did not want to collect what did not help them immediately – People interpreted things differently

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 17 Experiences After Use - II Security – How to keep data within HP – Misinterpretation of data – Project name vs. product name Don’t always match Not always kept consistent Don’t want customers to associate data with products

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 18 Experiences After Use - III Data upload and download – Started with electronic approach, but: It choked the network on metrics collection days -- needed to stagger – Managers wanted physical copies So they went to a floppy disk method – Perceived as inefficient, but it worked

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 19 Experiences After Use - IV Frequency of collection and distribution – Initially they collected too often and distributed too seldom – Different data needed different collection rates – Different data needed at different organizational levels – This problem had not been fully resolved when Grady & Caswell’s book was written

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 20 Experiences After Use - V Funding for analysis and follow up – This is always a problem because it is an overhead cost, not charged to specific projects – Even though metrics experts say you get more for your money from analysis than you do from collecting lots of data

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 21 Summary of Database Design Issues - I What seems logical often does not work Expect a high degree of variation among projects Expect a high degree of concern about misuse

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 22 Summary of Database Design Issues - II Must be very easy to learn and to use Must have immediate benefit to the user if regular use is to be expected – Otherwise you have rapid fall-off People fail to collect after a while – One way to improve benefit is to measure things that benefit multiple levels in the organization

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 23 Experience Using Data at Multiple Organizational Levels Continuing the HP Case Study

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 24 Database Levels Needs and uses for data will vary depending on the level in the company It will also depend on the function of the organization Project Division Company.... We addressed this in a previous module The database design must accommodate these varying needs

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 25 Project Level Data Collection at HP Coverage was uneven and spotty – Inconsistent from month to month – Inconsistent between projects – Many omissions – “Fudged” data and dishonesty Collection of productivity data was more common than defect data – People saw immediate uses for it – Not always “good” uses Project Division Company....

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 26 Two different philosophies emerged about the key data for project use – Productivity data – Defect data In general, productivity data were kept better than defect data – Used to track & validate cost & size estimates – Used to record improvements in quality & productivity Project Use at HP

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 27 Productivity Data at Project Level trend line

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 28 Quality Data at Project Level trend line

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 29 Notes about Local Use The examples shown are all post-facto data Other project-specific data were collected but not reported or used at higher levels in the organization Projects varied significantly in what they collected and how they used it

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 30 Use at Division Level Tended to aggregate data from projects by product line or organization or type Primary impacts were: – Understanding long term effects – Understanding actual vs. plans and accuracy of estimating techniques – Database refinements Project Division Company....

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 31 Division Level Made Most Effective Use of Data Most analysis was done at this level Most trends were spotted at this level Most resources for metrics were available at this level

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 32 Productivity Data at Division Level trend line

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 33 Quality Data at Division Level

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 34 Division Level Notes Insight tended to be better when segregating data by class – Type of software real time, transactions, database, etc. – Size of software – Type of product A lot was learned about improving the metrics and improving the database

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 35 Corporate Level Notes There was very little transfer of data to the corporate level Different divisions were different in their data collection Project Division Company.... Issues of corporate concern were much different from those at division and project level

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 36 Data Collection

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 37 Data Collection Primitive data are collected at the project level – Time Logging Can get from cost accounting system? – Defect/Failure logging Get from trouble report system? Many details of severity, etc. can be collected – Code size -- can get from CM system? – Code Status: -- can get from CM system? Coded, walked through, debugged, ??

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 38 Don’t Reinvent! The trouble report system and the CM (configuration management) system can be used to collect much data of interest Other systems already in place may also help collect data – Cost accounting systems may be useful for collecting productivity data – Engineering systems may be useful for collecting performance data

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 39 A Typical Story Configuration management system had a “software size” counter that had been measuring software for ten years – Ten years of consistently measured historical data But the software developers decided that the way it counted software size was not accurate – So they set about defining a better way

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 40 Continuing the Story They debated for 6 months and finally compromised on a new definition of software size But the new definition of size turned out to be very difficult to count Five different groups implemented software size counters and each one gave a different answer!

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 41 Concluding the Story Finally they selected one and agreed to use it – But almost no projects did use it And meanwhile, the CM system continued to count software size the old way After two years, they found a lot of useful data in the CM system and almost none from the new size counter

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 42 Moral of the Story If you already collect data, try very hard to find a way to use it If it takes much energy to use a data collection system, people probably will not use it consistently Software developers tend not to find simple solutions to problems like this Consistency and completeness count more than perfect accuracy

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 43 Data Must be Validated It is quite common for different projects to provide inconsistent data – They do what is convenient for them – They interpret the definitions differently And it is quite common for projects to provide inaccurate or incomplete data – It tends to have low priority compared with project imperatives

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 44 The Data Collection Process Must Account for This Help people understand what to do Help people do it Verify that they did it correctly Validate the data for consistency, completeness, etc. It is not unusual to take more than a year for an organization’s data collection process to achieve wide, consistent use

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 45 Typical Graph of Collection Process Deployment

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 46 FURPS+ and similar systems can help you decide on what is a complete set Goal/Question/Metric can help you tell what will be important to collect If it won’t be used, don’t collect it – If it isn’t used, it won’t be accurate anyway – Inaccurate data may result in incorrect decisions Which Data are Worth Collecting?

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 47 Understand customer needs & satisfaction Responsiveness to customer Management & engineering effort Each of these led to questions and, eventually, to specific measurements See Module on Selecting Software Metrics HP Based Their Collection on Three Goals

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 48 Metrics to help Understand and Satisfy Customer Needs Survey results Product performance Unresolved defects and failures Calendar times for customer-visible activities (installation, delivery, etc.) Defect counts Enhancement request counts Break/Fix ratio – New problems / fixed problems

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 49 Metrics to Evaluate Responsiveness to Customer Time to fix problems Number of problems fixed/not fixed over time Both of these were characterized by severity level of the problem

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 50 Metrics to Assess Management & Engineering Effort - I Staff Months per -- – Product – Component – Task Remaining defects (by criticality) Calendar time / staff hours for -- – Phase – Activity – Type of process – Type of product

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 51 Metrics to Assess Management & Engineering Effort - II Code Stability (% changed over time) Percent Reuse (planned, actual)

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 52 Summary Database design involves many complex issues What seems logical does not always work You may need several attempts before you get something that works Collection must account for errors, inconsistencies, and reluctance to participate

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 53 Summary (continued) Each level in the organization has different data needs Most analysis may tend to occur at mid-levels in the organization Use existing data if you can Consistency is more important than perfection Data can be used to guage the impact of process changes

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 54 In “Quantitative Process Management” Using data for fact based management Advanced uses of defect containment data Advanced uses of earned value data Setting and using thresholds Statistical process control techniques

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 55 References Grady, Robert B. Practical Software Metrics for Project Management and Process Improvement. Englewood Cliffs, N.J., Prentice-Hall, Inc., ISBN ‑ Grady, Robert B. and Deborah L. Caswell, Software Metrics: Establishing a Company- Wide Program. Englewood Cliffs, N.J., Prentice-Hall, Inc., ISBN

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M32 8/20/2001Slide 56 END OF MODULE 32