PSP Quality Strategy [SE-280 Dr. Mark L. Hornick 1

[ 2 What is software quality? Basic definition: Meeting users' needs What about “wants”? True needs may be hard to establish Hierarchy of needs Do required tasks Be Usable & convenient  Install quickly & easily  Run consistently Economical Timely Dependable  No destructive or unexpected behavior

[SE-280 Dr. Mark L. Hornick 3 How do you measure quality?

[SE-280 Dr. Mark L. Hornick 4 In the PSP, defects are the basic quality measure; is this reasonable? On the one hand, defects are only important to the customer or user if they cause visible adverse effects. Affect operations Cause inconvenience Cost time or money Result in a loss of confidence However, defects have a big impact on the software development process, costing money and taking time that could be better spent in adding value for those customers and users.

[SE-280 Dr. Mark L. Hornick 5 Why focus on defects? Defects have a big impact on the software development process: Defects must be managed, costing money and taking time that could be better spent in adding value for those customers and users. With poor quality, there is no time to address other quality issues such as: Ease of installation Safety Recovery & robustness Usability & convenience

Generally, defects are more expensive to remove if they are not found until later in the software development process.

[SE-280 Dr. Mark L. Hornick 7 CMM – Capability Maturity Model CMM is an assessment of an organization’s software capabilities; an SEI-developed metric with 5 levels: Level 1 = “no defined process” Level 5 = “process exhibiting continuous improvement”

[SE-280 Dr. Mark L. Hornick 8 A mature organization at CMM level 5: Creates and reviews requirements Does design inspections (design is reviewed by a group) Does code inspections (code is reviewed by a group) Does a lot of testing And still leaves about 1 defect/KLOC

[SE-280 Dr. Mark L. Hornick 9 With the growth of software systems, we are facing a significant quality challenge What if we want 1-10 defects per MLOC? Level 5 CMMI is about 1000 defects/MLOC! You probably can't "test in" this quality level And, probably can't "inspect it in" either: Assume 1000 LOC  30 pages of listings Then, 1 MLOC  30,000 pages Can we inspect 30K pages well enough to leave only 1-10 defects??

[SE-280 Dr. Mark L. Hornick 10 Quality management (defect removal and prevention) is a critical part of the PSP. Q. Who injects software defects? A. Individual software engineers! Fundamental Axiom: The personal process of the individual software engineer is the place to remove defects, determine their causes, and learn to prevent them.

[SE-280 Dr. Mark L. Hornick 11 Remove Quality Metrics: Yield is a measure of how well we find and remove defects. Development phase Yield is the % of existing defects we remove in a phase. Generally, defects are injected in development phases like Design and Coding, but it is possible for defects to be injected or removed in any phase.. Inject Development phase RemoveInject Development phase RemoveInject Escapes are defects that pass from one phase to the next.

Process Yield is the composite yield for all phases before Test. The process yield (or just "yield") target is about 70%; that is, no more than 30% of defects should be left to be found during testing, since testing is expensive!

[SE-280 Dr. Mark L. Hornick 13 There are different kinds of costs associated with quality. Appraisal costs are the costs of inspecting for defects, (in the PSP, Design and Code review times). Failure costs result from rework, repair, and scrap in the PSP, this is time you spend in Test Both of these metrics are costs of quality; that is, the “price” you pay to ensure the quality of your code.

The relationship between these two kinds of quality costs are described by a ratio: In the PSP, the A/F ratio should be about 2.0. A/FR tracks ratio of Appraisal cost (review time) Failure cost (test time)

SE-280 Dr. Mark L. Hornick 15 If measured in many organizations, A/FR would be near zero! but (mature) PSP processes typically exceed 2.0 High A/FR is associated with low test defect density and high product quality

The five PQI components are calculated in the following manner. Note that all PQI values are limited to a range of 0.0 … 1.0 Target value: < 10 defects/KLOC A+M Target value: < 5 defects/KLOC A+M Overall PQI is computed by taking the product of the components. Target: > 0.4

Here is an example of a PQI calculation. PQI Raw PQI Adj PQI DesignCodeTime DesignReviewTime CodeReviewTime CompileDefects UnitTestDefects Total PQI Design Time125 Design Review Time38 Code Time67 Code Review Time32 Compile Defects/KLOC (A+M)0 UT Defects/KLOC (A+M)6.1 PQI DesignCodeTime PQI Code Review Time PQI Compile Defects PQI Unit Test Defects PQI Design Review Time Note that Adjusted PQI values are limited to a range of 0.0 … 1.0

Results of the PQI calculations PQI Raw PQI Adj PQI DesignCodeTime DesignReviewTime0.61 CodeReviewTime0.96 CompileDefects UnitTestDefects0.90 Total PQI0.52 Design Time125 Design Review Time38 Code Time67 Code Review Time32 Compile Defects/KLOC (A+M)0 UT Defects/KLOC (A+M)6.1 PQI DesignCodeTime PQI Code Review Time PQI Compile Defects PQI Unit Test Defects PQI Design Review Time

The process quality index (PQI) is a compact representation of five quality measures. Text, page 150 The value at each vertex is 1.0 (good), while the center is zero (bad).

The Process Dashboard tool computes a number of quality metrics. Start looking at your own quality metrics!

What about the use of automated defect-finding tools? If you have a tool that can remove 100% of a certain type of defect, use it as early as possible to get rid of them! If you have a tool that can remove about 10% of a certain type of defect, it is not very effective, but also relatively harmless since you won’t rely on it. If you have a tool that can remove 90% of a certain type of defect, it can be dangerous because you will tend to believe that it found them all, and not aggressively seek out the remaining ones.

[SE-280 Dr. Mark L. Hornick 22 Conclusion Inspections and Testing will not catch enough defects It is more much more expensive to inspect and test than to review Personal Design and Code reviews are key to high quality software