Software Quality Chapter 20-21

Software Quality  How can you tell if software has high quality?  How can we measure the quality of software?  How can we make sure software has high quality?

Perspective on quality  Customer  system not crashes  system follows documentation  system is logical and easy to use  Developer  system is easy to change  system is easy to understand  system is pleasant to work on

Characteristics of Software Quality  Some external characteristics that the user is aware of:  Correctness  Usability  Efficiency  Reliability  integrity  extent to which it allows unauthorized access to data  Adaptability  Accuracy  Robustness

Characteristics of Software Quality  Some internal characteristics of software that are code- centered:  Maintainability  Flexibility  Portability  Reusability  Readability  Testability  Understandability

Total Quality Management  Factories  Goal is for every item coming off the assembly line to be perfect  Management, production, engineering, QA  Everyone is involved in quality  Develop a reliable, repeatable process  Continuously improve the process

Failure vs. flaw  Failure - program didn’t work right  Flaw - mistake in the text of the program  Failure analysis - what flaw caused this failure?  Flaw analysis - what is wrong with our process that allowed this flaw to be created and not detected?

Failure costs  Internal  rework  repair  failure analysis  External  resolving complaints  returning and replacing product  help line

Prevention costs  Prevention  planning  managing and collecting information  reviews  Appraisal  inspection  Testing

Cost of fixing an error DesignCodeDev. Test System Test Field operation 3-6 times Req. 10 times times times times 10 1 time

Johnson’s Law If you don’t test for it, your system doesn’t have it. Is it easy to use? Easy to maintain? Does it crash? Does it match the documentation? Does it make customers happy?

Ways not to improve quality  Say “Be more careful!”  Say “Quality is important.”  Find out whose fault it is and fire him.

How to improve quality  Measure and compare  Determine root cause of problems  Create ways to eliminate problems

 If you don’t see it, it doesn’t exist  Measure quality over time (metrics)  Display in a public place  Make quality goals, then check to see if you meet them.

How to appraise quality  Requirements  reviews by customers  prototyping  Analysis and design models  formal reviews, inspections  Current system  bug reports  user tests  surveys

Bug tracking  Keep track of  who reported the bug (the failure)  description of the failure  severity  the flaw that caused this failure  who is repairing it  the repair

Bug tracking  Use information about failures to estimate reliability  Compare  critical nature of failure  iteration failure discovered  module that had the flaw

Use quality information to make decisions  “Must repair all level 1 failures before shipping”  “Half of all level 1 and 2 failures in the alpha release were in the Call Processing module; we should rewrite it.”  “Half of all level 1 and 2 defects found in the design reviews were in Call Processing; we should rewrite it.”

Bug tracking  Discover the flaw (defect) that caused each bug  Categorize flaws  Look at categories with the most flaws and improve your process to eliminate them.

SQA Manager  Responsible for SQA policy  Develops testing plan  Checks that plan is being followed  Develops review process  Trains reviewers  Monitors reviews  Develops customer survey plan  …

Technical Reviews  A way to evaluate the quality of requirements, designs, and software  A way to improve the quality of requirements, designs, and software  A way to educate new developers and ensure that developers are consistent  Proven to be cost-effective!

Main goal: evaluate quality  Produce a report describing  potential problems  summary of overall quality  pass/fail  Evaluated by expert outsiders  must know enough  shouldn’t know too much

Secondary goal: improve quality  Find flaws  Enforce standards  Improve standards  Provide feedback to management

Review methods  The most important step to improve the software engineering performance  Various review methods  Inspection  Walk-throughs  Personal reviews

Inspections  A structured procedure for the team review of a software product  Has a formal structure, and each participant has a defined role  Typical inspection process  Preparation  Inspection meeting  Repair and report  To be more effective, inspections are measured, a report is produced, and the action items are tracked to closure.

Walk-Throughs  A less formal process that usually follows a presentation format  A developer walks through how the program performs, while the audience raises issues and asks questions.  Generally lack of advance preparation or follow-up

Personal Reviews  You examine your own products.  To find and fix as many defects as possible before implementing, inspecting, compiling, or testing the program  As the PSP data will show, a little time spent on reviewing code can save the much longer amount of time that would be spent on debugging and fixing during compile and test.

Code Reviews are More Efficient than Testing (1)  In reviews, you find the defects directly.  When you review a program, you know where you are and the results its logic is supposed to produce.  You establish logical relationships and construct a mental context of the program’s behavior.  In testing, you get only symptoms.  You start with some unexpected system behavior.  You then spend a lot of time figuring out what caused those symptoms, i.e. debugging.

Code Reviews are More Efficient than Testing (2)  A debugger's principal advantage is that it helps you to step through the program logic and check the important parameter values.  This process is effective only if you know what the parameter values are supposed to be.

Review Principles  Establish defined review goals.  Follow a defined review process.  Measure and improve your review process.

Establish Review Goals  To find and fix all defects before the first compile or test  When engineers start doing reviews, they find only about 1/3 to 1/2 of defects.  By analyzing the data on reviews and making appropriate changes in review process, the rate can be improved significantly.  With care can practice, some find they can reach 80%, or even 95%.

Follow a Defined Review Process  Code review script  Code review guideline and checklist (C++)

Measure and Improve Your Review Process  To improve review quality  Generally, a high-quality review is one that finds the greatest number of defects in the least amount of time.  Need to measure the time spent on review, and track all defects found in review and later

Review Measures

Who founds more defects?

Review Measures Four explicit review measures  The size of the program being reviewed  LOC  If no code available, use text lines or pages of design, or estimated LOC.  The review time in minutes  The number of defects found  The number of defects in the program that were later found  these are called escapes

Derived Review Measures Several measures can be derived from the basic measures.  The review yield  the percentage of defects in the program that were found during the review  The defect found per KLOC of design or code reviewed  The defects found per hour of review time  The LOC reviewed per hour  Defect removal leverage (DRL), or the relative rate of defect removal for any two process phases

Review Yield  Yield  Cannot be precisely calculated until the reviewed has been extensively tested and used  Still able to make useful early approximations; upper bound  A high yield would be good and a low yield poor.

Result of review  Review summary  who, what, when and the conclusion  Issues list  Can result in more detailed reports  Give priority to issues  Can be disagreement on issues  Most issues are about product, but can also be about process or standards

In real life  Usually there are several meetings until all issues are resolved  Project has a policy that determines what would be reviewed  “Passing review” is a measure of progress  Reviews improve checklists, not just the product under review

Summary  Reviews are a key quality control technique  Use reviews to improve your process, not just your product