Software Engineering Lecture 8: Quality Assurance

Today’s Topics l Elements of Software Quality l Quality Control & Assurance l The Cost of Quality l Defect Amplification & Removal l The Formal Review Process l Software Reliability & Safety

Software Quality Assurance l The goal of SE: “to produce high-quality software” l Question: “What is software quality?” l Poor definition: “Something you worry about after code has been generated”

SQA Principles l Quality management approach l Effective SE methods / tools l Ongoing technical reviews l Multi-tiered testing strategy l Control of documentation, updates l Standards compliance procedure l Ongoing measurement, reporting

Variation Control l One goal is to minimize variation: Among products, product versions Process results (estimated vs. actual) Behavior (defects noted over time) l Challenge: constantly evolve the product without degrading its reliability, functionality, etc.

What is Quality? l Quality of design Requirements analysis Specifications Architectural design, detailed design l Evaluating the product Does design follow requirements? Does implementation follow design? Does product meet performance goals?

Quality Control l Inspections, reviews, tests: Each module meets requirements? Feedback to development process Automated, manual, hybrid tests l Key: All modules must have defined and measurable specifications for output evaluation

Quality Assurance l Auditing and reporting functions l Goals: Provide management with information on product quality Trigger identification & resolution of quality problems

Software Quality Assurance l Conformance to: Explicitly stated functional and performance requirements Explicitly documented development standards Implicit characteristics of professional software

SQA [2] l Three important points: requirements are the foundation for measuring quality specified standards guide the development process implicit requirements (e.g., maintainability)

The Cost of Quality l Prevention costs quality planning formal technical reviews test equipment & training l Appraisal costs “first time through” single project, historical evolution equipment calibration & maintenance testing effort

The Cost of Quality [2] l Failure costs include: Internal failure costs (before release) rework & repair failure mode analysis External failure costs (after release) complaint resolution product return and replacement help line support warranty work

Cost of Correcting Defects l Relative costs to find and repair a defect increase dramatically: From prevention to detection From internal vs. external l IBM example: Prevention = ~$90/defect Field fix = ~$25,000/defect 18x more expensive!

[From SEPA 4/e]

SQA Activities l Independent person or group “...large classes of errors escape the originator more easily than … anyone else” [FRE90] l Prepare SQA Plan Evaluations to be performed Audits and reviews to be performed Standards applicable to the product Error reporting and tracking Documents produced by SQA Amount/type of feedback to developers

Types of Reviews l Informal discussion of technical problems l Formal presentation to customers / management / staff l Formal technical review (FTR) “code walkthrough” l Find errors before they become defects!

Reviews [2] l Design activities introduce 50-65% of all errors l Reviews can uncover up to 75% of all design flaws l Substantial cost savings in development and maintenance

Defect Amplification Model [from SEPA 4/e]

Defect Amplification: No Reviews [From SEPA 4/e]

Defect Amplification: w/Reviews [From SEPA 4/e]

Defect Removal Percentage of Costly Defects Is Higher (66% vs. 93%) [From SEPA 4/e] Cost Before test Much Greater (34% vs. 7%) Total Cost of Defects Much Greater

The Review Meeting l 3-5 people l Advance preparation (< 2 hours per person) Review software, read documentation l Meeting duration less than 2 hours l Constraints limit focus of each meeting (e.g., single component) l Improve likelihood of uncovering errors

Review Meeting [2] l Agenda: Introduction by developer Walk-through with Q/A Review “issues list” is created Decide whether to: accept module without modifications reject due to severe errors (repeat review at a future time) accept with revisions (no further review) Summary Report / Action Items

Review Guidelines l “Review the product, not the producer” l Set and maintain agenda l Limit debate and rebuttal l Enunciate, don’t solve problems l Take written notes l Insist on advance preparation

Review Guidelines [2] l Allocate resources for reviews l Conduct meaningful training l Review your early reviews!

Causes of Defects l Incomplete / erroneous spec l Misinterpret customer communication l Intentional deviation from spec l Programming standards not met l Error in data representation l Inconsistent module interface l Error in design logic

Causes of Defects [2] l Incomplete / erroneous testing l Inaccurate / incomplete documentation l Error in implementation of design l Ambiguous / inconsistent UI

Statistical QA l Collect info about defects l Trace defects to causes, tabulate l Prioritize (e.g. “80/20 rule”) l Move to correct problems l Create error index for each module, weighting more heavily those errors that were discovered later

Software Reliability l “Probability of failure-free operation of a module in a specified environment for a specified time” [MUS87] l How to define “failure”? l Annoying vs. catastrophic l Varying effort to fix

Reliability Measures l Mean time between failures (MTBF) l Obscure defects occur rarely l Some defects never detected l Defects that are more frequent and costly are most important

Software Safety l Identify hazards l Prioritize (probability, severity) l Design to minimize hazards l Execution modeling: Fault tree analysis [VES81] Real-time logic [JAN86] Petri nets [LEV97 l Poka-yoke devices:

Questions?