Software Quality Assurance Software Engineering Software Quality Assurance

Objectives To motivate for software quality assurance. To provide a historical context for software quality assurance. To consider the benefits of formal technical reviews. To introduce statistical software quality assurance (SSQA).

Steps in Project Planning Scope—understand the problem and the work that must be done. Estimation—how much effort? how much time? Risk—what can go wrong? how can we avoid it? what can we do about it? Schedule—how do we allocate resources along the timeline? what are the milestones? Control strategy—how do we control quality? how do we control change? Project Scope Estimates Risks Schedule Control strategy Software Project Plan

Defining Quality One goal of software engineering is to produce high-quality software. But what is “software quality?” Software quality assurance (SQA) is an umbrella activity applied throughout the software process. General engineering objective: reduce the “variation between samples.” But how does this apply to software? Quality of Design: Characteristics that designers specify for an item. Encompasses requirements specification, analysis and design. Quality of Conformance: The degree to which the design specifications are followed during manufacturing. Covers implementation. Alternatively: User satisfaction = compliant product + good quality + delivery within budget and schedule.

McCall’s Triangle of Quality Maintainability Portability Flexibility Reusability Testability Interoperability P R O D U C T E V I S N P R O D U C T A N S I P R O D U C T E A I N Correctness Usability Efficiency Reliability Integrity

Deriving Quality Metrics Each of McCall’s Quality Metrics combines different quality factors. Examples: Correctness = Completeness + Consistency + Traceability Portability = Generality + Hardware Independence + Modularity + Self-Documentation + System Independence Maintainability = Conciseness + Consistency + Instrumentation + Modularity + Self-Documentation + Simplicity “McCall’s quality factors were proposed in the early 1970s. They are as valid today as they were in that time. It’s likely that software built to conform to these factors will exhibit high quality well into the 21st century, even if there are dramatic changes in technology.”

Quality Concepts Quality control: A series of inspections, reviews, tests to ensure that work products meet their requirements. Must include feedback. Quality assurance: Analysis, auditing and reporting activities which communicate information about quality to management. Cost of quality: all costs incurred in performing quality related activities. Prevention Costs (quality planning, formal technical reviews, test equipment and training) Appraisal costs (trying to gain insight into product condition = in-process inspection, equipment calibration and maintenance, testing) Failure costs (detect an error prior to shipping = rework, repair, failure mode analysis) External failure costs (detect an error after shipping = complaint resolution, product return and replacement, help line support, warranty work)

Why SQA Activities Pay Off? Cost to find and fix a defect 100 60.00-100.00 log 10.00 scale 10 3.00 1.50 1.00 1 0.75 design test field system anal. code use test

Software Quality Assurance “Conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics.” SQA Process Definition & Standards Formal Technical Reviews Analysis & Reporting Test Planning & Review Measurement

SQA Definition Aspects of SQA: History: Software requirements are the foundation from which quality is measured. Lack of conformance to requirements is lack of quality. Specified standards must be followed. A set of implicit requirements often goes unmentioned (e.g. ease of use). If software conforms to explicit requirements but fails to meet implicit requirements, software quality is suspect. History: Prior to 20thC province of the individual craftsman. First formal quality assurance at Bell labs in 1916. Software SQA parallels this (1950s and 1960s done by individual programmers, 1970s military introduced structured SQA)

SQA Group Prepares an SQA plan for the project. This must identify evaluations, audits, reviews, standards, procedures for error reporting and tracking, and feedback. Participates in the development of the project process. Reviews software engineering activities to verify compliance with the defined software process. Audits designated software work products to verify compliance with those defined as part of the software process. Ensures that deviations in work products are documented and handled according to set procedures. Records any non-compliance and reports to senior management.

Formal Technical Reviews Formal technical reviews can be up to 75% effective in uncovering design flaws (which cover 50-65% of all errors) “There is no particular reason why your friend and colleague cannot be your sternest critic” - Jerry Weinberg Objectives: To uncover errors in function, logic, or implementation for any representation of the software. To verify that the software under review meets its requirements. To ensure that the software has been represented according to predefined standards. To achieve software that is developed in a uniform manner. To make projects more manageable.

A Case for Reviews Defect amplification: a model for the generation and detection of errors during a sequence of tasks. Example: Each test step uncovers 50% of errors but no reviews conducted in earlier stages. 10 errors in analysis lead to 12 latent defects. Assume amplification of 1:1.5. Formal technical reviews with 50-70% efficiency in analysis design and coding reduce latent defects to 3. Defects Detection Errors from previous step Errors passed to next step Errors passed through Percent efficiency for error detection Amplified errors 1:x Newly generated errors

What Are Reviews? What reviews are: What reviews are not: a meeting conducted by technical people for technical people a technical assessment of a work product created during the software engineering process a software quality assurance mechanism a training ground What reviews are not: A project budget summary A scheduling assessment An overall progress report A mechanism for reprisal or political intrigue

The Players review leader producer reviewer recorder standards bearer (SQA) producer maintenance oracle reviewer recorder user rep

Conducting the Review 1. be prepared—evaluate product before the review 2. review the product, not the producer 3. keep your tone mild, ask questions instead of making accusations 4. stick to the review agenda 5. raise issues, don't resolve them 6. avoid discussions of style—stick to technical correctness 7. schedule reviews as project tasks 8. record and report all review results

Metrics Derived from Reviews Inspection time per page of documentation Inspection time per KLOC or FP Inspection effort per KLOC or FP Errors uncovered per reviewer hour Errors uncovered per preparation hour Errors uncovered per SE Task (e.g. design) Number of minor errors (e.g. typos) Number of major errors (e.g. non-conformance to design) Number of errors found during preparation

measurement Statistical SQA • collect information on all defects • find the causes of the • move to provide fixes for the process Product & Process measurement ... an understanding of how to improve quality ...

Aspects of Quality Software Reliability: Software Availability: Important aspect of overall quality The probability of failure free operation of a computer program in a specified environment for a specified time. Where failure is non-conformance to specification. Example: program X is estimated to have a reliability of 0.96 over 8 elapsed processing hours. Mean Time Between Failure (MTBF) = Mean Time To Failure (MTTF) + Mean Time To Repair (MTTR). Regarded by some as better than num. defects. Software Availability: Probability that a program is operating according to requirements at a given point in time. Availability = [MTTF / (MTTF + MTTR)] * 100%.

Further Aspects of Quality Software Safety: Important in safety critical systems Unlike software reliability, failure leads to a hazard or mishap. Use techniques from Risk Analysis. Hazards are identified and categorized according to probability and criticality. Example: Cruise control in a car. Hazards: (1) Causes uncontrolled acceleration, (2) does not respond to depression of break pedal, (3) does not engage when switch is activated, (4) slowly loses or gains speed.

Mistake Proofing Poka-Yoke (Mistake Proofing) developed by Shigeo Shingo at Toyota in the 1960s. A quality assurance technique applicable to software. Automatic error detection and prevention. Devices: Prevention of potential quality problems before they occur. Rapid detection of quality problems if they are introduced. Everyday examples: seatbelt warning signal (detection) or Car unable to start when in gear (prevention). Characteristics: It is simple and cheap. Otherwise it will not be cost effective. It is part of the process. Integrated into the process. It is located near the process task where the mistakes occur. Allows rapid feedback and error correction.

ISO9000 Quality Standards A quality assurance system is defined as the organizational structure, responsibilities, processes and resources for implementing quality management. Satisfy customers expectations by meeting their specifications. ISO9000: A generic description of quality assurance activities applicable to any business Adopted by many countries Often required to supply goods and services to the government Compliance scrutinized by third party auditors and subject to semi-annual review ISO9001: Standards that apply to Software Engineering