1. Software Quality Assurance
Software Engineering
Software Quality Assurance

2. Objectives To motivate for software quality assurance.
To provide a historical context for software quality assurance.
To consider the benefits of formal technical reviews.
To introduce statistical software quality assurance (SSQA).

3. Steps in Project Planning
Scope—understand the problem and the work that must be done.
Estimation—how much effort? how much time?
Risk—what can go wrong? how can we avoid it? what can we do about it?
Schedule—how do we allocate resources along the timeline? what are the milestones?
Control strategy—how do we control quality? how do we control change?
Project Scope
Estimates
Risks
Schedule
Control strategy
Software
Project
Plan

4. Defining Quality
One goal of software engineering is to produce high-quality software. But what is “software quality?”
Software quality assurance (SQA) is an umbrella activity applied throughout the software process.
General engineering objective: reduce the “variation between samples.” But how does this apply to software?
Quality of Design: Characteristics that designers specify for an item. Encompasses requirements specification, analysis and design.
Quality of Conformance: The degree to which the design specifications are followed during manufacturing. Covers implementation.
Alternatively: User satisfaction = compliant product + good quality + delivery within budget and schedule.

5. McCall’s Triangle of Quality
Maintainability
Portability
Flexibility
Reusability
Testability
Interoperability P R O D U C T E V I S N P R O D U C T A N S I P R O D U C T E A I N
Correctness
Usability
Efficiency
Reliability
Integrity

6. Deriving Quality Metrics
Each of McCall’s Quality Metrics combines different quality factors.
Examples:
Correctness = Completeness + Consistency + Traceability
Portability = Generality + Hardware Independence + Modularity + Self-Documentation + System Independence
Maintainability = Conciseness + Consistency + Instrumentation + Modularity + Self-Documentation + Simplicity
“McCall’s quality factors were proposed in the early 1970s. They are as valid today as they were in that time. It’s likely that software built to conform to these factors will exhibit high quality well into the 21st century, even if there are dramatic changes in technology.”

7. Quality Concepts
Quality control: A series of inspections, reviews, tests to ensure that work products meet their requirements. Must include feedback.
Quality assurance: Analysis, auditing and reporting activities which communicate information about quality to management.
Cost of quality: all costs incurred in performing quality related activities.
Prevention Costs (quality planning, formal technical reviews, test equipment and training)
Appraisal costs (trying to gain insight into product condition = in-process inspection, equipment calibration and maintenance, testing)
Failure costs (detect an error prior to shipping = rework, repair, failure mode analysis)
External failure costs (detect an error after shipping = complaint resolution, product return and replacement, help line support, warranty work)

8. Why SQA Activities Pay Off?
Cost to find and fix a defect
100
log
10.00
scale 10
3.00
1.50
1.00 1
0.75
design
test
field
system
anal.
code
use
test

9. Software Quality Assurance
“Conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics.”
SQA
Process
Definition &
Standards
Formal
Technical
Reviews
Analysis
&
Reporting
Test
Planning
& Review
Measurement

10. SQA Definition Aspects of SQA: History:
Software requirements are the foundation from which quality is measured. Lack of conformance to requirements is lack of quality.
Specified standards must be followed.
A set of implicit requirements often goes unmentioned (e.g. ease of use). If software conforms to explicit requirements but fails to meet implicit requirements, software quality is suspect.
History:
Prior to 20thC province of the individual craftsman.
First formal quality assurance at Bell labs in 1916.
Software SQA parallels this (1950s and 1960s done by individual programmers, 1970s military introduced structured SQA)

11. SQA Group
Prepares an SQA plan for the project. This must identify evaluations, audits, reviews, standards, procedures for error reporting and tracking, and feedback.
Participates in the development of the project process.
Reviews software engineering activities to verify compliance with the defined software process.
Audits designated software work products to verify compliance with those defined as part of the software process.
Ensures that deviations in work products are documented and handled according to set procedures.
Records any non-compliance and reports to senior management.

12. Formal Technical Reviews
Formal technical reviews can be up to 75% effective in uncovering design flaws (which cover 50-65% of all errors)
“There is no particular reason why your friend and colleague cannot be your sternest critic” - Jerry Weinberg
Objectives:
To uncover errors in function, logic, or implementation for any representation of the software.
To verify that the software under review meets its requirements.
To ensure that the software has been represented according to predefined standards.
To achieve software that is developed in a uniform manner.
To make projects more manageable.

13. A Case for Reviews
Defect amplification: a model for the generation and detection of errors during a sequence of tasks.
Example:
Each test step uncovers 50% of errors but no reviews conducted in earlier stages. 10 errors in analysis lead to 12 latent defects. Assume amplification of 1:1.5.
Formal technical reviews with 50-70% efficiency in analysis design and coding reduce latent defects to 3.
Defects
Detection
Errors from previous step
Errors passed to next step
Errors passed through
Percent efficiency for error detection
Amplified errors 1:x
Newly generated errors

14. What Are Reviews? What reviews are: What reviews are not:
a meeting conducted by technical people for technical people
a technical assessment of a work product created during the software engineering process
a software quality assurance mechanism
a training ground
What reviews are not:
A project budget summary
A scheduling assessment
An overall progress report
A mechanism for reprisal or political intrigue

15. The Players review leader producer reviewer recorder
standards bearer (SQA)
producer
maintenance
oracle
reviewer
recorder
user rep

16. Conducting the Review 1. be prepared—evaluate
product before the review 2.
review the product, not
the producer 3.
keep your tone mild, ask
questions instead of
making accusations 4.
stick to the review agenda 5.
raise issues, don't resolve them 6.
avoid discussions of style—stick to technical
correctness 7.
schedule reviews as project tasks 8.
record and report all review results

17. Metrics Derived from Reviews
Inspection time per page of documentation
Inspection time per KLOC or FP
Inspection effort per KLOC or FP
Errors uncovered per reviewer hour
Errors uncovered per preparation hour
Errors uncovered per SE Task (e.g. design)
Number of minor errors (e.g. typos)
Number of major errors (e.g. non-conformance to design)
Number of errors found during preparation

18. measurement Statistical SQA • collect information on all defects
• find the causes of the
• move to provide fixes for
the process
Product
& Process
measurement
... an understanding of how
to improve quality ...

19. Aspects of Quality Software Reliability: Software Availability:
Important aspect of overall quality
The probability of failure free operation of a computer program in a specified environment for a specified time.
Where failure is non-conformance to specification.
Example: program X is estimated to have a reliability of 0.96 over 8 elapsed processing hours.
Mean Time Between Failure (MTBF) = Mean Time To Failure (MTTF) + Mean Time To Repair (MTTR).
Regarded by some as better than num. defects.
Software Availability:
Probability that a program is operating according to requirements at a given point in time.
Availability = [MTTF / (MTTF + MTTR)] * 100%.

20. Further Aspects of Quality
Software Safety:
Important in safety critical systems
Unlike software reliability, failure leads to a hazard or mishap.
Use techniques from Risk Analysis. Hazards are identified and categorized according to probability and criticality.
Example: Cruise control in a car.
Hazards: (1) Causes uncontrolled acceleration, (2) does not respond to depression of break pedal, (3) does not engage when switch is activated, (4) slowly loses or gains speed.

21. Mistake Proofing
Poka-Yoke (Mistake Proofing) developed by Shigeo Shingo at Toyota in the 1960s.
A quality assurance technique applicable to software. Automatic error detection and prevention.
Devices:
Prevention of potential quality problems before they occur.
Rapid detection of quality problems if they are introduced.
Everyday examples: seatbelt warning signal (detection) or Car unable to start when in gear (prevention).
Characteristics:
It is simple and cheap. Otherwise it will not be cost effective.
It is part of the process. Integrated into the process.
It is located near the process task where the mistakes occur. Allows rapid feedback and error correction.

22. ISO9000 Quality Standards
A quality assurance system is defined as the organizational structure, responsibilities, processes and resources for implementing quality management.
Satisfy customers expectations by meeting their specifications.
ISO9000:
A generic description of quality assurance activities applicable to any business
Adopted by many countries
Often required to supply goods and services to the government
Compliance scrutinized by third party auditors and subject to semi-annual review
ISO9001:
Standards that apply to Software Engineering