1. SOFTWARE QUALITY ASSURANCE
SOFTWARE ENGINEERING
SOFTWARE QUALITY ASSURANCE

2. LEARNIING OBJECTIIVES
1. Understand the quality management process and the central
process activities of quality assurance, quality planning and
quality control.
2. Understand the importance of standards and metrics in the
quality management process.
3. Know what is software configuration management, why it is
important and what are its major activities.
4. Understand the principles of software process improvement and
why process improvement is worthwhile.

3. SOFTWARE QUALITY ASSURANCE : OUTLINE
10, 12
SOFTWARE QUALITY ASSURANCE : OUTLINE
Software Quality Assurance
Life Cycle Role
Purpose and Impoetance
Techniques : Standards and Metrics
Achieving Software Quanlity
Achieving Product Quality
Design Quality Metrics
Formal Approaches
Program Quality Metrics
Achieving Project Quality
Reviews
Software Configuration Management
Achieving Process Quality
ISO 9001/9000-3
SEI Process Capability Maturity Model
Achieving People Quality
The People Capability Maturity Model

4. SOFTWARE QUALITY ASSURANCE: LIFE CYCLE ROLE

5. THE PURPOSE SOFTWARE QUALITY ASSURANCE
Quality assurance consists of those procedures, techniques, and tools applied by professionals to ensure that a product meets or exceeds pre-specified standards during it’s development cycle. E.H. Bersoff
quality assurance Defines organizational standards that lead to
high quality software.
quality planing Selects appropriate standards and tailors them
to a specific software product.
quality control Ensures standards are followed.
Continuous quality improvement should be the overall goal.
anyone (and everyone) involved in software development has responsibility for quality
we want quality to be a natural outcome of software engineering

6. WHY SQA ACTIVITIES PAY OFF
cost to find
and fix a defect 1 10
100
Field
use
80-130
log
scale
System
test
13.00
SQA pays off here because
defects are cheap to fix
cost of fixing an error increases exponentially the later in the system life cycle it is uncovered
“pay a little now or pay much more later”
Test
4.00
Code
2.00
Reqmts
1.00
Design
1.30

7. SQA — AN UMBRELLA ACTIVITY
Standards
and
Procedures
Metrics
and
Measurement
The foundation of quality assurance.
Methods
and
Tools
Testing
Software Quality
Assurance Techniques
Software
Configuration
Management
Formal
Technical
Reviews

8. SOFTWARE QUALIITY ASSURANCE: TECHNIQUES
Includes all techniques used to improve the quality of the software:
1. developing and applying standards
To ensure repeatability of the development process
2. developing and using metrics and measurement procedures
To meet designgoals and improve the software development process
3. using appropriate methods and tools for system development
To achieve a high quality system
4. doing formal technical reviews at each step
To uncover quality problems and to sign-off
5. using software configuration management and change control
To ensure changes are managed and controlled
6. performing multi-tiered testing
To ensure effective defect detection. (BUT testing is not a panacea)
SQA—An Umbrella Activity
1. methods and tools for System Requirements Capture, System Analysis, System Design, Implementation and Testing
to help ensure we achieve a high quality system
2. formal technical review procedures at each step
to help uncover quality problems
sign-off at each step
3. multi-tiered testing
to help ensure effective error detection
not a panacea! cannot uncover all errors
4. software configuration management and change control
to help ensure changes are managed and controlled
5. standards compliance
may not exist! if they exist, should be followed (or changed)
may be mandated by law/contract (e.g., military)
6. metrics and measurement procedures
establish and monitor software metrics
historical record, institutional memory

9. Standards Handbook needed!
SQA : STANDARDS
Why are software standards important for SQA ?
1. They encapsulate best (or most appropriate) practices
Acquired after much trial and error ® helps avoid previous mistakes
2. They provide a framework around which to implement SQA process
Ensures that best practices are properly followed
3. They assist in ensuring continuity of project work
Reduces learning effort when starting new work
product standards: define the characteristics all product artifacts should exhibit so as to have quality
process standards: define how the software process should be conducted to ensure quality software
Standards
Handbook
needed!
Each project needs to decide which standards should be: ignored; used as is; modified; created.

10. SQA TECHNIQUES : METRICS
A metric is any type of measurement that relates to a software system product , process or related artifact
Why are metrics important for Software Quality Assurance?
1. Metrics can be used to control (i.e., plan and manage) the
development process (e.g., effort expended, elapsed
time,budget spent, etc.).
2. Metrics can be used to predict an associated product quality
(e.g., cyclomatic complexity can predict ease of maintenance).
Metrics are the only objective way to measure quality attributes of software; otherwise it is all opinion and guess work.

11. ACHIEVING SOFTWARE QUALITY
set of activities
How can
we achieve
software quality?
Process
measurement
&
feedback
template
software engineers
set of artifacts
People
Project
Product
participate
result
education
&
training
management
&
monitoring
testing
&
measurement

12. ACHIEVING SOFTWARE QUALITY(cont’d)
1. We have a set of standards and quality attributes that a software product must meet.
There are design goal to achieve.
key
point
2. We should be able to measure the quality of a software product.
There is a way to determine how well the product conforms to the design goal.
3. We should track the values of the quality attributes.
So that it is possible to assess ,over time, how well we are doing in achieving the design goal.
4. We use information about software quality to improve the quality of future software products.
There is feedback into the software development process.

13. ACHIEVING PRODUCT QUALITY : DESIGN GOALS
Recall that a design goal is an (external) attribute that we want
the system to have such as:
– safety – understandability – portability
– security – testability – usability
– reliability – adaptability – reusability
– resilience – modularity – efficiency
– robustness – maintainability – learnability
Usually, we can only asses whether a system has one of these
attributes (i.e., meets its design goals) after it is completed!
However, we would like to assess whether we are achieving a
design goal during the system’s development? How to do it?
1. Operational characteristics
correctness - the extent to which a program satisfies its specification and fulfills the customers mission objectives
reliability - the extent to which a program can be expected to perform its intended function with required precision
efficiency - the amount of computing resources and code required by a program to perform its function
integrity - the extent to which access to software or data by unauthorized persons can be controlled
usability - the effort required to learn, operate, prepare input, and interpret output of a program
2. Changeability characteristics
maintainability - the effort required to locate and fix an error in a program
flexibility - the effort required to modify an operational program
testability - the effort required to test a program to ensure that it performs its intended function
3. Adaptability characteristics
portability - the effort required to transfer the program from one hardware and/or software system environment to another
reusability - the extent to which a program (or its parts) can be reused in other applications
interoperability - the effort required to couple one system to another
can only measure subjectively  use a checklist to “grade” each attribute
Try to use internal attributes to assess (predict)
whether the system will meet its design goals.

14. ACHIEVING PRODUCT QUALITY : DESIGN GOALS(cont’d)
Examples of possible relationships between external andinternal attributes:
External(design goal)
maintainability
reliability
portability
usability
Internal(measurable)
# of parameters
cyclomatic complexity
lines of code
# of error messages
length of user manual
Problems: 1. hard to formulate and validate relationships between internal and external attributes
2. software metrics must be collected, calibrated, interpreted

15. ACHIEVING PRODUCT QUALITY: DESIGN QUALITY METRICS
For a design component, a key design goal is maintainability.
For design components maintainability is related to:
cohesion - how closely related is the functionality of the component?
coupling - how independent is the component?
understandability - how easy is it to understand what the component does?
adaptability - how easy is it to change the component?
Problem: most of these cannot be measured directly, but it is reasonable to infer that there is a relationship between these attributes and the “complexity” of a component
measure complexity
How?

16. ACHIEVING PRODUCT QUALITY: DESIGN QUALITY METRICS
1) Structural fan-in/fan-out
fan-in – the number of calls to a component by other components
fan-out – the number of components called by a component
high fan-in => high coupling
high fan-out => calling component has high complexity
2) Informational fan-in/fan-out
consider also the number of parameters passed plus access to shared data structures
complexity = component-length x (fan-in x fan-out)2
It has been validated using the Unix system
It is a useful predictor of effort required for implementation

17. ACHIEVING PRODUCT QUALITY: DESIGN QUALITY METRICS
3) IEEE Standard
looks at: subsystem properties (number of subsystems and degree of coupling) database properties (number of attributes and classes)
Compute a design structure quality index—DSQI ® (0-1)
Used to compare with past designs; if DSQI is too low, further design work and review may be required
Measuring Software Quality—Metrics
1. IEEE Standard
looks at subsystem properties (coupling) and data properties (size)
design structure quality index (DSQI)
ranges from 0 to 1 (1 is the best)
compare to previous values; if too low  review design
looks at number of changes made in each release (subsystems changed, added, deleted)
software maturity index (SMI)
as SMI approaches 1, product begins to stabilize
We can also consider changes made throughout the lifetime of the software and compute how stable the product is (i.e., how many changes have been made in subsystems in the current release)
Define a software maturity index—SMI ® (0-1)
As SMI approaches 1, the product begins to stabilize

18. This slide is for information only – you are not required to know this!
IEEE STANDARD
S1 = the total number of subsystems defined in the program architecture
S2 = the number of subsystems whose correct function depends on the source of data input or that produces data to be used elsewhere
S3 = the number of subsystems whose correct function depends on prior processing
S4 = the number of database items (includes data objects and all attributes that define objects)
S5 = the total number of unique database items
S6 = the number of database segments (different records or individual objects
S7 = the number of subsystems with a single entry and exit

19. This slide is for information only – you are not required to know this!
IEEE STANDARD
Program structure: D1 = 1 if the architecture was developed using a distinct method D1 = 0 otherwise
Subsystem independence: D2 = 1 - (S2/S1)
Subsystems not dependent on prior processing: D3 = 1 - (S3/S1)
Database size: D4 = 1 - (S5/S4)
Database compartmentalization: D5 = 1 - (S6/S4)
Subsystem entrance/exit characteristic: D6 = 1 - (S7/S1)
DSQI = wiDi wi = relative weighting of each Di

20. SMI = [MT - (Fa + Fc + Fd)] MT
This slide is for information only – you are not required to know this!
IEEE STANDARD
SMI = [MT - (Fa + Fc + Fd)] MT
MT = the number of subsystems in the current release
Fc = the number of subsystems in the current release that have been changed
Fa = the number of subsystems in the current release that have been added
Fd = the number of subsystems in the preceding release that were deleted in the current release

21. ACHIEVING PRODUCT QUALITY: PROGRAM QUALITY METRICS
For an implementation component, the key quality attributes are reliability and difficulty of implementation.
Some approaches to measure reliability and/or difficulty: a) Halstead’s Software Science
Looks at operators and operands in an implementation component and calculates values for component volume, V, component difficulty, D, and effort, E, required to implement the component.
n1 = number of unique operators in a component
n2 = number of unique operands in a component
N1 = the total number of operators
N2 = the total number of operands
L = N1+ N2 (component length)
V = L * log2(n1 + n2) (component volume in bits)
D = (n1/2) * (N2/n2) (difficulty of implementing the component)
E = V * D (effort required to implement the component)

22. ACHIEVING PRODUCT QUALITY: PROGRAM QUALITY METRICS
2) McCabe’s Complexity Metric
Looks at control flow in a component.
Cyclomatic Complexity ® measures component’s logical complexity
an indication of the testing difficulty of a component
Studies have shown a distinct relationship between the Cyclomatic Complexity of a component and the number of errors found in the source code, as well as the time required to find and fix errors.
3) Other program quality metrics
line of code (LOC) – length of identifiers
depth of conditional nesting
Standards can be established to avoid complex components and/or highlight problem components.

23. ACHIEVING PRODUCT QUALITY: FORMAL APPROACHES
a) Proving programs/specifications correct
Logically prove that requirements have been correctly transformed into programs
(e.g., prove assertions about programs)
b) Statistical Quality Assurance
Categorize and determine cause of software defects
Use rule  80% of defects can be traced to 20% of causes
Isolate and correct 20% of the causes
effort directed to things that cause the majority of defects
c) The Cleanroom Process
A combination of above two approaches
Formal Approaches to SQA
1. Proving programs/specifications correct
logically prove that requirements have been correctly transformed into programs
2. Statistical Quality Assurance
categorize software defects and determine their cause
isolate 20% of causes (vital few)  rule
80% of defects can be traced to 20% of all possible causes
correct problems which cause 80% of defects (i.e., 20% of causes)
can also use this technique to determine which phase in development process contributes most errors
3. The Cleanroom Process
combination of 1. and 2.

24. ACHIEVING PROJECT QUALITY: REVIEWS
The principal method of validating the quality of a project.
requirements
walkthroughs
Leads to early
discovery of defects
Requirements
capture
analysis
walkthroughs
Analysis
design
walkthroughs
Requirements, Analysis and Design
introduce 50-60%
of all defects.
Design
Ensuring Quality—Reviews
“filter” for SE process
consider them similar to debugging
each stage/phase should have its own review
“blind” to our own mistakes
get opinion of others (two heads are better than one!)
“walkthroughs” - lead analyst/designer walks through the documentation/code
benefits?
early discovery of software defects
analysis and design activities introduce between 50-60% of all errors during development
but, formal technical reviews can uncover 75% of these errors
code
walkthroughs
Implementation
test plan
review
Formal technical reviews can uncover 75% of these!
Testing

25. ACHIEVING PROJECT QUALIITY:SOFTWARE CONFIGURATION MANAGEMENT (SCM)

26. SOFTWARE CONFIGURATION MANAGEMENT (SCM)
10.2
SOFTWARE CONFIGURATION MANAGEMENT (SCM)
Software Configuration Management is an activity executed throughout the system life cycle to control the change to the products and life cycle artifacts.
To what do we want to control changes?
Software Configuration Management
• plans • programs
• specifications • documents/manuals
• procedures • data
identify
support
control
report
audit
configuration item: an artifact to which we want to control changes

27. SCM ITEM : ITEM IDENTIFCATION AND DESCRIPTION
10.3.1
SCM ITEM : ITEM IDENTIFCATION AND DESCRIPTION
Each configuration item must be identified and described
a unique name
configuration item type
project identifier
change and/or version information
resources the configuration item provides or requires
pointer to actual configuration item
The configuration items also must be organized (i.e., need to define the relationships that exist between configuration items)
object diagram <part-of> domain model
domain model <related-to> use-case model
This is
metadata
about
configuration
items
This information is use to define and construct a software library (database) that stores, manages and tracks configuration items.

28. SCM ITEMS : VERSIONS
A configuration item usually evolves throughout the software engineering process (i.e., it will have several versions)
An evolution graph can be used to describe a configuration item’s change history
version
Configuration itemk is obtained by modifying configuration itemi
usually a result of bug fixes, enhanced system functionality, etc.
itemk supercedes original itemi; created in a linear order
branch
A concurrent development path requiring independent configuration management
variant
Different configurations that are intended to coexist
e.g., different configurations depending on operating system type
Oracle for Windows Oracle for Linux

29. SCM ITEMS : EVOLUTION GRAPH
version
1.0
branch line
branch line
2.0
2.0.1
2.1
2.0.2
2.2
3.0
parallel development
mainline development

30. SCM SUPPORT : SOFTWARE LIBRARY
10.3.5
SCM SUPPORT : SOFTWARE LIBRARY
A software library provides facilities to store, label, identify versions and track the status of the configuration items
Used for
used by a
single developer
developer’s
workspace
everyday
development
check-in/check-out of configuration items
Stores & tracks
used by
other developers
master
directory
promotions
after SQA
Stores & tracks
software
repository
for users
releases

31. 10.3.2
SCM : BASELINES
A baseline is a time/phase in the software development (usually a project milestone) after which any changes must be formalized (e.g., go through a formal change control procedure).
To become a baseline, the configuration item must first pass a set of formal review procedures (e.g., formal code review, documentation review, etc.).
It then becomes part of the project software library.
After this a “check-out” procedure is applied to the item (i.e., access to and change of the configuration item is controlled)
Any modified configuration item must again go through a formal review process before it can replace the original (baselined) item.
Requires a version control mechanism.

32. SCM BASELINES
Requirements
gathering
Requirements Specification
Analysis
Analysis Specification
Design
Design Specification
Implementation
Source code, manuals
Testing
Test Plans/Procedures/Data
Operational System

33. CHANGE Uncontrolled change rapidly leads to chaos ! SCM CHANGE CONTROL
STOP
change request submitted by users/developers
change control authority evaluates, decides and issues an engineering change order if approved
configuration item is checked-out, changed, checked-in after SQA, and version controlled
configuration item made available for use (promotion; release)
CHANGE
Uncontrolled
change rapidly
leads to chaos !

34. SCM CHANGE CONTROL
change request from user/developer
change control authority evaluates and decides
request is queued for action change request is denied
assign individuals to configuration items user is informed
“check-out” configuration items
make changes
review (audit) the change
“check-in” the configuration items that have been changed
establish a baseline for testing
perform quality assurance and testing activities
“promote” changes for inclusion in next release
rebuild appropriate version of software
review (audit) the change to all configuration items
include changes in release
distribute the release
can be applied to internal requests (i.e., user = developer) and external requests (i.e., user = end-user)

35. SCM : AUDIT AND STATUS REPORTING
Audit: ensures that changes have been properly implemented.
Have the proper steps and procedures been followed? ® checklist
Usually done by Quality Assurance (QA) group if SCM is a formal activity.
Status reporting: keeps all parties informed and up-to-date on the status of a change.
Status reporting is a communication mechanism among project members to help keep them coordinated.
Status reporting allows management/users to determine who made
what changes, when and why.

36. SCM BENEFITS Reduces the effort required to manage and effect change
Resulting in improved productivity
Leads to better software integrity and security
Resulting in increased quality
Generates information about the process
Resulting in enhanced management control
Maintains a software development database
Resulting in better record keeping and tracking

37. ACHIEVING PROCESS QUALITY
Does process quality product quality?
Unlike manufactured products, software development has some unique factors that affect its quality:
software is designed, not manufactured
software development is creative, not mechanical
individual skills and experience have a significant influence
external factors (application novelty, commercial pressure)
Software development processes are organization specific
Sometime people&technology may be more important than process
Insufficient resources will always adversely affect quality

38. ACHIEVING PROCESS QUALIITY: ISO 9001/9000-3

39. ACHIEVING PROCESS QUALITY: ISO 9001/ 9000-3
Specific a generic
quality model
. . .
ISO 9000/9001
quality model
Organization
Project 1
quality plan
Project 2
Organizational
quality process
Project quality
management
instantiated as
documents
supports
is used
to develop
Focus is on
process
management
The ISO quality
Model need to be
customized for
a particular
organization
certification is easy; can be a marketing ploy

40. This slide is for information only – you are not required to know this!
ISO 9001/ STANDARD (1/3)
A. Quality system framework
1. Management responsibility
quality policy defined, documented, understood, implemented and maintained
responsibilities and authorities for all personnel defined
in-house verification resources defined, trained and funded
a designated manager ensures that the quality program is implemented and maintained
2. Quality system
requires a documented quality system be established
should be an integral process throughout the entire life cycle
3. Internal quality system audits
requires audits be planned and performed
results communicated to management
deficiencies corrected
4. Corrective action
requires causes of non-conforming product be identified
potential causes of non-conformance eliminated
procedures changed as a result

41. This slide is for information only – you are not required to know this!
ISO 9001/ STANDARD (2/3)
B. Quality Life Cycle Activities
1. Contract review
contracts are reviewed to determine whether the requirements are adequately defined, agree with the bid, and can be implemented
2. Purchaser’s requirements specification
procedures to identify and collect purchaser’s requirements
3. Development planning
production processes are defined and planned
4. Quality record
quality assurance process is planned
5. Design and implementation
procedures to control and verify the design
includes planning design
activities, identifying inputs and outputs, verifying the design and controlling design changes
6. Testing and validation
requires systems/products to be tested and validated before use
records of testing are kept
7. Acceptance
procedures for acceptance should be established
8. Replication, delivery and installation
procedures for the above should be established and maintained
9. Maintenance
requires that maintenance activities be performed as specified

42. This slide is for information only – you are not required to know this!
ISO 9001/ STANDARD (3/3)
C. Quality System Supporting Activities
1. Configuration management
process of development and maintenance should be documented and controlled
2. Document control
distribution and modification of documents should be controlled
3. Quality records
quality records should be collected, maintained and dispositioned
4. Measurement
where appropriate, adequate statistical techniques should be identified and used to verify the acceptability of process capability and product characteristics
5. Rules, practices and conventions
are in place and followed
6. Tools and Techniques
are applied appropriately to support the development process
7. Purchasing
purchased products conform to their specified requirements
8. Included software product
should be verified and maintained
9. Training
training needs should be identified and training provided since related tasks may require qualified personnel
training records should be maintained

43. ACHIEVING PROCESS QUALITY: THE SEI CAPABILITY MATURITY MODEL (CMM)
12.3
ACHIEVING PROCESS QUALITY: THE SEI CAPABILITY MATURITY MODEL (CMM)
SEU-CMM intended to help a software organization improve their software development processes.
Level 1 Organization: Initial process (ad hoc)
no formal procedures, no cost estimates, no project plans, no management mechanism to ensure procedures are followed
Level 2 Organization: Repeatable process (intuitive)
basic project controls; intuitive methods used
Level 3 Organization: Defined process (qualitative)
development process defined and institutionalized
Level 4 Organization: Managed process (quantitative)
measured process; process database established
Level 5 Organization: Optimizing process
improvement feedback; rigorous defect-cause analysis and prevention
Focus is on
process
improvement
intended to help a software organization improve their software development processes
evolutionary path from ad hoc, chaotic process to mature, disciplined software process
each maturity level is a well-defined evolutionary plateau
Level 1: Initial process (ad hoc)
ad hoc and occasionally even chaotic
few processes defined
success depends on individual effort and heroics
Level 2: Repeatable process (intuitive)
basic project management processes established to track cost, schedule and functionality
process disciplines in place to repeat earlier successes on projects with similar requirements
Configuration Management
Quality Assurance
Subcontract Management processes
Project Tracking and Oversight in place
Project Planning
Requirements Management

44. CAPABILITY MATURITY MODEL (CMM)
This slide is for information only – you are not required to know this!
CAPABILITY MATURITY MODEL (CMM)
Key Processes in Place
Level 1: Initial process
Level 2: Repeatable process
Requirements Management
Software Project Planning
Software Project Tracking & Oversight
Software Subcontract Management
Software Quality Assurance
Software Configuration Management
Level 3: Defined process
Organization Process Focus
Organization Process Definition
Training Program
Integrated Software Management
Software Product Engineering
Intergroup Coordination
Peer Reviews
Level 4: Managed process
Quantitative Process Management
Software Quality Management
Level 5: Optimizing process
Fault Prevention
Technology Change Management
Process Change Management

45. ACHIEVING PROCESS QUALITY — NOTES
It is possible for a Level 1 organization to receive ISO certification!
A Level 3 organization would have little difficulty in obtaining ISO 9000 certification.
A Level 2 organization would have significant advantage in obtaining ISO 9000 certification.
excellent software developers (e.g., space shuttle) attain around level 3-4 only

46. ACHIEVING PEOPLE QUALITY: PEOPLE CAPABILITY MATURITY MODEL (PCMM)
Intended to improve knowledge and skill of people.
Level 1 – Initial
no technical or management training provided; staff talent not a critical resource; no organizational loyalty
Level 2 – Repeatable
focus on developing basic work practices; staff recruiting, growth and development important; training to fill skill “gaps”; performance evaluated
Level 3 – Defined
focus on tailoring work practices to organization’s business; strategic plan to locate and develop required talent; skills-based compensation
Level 4 – Managed
focus on increasing competence in critical skills; mentoring; team-building; quantitative competence goals; evaluation of effectiveness of work practices
Level 5 – Optimizing
focus on improving team and individual skills; use of best practices
Focus is on
people
improvement

47. SQA — THE BOTTOM LINE
an organization should have a quality manual which documents its quality assurance procedures
each project should have a quality plan which sets out the quality attributes that are most important for that project and how they will be assessed
an organization should have established standards for documentation
mechanisms (processes) should be established that monitor compliance with all quality requirements of the organization
reviews are the primary means of carrying out quality assurance
metrics can be used to highlight anomalous parts of the software which may have quality problems

48. SUMMARY — SOFTWARE QUALITY
Quality software does not just happen!
Quality assurance mechanisms should be built into the software development process
Developing quality software requires:
Management support and involvement
Gathering and use of software metrics
Policies and procedures that everyone follows
Commitment to following the policies and procedures even when things get rough!
Testing is an important part of quality assurance, but its not all there is to obtaining a quality software product.