1. Software Testing & QA (VI)
Kerry Zhu

2. SUMMARY SO FAR Part VI – The Future
Part I- The Big Picture - an Introduction
Part II- Testing Fundamentals - heart of course
Part III- Applying Your Testing Skills
Part IV- Supplementing Your Testing
Part V- Working with Test Documentation
Part VI – The Future
- Chapters

3. Part VI: The Future 20. Software Quality Assurance.
20. Software Quality Assurance.
21. Your career as a Software Tester.

4. The Future Software Quality Assurance
Chapter 20
The Future Software Quality Assurance

5. 1. Quality Process - Terminology
Quality Objectives
Quality Policy
Quality Management (QM)
Quality System (QS)
Quality Control (QC)
Quality Assurance (QA)
Software Quality Assurance
Verification and Validation
Total Quality Management (TQM)
Continuous Improvement

6. What is the quality ?

7. Terminology – QO, QP Quality Objectives: Quality Policy
to achieve and sustain the quality of the product or service to meet the customer’s needs
to provide confidence to management that the quality is being achieved and maintained
to provide confidence to the customer that the quality is being achieved
Quality Policy
is ‘the overall quality intentions and direction of an organization as regards quality, as formally expressed by top management.’
(ISO9000, Clause 3.1)

8. Terminology – QM, QS Quality Management (QM) Quality System (QS)
is ‘that aspect of the overall management function that determines and implements the quality policy.’ (ISO9000, Clause 3.2)
Quality System (QS)
is ‘the organisational structure, responsibilities, procedures, processes and resources for implementing quality management.’ (ISO9000, Clause 3.3)

9. Terminology – QC Quality Control (QC)
is ‘the operational techniques and activities that are used to fulfill requirements for quality.’ (ISO9000, Clause 3.4)
involves evaluating performance, comparing goals and acting on the differences
Quality control involves the checking of the product

10. Terminology - QA Quality Assurance (QA)
‘all those planned and systematic actions necessary to provide adequate confidence that a product or service will satisfy given requirements for quality.’ (ISO9000, Clause 3.5)
‘a set of activities designed to evaluate the process by which products are developed or manufactured.’ (IEEE Standards )
Quality Assurance involves the checking of the process

11. Terminology – QA … Quality Assurance (QA) ... Purpose:
to prevent problems from occurring,
detect them when they do,
identify the cause,
remedy the cause and
prevent reoccurrence

12. Terminology – QA vs. QC QC / QA Differences QC - works with products
measures products against a standard
identifies defects
is limited to looking at products
QA - works with processes
a function that manages quality
sets up quality control
uses the results of QC to evaluate and improve the processes that produce the products

13. Terminology - QA … Quality Assurance (QA) ...
evaluates Goals, Methods, Performance

14. Terminology - V&V Verification and Validation
Verification: ‘Building the product RIGHT’
- confirms that the software meets its specification
Validation: ‘Building the RIGHT product’
- confirms that the software meets the user’s requirements
Never assume the specifications are correct when testing
Quality Assurance ensures that Verification and Validation takes place

15. Terminology - TQM Total Quality Management (TQM)
‘Managing a quality company is more than just implementing a quality system ... It is the creation of a quality culture that permeates the entire organization.’ (Sanders and Curran, 1994)
Characteristics of a quality culture:
dedication to customer satisfaction
emphasis on continuous improvement
Continuous Improvement
the continued improvement of the quality culture

16. Quality Measurement Remember, high quality products
Must meet all user requirements
Must be free of defects
In the PSP, the main measure of software product quality is total defect density
Total defect density
is measured as defects/KLOC (number of defects removed in development per thousand LOC)
Personal Software Process SM (PSPSM)
Team Software Process sm ( TSP sm)

17. Planning for Quality How to plan for quality:
WHAT attributes of the product manifest quality?
HOW is quality to be measured?
WHEN do we evaluate the product and the process?
WHO is responsible for carrying out the process?

18. Planning for Quality ... Establish a Quality Infrastructure
Establish a quality policy
Set up an organization to create the quality system - Software Quality Function
Develop a plan to implement the quality policy

19. Planning for Quality Establish a Quality Infrastructure ...
Review the process
Create standards and procedures
Implement Configuration Management
Control the development environment
Train staff
Keep quality related records
Take action to correct problems

20. 2. Software Quality - Function (SQA team)
Provides management with a degree of confidence that an independent, technically trained group is monitoring the goals, methods, and performance of applications from the beginning of the project
Relieves IT management of personally performing this function.

21. Software Quality - Management Environment

22. Software Quality Function (SQF) -Size
Now the rate is much higher than 1983
Quality staff: SQA group
Development staff:
Analyst
Designer
Programmer
Tester …

23. Role of SQF

24. Tasks of SQF

25. SQF - Reviewing Applications
Includes Verification and Validation
Software Reviews
Traceability of Software Deliverables
Testing
Auditing of selected key software items

26. SQF- Software Reviews
A software review is an evaluation of a software element to ascertain discrepancies from planned results and to recommend improvements
eg Design Review, Code Review
Three different types:
Walkthrough
Software Inspection
Technical Review

27. SQF- Software Reviews …
A way of using the diversity of a group to:
uncover errors in function, logic, or implementation of the software
to verify that software under review meets requirements
to ensure that the software has been represented according to predefined standards
to achieve software that is developed in a uniform manner
to make projects more manageable

28. SQF- Software Reviews …
Walkthrough
to evaluate a specific software element with a view to identifying defects and considering possible solutions
producer ‘walks’ participants through the product
not directed - topics suggested and questions asked
a moderator and minute-taker are used

29. SQF- Software Reviews …
Software Inspections
for evaluation of documents and code prior to technical review or testing
rigorous formal peer examination using a checklist of items to be verified
Objectives:
identify non-conformance to specifications and standards
measure progress
ignore stylistic issues and solutions

30. SQF- Software Reviews …
Technical Reviews
All major software items produced during a phase should be subjected to a technical review before being signed off
Objective: to provide evidence that it:
conforms to specification
was produced with appropriate project standards
has change control so all changes implemented

31. SQF- Technical Reviews ...
The Review Meeting
between 3 to 5 people (reviewers) should be involved
advance preparation should occur (< 2 hours work)
meeting should be < 2 hours
should be for a small part of the overall software
is attended by the producer, review leader and reviewers
must have a follow-up procedure to ensure any corrections are completed

32. SQF- Technical Reviews ...
The Review Meeting
At the end of the review the attendees must decide whether to:
accept the work product without modification
reject the work product due to severe errors (once corrected another review is performed)
accept the work product provisionally (minor errors to be corrected but no further review)

33. SQF - Technical Reviews ...
Review Reporting and Record Keeping
a reviewer records all issues during the review
a Reviews Issues List
summarizes these issues
serves as an action item checklist
a one-page Review Summary Report identifies:
what was reviewed
who reviewed it
what were the findings and conclusions

34. SQF - Review Report should contain:
Type of Review, Team Members, Date, Time, Place
Item being reviewed
Agenda
Checklist
Comments per Checklist item or per Issues list item (Pressman) with problems mentioned
Action per Comment - No action, Fix error, Reconsider 
Decision for the item being reviewed - Accept, Reject (severe errors), Accept (minor errors)

35. SQF - Technical Reviews ..
Review Guidelines
Review the product not the producer
Set an agenda and maintain it
Limit debate and rebuttal
Enunciate problem areas, but don’t attempt to solve every problem
Take written notes

36. SQF - Technical Reviews ..
Review Guidelines ...
Limit the number of participants and insist on advance preparation
Develop a checklist for each work product
Allocate resources and time schedule
Conduct meaningful training for all reviewers
Review earlier reviews

37. SQF - Traceability .. Forwards Traceability
each input to a phase must be matched to an output of the same phase to show completeness
Backwards Traceability
each output is traceable to an input of a phase

38. SQF - Testing .. The process of evaluating a system or component to:
confirm that it satisfies requirements
identify differences between actual and expected results
Frequently accounts for up to 50% of project costs so use cheaper methods (walkthroughs and inspections) before testing starts

39. SQF - Auditing ..
Audits are independent reviews that assess compliance with specifications, standards and procedures.
Perform audits before software release:
- Physical audit checks that all items identified as part of the configuration are present
- Functional audit checks that unit, integration and system test have been carried out and records their success or failure

40. Q & A

41. Software Quality Standards
Standards and Guidelines
‘A standard is an approved, documented, and available set of criteria used to determine the adequacy of an action (process standard) or object (product standard).’
‘A guideline is a well-defined and documented set of criteria that guides an activity or task.’ (Dorfman & Thayer, 1990)
differ from standards - allows for judgement and flexibility
Categories of Standards and Assessment
Product Standards
Calibration and Measurement Standards
Quality Management Systems Standards

42. Software Quality Standards ...
Software Quality Management Systems
International standard - ISO9000, is the international quality management systems standard
National standards eg AS3563
TickIT
Software Quality Management Process
Capability Maturity Model (CMM)
Software Process Improvement and Process dEtermination
(SPICE - now ISO15000)

43. Software Quality Standards ...
ISO9000 Standard
an agreement between the world’s authorities on the standards for quality systems
aim: to increase the confidence of customers in the quality systems of their suppliers
TickIT
a formal accreditation program for software developers based on the ISO Guidelines
first published in 1987 by the International Organisation for Standards (ISO)
the approach is formal and documented with two primary roles;
Quality Management
guidance is provided for suppliers to implement or improve quality systems in their organisations
Quality Assurance
generic requirements are provided for customers wanting to determine the adequacy of a supplier’s quality system
countries adopting ISO9000 assign a name or number to ISO9000 consistent with other existing national standards e.g.
USA - American National Standards Institute / American Society for Quality Control (ANSI/ASQC) Q90 Series
UK - British Standard (BS) 5750
Europe - European Norm (EN) 29000
Australia - Australian Standard (AS) 3900
TickIT Guide (Guide to Software Quality Management System Construction and Certification using EN29001)
TickITvision: by 1999 the top 5000 European firms will have ISO9001 registration and all software contracts will require ISO9001 certification

44. Software Quality Improvement
For companies and individuals
there may be many mistakes at producing a quality system at first
the idea is to learn by these mistakes and establish a program of continuous quality improvement
The source of the information is: PSP and TSP
Personal Software Process SM (PSPSM)
Team Software Process sm ( TSP sm)
Software Engineering Institute, Carnegie Mellon University, USA, 2001

45. PSP Process

46. PSP Structure

47. PSP-TSP

48. TSP Process Flow

49. TSP launch process

50. PSP-TSP-CMM

51. Quality Techniques Program Defect Removal Techniques Review a program
Compile a program
Test a program
Where and how do you remove defects?

52. Problems with Testing
Problems in testing are one of the major contributors to cost and schedule overruns
Many projects spend 40-50% of total development time in test
With today’s complex programs
There are many operating conditions
It is impossible to test every condition
User testing finds more defects
Defect Removal Costs Examples:
The later a defect is removed the higher its removal costs. Therefore if you don’t put a quality product into testing then you won’t get a quality product out

53. Personal Defect Management
The most economical and effective time to remove a defect is when it is injected
Student best recognizes the defect
Student understands what program does
Student is most likely to fix the defect correctly
The minimum fix time is when students
fix their own defects
make the fix shortly after the defect was injected

54. Personal Defect Management
PSP2 and PSP2.1 Objectives:
to introduce design and code reviews
and methods for improving the quality of reviews
Using PSP students
conduct personal design and code reviews
use a defined review process
design their own checklists
use a review script and follow the checklists
When PSP2 is used thoroughly there are more improvements from design and code reviews than any other process change

55. Personal Review Process
For effective reviews students must do the following:
follow the process
use a personal checklist designed to find the defects they make
devise a review strategy and use it
review one product component at a time
check for one topic at a time
treat each check as a certification that the product is free of this defect

56. Quality Management Summary
The objective of a high quality software process is early defect removal
PSP code and design reviews help produce quality products without increasing development costs
PSP data shows that personal reviews produce significant improvements in quality

57. Q & A

58. Exercise
P333: 4., 5.

59. The Future Your career as a Software Tester
Chapter 21
The Future Your career as a Software Tester

60. Big Space of Personal Development
测试人员 测试主管 项目经理

61. Q & A

62. End
Thanks !