Software Testing & QA (VI) Kerry Zhu Zhu.Kerry@gmail.com

SUMMARY SO FAR Part VI – The Future Zhu.Kerry@gmail.com Part I- The Big Picture - an Introduction Part II- Testing Fundamentals - heart of course Part III- Applying Your Testing Skills Part IV- Supplementing Your Testing Part V- Working with Test Documentation Part VI – The Future - Chapters 20 - 21

Part VI: The Future 20. Software Quality Assurance. Zhu.Kerry@gmail.com 20. Software Quality Assurance. 21. Your career as a Software Tester.

The Future Software Quality Assurance Chapter 20 The Future Software Quality Assurance

1. Quality Process - Terminology Quality Objectives Quality Policy Quality Management (QM) Quality System (QS) Quality Control (QC) Quality Assurance (QA) Software Quality Assurance Verification and Validation Total Quality Management (TQM) Continuous Improvement

What is the quality ?

Terminology – QO, QP Quality Objectives: Quality Policy to achieve and sustain the quality of the product or service to meet the customer’s needs to provide confidence to management that the quality is being achieved and maintained to provide confidence to the customer that the quality is being achieved Quality Policy is ‘the overall quality intentions and direction of an organization as regards quality, as formally expressed by top management.’ (ISO9000, Clause 3.1)

Terminology – QM, QS Quality Management (QM) Quality System (QS) is ‘that aspect of the overall management function that determines and implements the quality policy.’ (ISO9000, Clause 3.2) Quality System (QS) is ‘the organisational structure, responsibilities, procedures, processes and resources for implementing quality management.’ (ISO9000, Clause 3.3)

Terminology – QC Quality Control (QC) is ‘the operational techniques and activities that are used to fulfill requirements for quality.’ (ISO9000, Clause 3.4) involves evaluating performance, comparing goals and acting on the differences Quality control involves the checking of the product

Terminology - QA Quality Assurance (QA) ‘all those planned and systematic actions necessary to provide adequate confidence that a product or service will satisfy given requirements for quality.’ (ISO9000, Clause 3.5) ‘a set of activities designed to evaluate the process by which products are developed or manufactured.’ (IEEE Standards ) Quality Assurance involves the checking of the process

Terminology – QA … Quality Assurance (QA) ... Purpose: to prevent problems from occurring, detect them when they do, identify the cause, remedy the cause and prevent reoccurrence

Terminology – QA vs. QC QC / QA Differences QC - works with products measures products against a standard identifies defects is limited to looking at products QA - works with processes a function that manages quality sets up quality control uses the results of QC to evaluate and improve the processes that produce the products

Terminology - QA … Quality Assurance (QA) ... evaluates Goals, Methods, Performance

Terminology - V&V Verification and Validation Verification: ‘Building the product RIGHT’ - confirms that the software meets its specification Validation: ‘Building the RIGHT product’ - confirms that the software meets the user’s requirements Never assume the specifications are correct when testing Quality Assurance ensures that Verification and Validation takes place

Terminology - TQM Total Quality Management (TQM) ‘Managing a quality company is more than just implementing a quality system ... It is the creation of a quality culture that permeates the entire organization.’ (Sanders and Curran, 1994) Characteristics of a quality culture: dedication to customer satisfaction emphasis on continuous improvement Continuous Improvement the continued improvement of the quality culture

Quality Measurement Remember, high quality products Must meet all user requirements Must be free of defects In the PSP, the main measure of software product quality is total defect density Total defect density is measured as defects/KLOC (number of defects removed in development per thousand LOC) Personal Software Process SM (PSPSM) Team Software Process sm ( TSP sm)

Planning for Quality How to plan for quality: WHAT attributes of the product manifest quality? HOW is quality to be measured? WHEN do we evaluate the product and the process? WHO is responsible for carrying out the process?

Planning for Quality ... Establish a Quality Infrastructure Establish a quality policy Set up an organization to create the quality system - Software Quality Function Develop a plan to implement the quality policy

Planning for Quality Establish a Quality Infrastructure ... Review the process Create standards and procedures Implement Configuration Management Control the development environment Train staff Keep quality related records Take action to correct problems

2. Software Quality - Function (SQA team) Provides management with a degree of confidence that an independent, technically trained group is monitoring the goals, methods, and performance of applications from the beginning of the project Relieves IT management of personally performing this function.

Software Quality - Management Environment

Software Quality Function (SQF) -Size Now the rate is much higher than 1983 Quality staff: SQA group Development staff: Analyst Designer Programmer Tester …

Role of SQF

Tasks of SQF

SQF - Reviewing Applications Includes Verification and Validation Software Reviews Traceability of Software Deliverables Testing Auditing of selected key software items

SQF- Software Reviews A software review is an evaluation of a software element to ascertain discrepancies from planned results and to recommend improvements eg Design Review, Code Review Three different types: Walkthrough Software Inspection Technical Review

SQF- Software Reviews … A way of using the diversity of a group to: uncover errors in function, logic, or implementation of the software to verify that software under review meets requirements to ensure that the software has been represented according to predefined standards to achieve software that is developed in a uniform manner to make projects more manageable

SQF- Software Reviews … Walkthrough to evaluate a specific software element with a view to identifying defects and considering possible solutions producer ‘walks’ participants through the product not directed - topics suggested and questions asked a moderator and minute-taker are used

SQF- Software Reviews … Software Inspections for evaluation of documents and code prior to technical review or testing rigorous formal peer examination using a checklist of items to be verified Objectives: identify non-conformance to specifications and standards measure progress ignore stylistic issues and solutions

SQF- Software Reviews … Technical Reviews All major software items produced during a phase should be subjected to a technical review before being signed off Objective: to provide evidence that it: conforms to specification was produced with appropriate project standards has change control so all changes implemented

SQF- Technical Reviews ... The Review Meeting between 3 to 5 people (reviewers) should be involved advance preparation should occur (< 2 hours work) meeting should be < 2 hours should be for a small part of the overall software is attended by the producer, review leader and reviewers must have a follow-up procedure to ensure any corrections are completed

SQF- Technical Reviews ... The Review Meeting At the end of the review the attendees must decide whether to: accept the work product without modification reject the work product due to severe errors (once corrected another review is performed) accept the work product provisionally (minor errors to be corrected but no further review)

SQF - Technical Reviews ... Review Reporting and Record Keeping a reviewer records all issues during the review a Reviews Issues List summarizes these issues serves as an action item checklist a one-page Review Summary Report identifies: what was reviewed who reviewed it what were the findings and conclusions

SQF - Review Report should contain: Type of Review, Team Members, Date, Time, Place Item being reviewed Agenda Checklist Comments per Checklist item or per Issues list item (Pressman) with problems mentioned Action per Comment - No action, Fix error, Reconsider  Decision for the item being reviewed - Accept, Reject (severe errors), Accept (minor errors)

SQF - Technical Reviews .. Review Guidelines Review the product not the producer Set an agenda and maintain it Limit debate and rebuttal Enunciate problem areas, but don’t attempt to solve every problem Take written notes

SQF - Technical Reviews .. Review Guidelines ... Limit the number of participants and insist on advance preparation Develop a checklist for each work product Allocate resources and time schedule Conduct meaningful training for all reviewers Review earlier reviews

SQF - Traceability .. Forwards Traceability each input to a phase must be matched to an output of the same phase to show completeness Backwards Traceability each output is traceable to an input of a phase

SQF - Testing .. The process of evaluating a system or component to: confirm that it satisfies requirements identify differences between actual and expected results Frequently accounts for up to 50% of project costs so use cheaper methods (walkthroughs and inspections) before testing starts

SQF - Auditing .. Audits are independent reviews that assess compliance with specifications, standards and procedures. Perform audits before software release: - Physical audit checks that all items identified as part of the configuration are present - Functional audit checks that unit, integration and system test have been carried out and records their success or failure

Q & A

Software Quality Standards Standards and Guidelines ‘A standard is an approved, documented, and available set of criteria used to determine the adequacy of an action (process standard) or object (product standard).’ ‘A guideline is a well-defined and documented set of criteria that guides an activity or task.’ (Dorfman & Thayer, 1990) differ from standards - allows for judgement and flexibility Categories of Standards and Assessment Product Standards Calibration and Measurement Standards Quality Management Systems Standards

Software Quality Standards ... Software Quality Management Systems International standard - ISO9000, is the international quality management systems standard National standards eg AS3563 TickIT Software Quality Management Process Capability Maturity Model (CMM) Software Process Improvement and Process dEtermination (SPICE - now ISO15000)

Software Quality Standards ... ISO9000 Standard an agreement between the world’s authorities on the standards for quality systems aim: to increase the confidence of customers in the quality systems of their suppliers TickIT a formal accreditation program for software developers based on the ISO9000-3 Guidelines first published in 1987 by the International Organisation for Standards (ISO) the approach is formal and documented with two primary roles; Quality Management guidance is provided for suppliers to implement or improve quality systems in their organisations Quality Assurance generic requirements are provided for customers wanting to determine the adequacy of a supplier’s quality system countries adopting ISO9000 assign a name or number to ISO9000 consistent with other existing national standards e.g. USA - American National Standards Institute / American Society for Quality Control (ANSI/ASQC) Q90 Series UK - British Standard (BS) 5750 Europe - European Norm (EN) 29000 Australia - Australian Standard (AS) 3900 TickIT Guide (Guide to Software Quality Management System Construction and Certification using EN29001) TickITvision: by 1999 the top 5000 European firms will have ISO9001 registration and all software contracts will require ISO9001 certification

Software Quality Improvement For companies and individuals there may be many mistakes at producing a quality system at first the idea is to learn by these mistakes and establish a program of continuous quality improvement The source of the information is: PSP and TSP Personal Software Process SM (PSPSM) Team Software Process sm ( TSP sm) Software Engineering Institute, Carnegie Mellon University, USA, 2001

PSP Process

PSP Structure

PSP-TSP

TSP Process Flow

TSP launch process

PSP-TSP-CMM

Quality Techniques Program Defect Removal Techniques Review a program Compile a program Test a program Where and how do you remove defects?

Problems with Testing Problems in testing are one of the major contributors to cost and schedule overruns Many projects spend 40-50% of total development time in test With today’s complex programs There are many operating conditions It is impossible to test every condition User testing finds more defects Defect Removal Costs Examples: The later a defect is removed the higher its removal costs. Therefore if you don’t put a quality product into testing then you won’t get a quality product out

Personal Defect Management The most economical and effective time to remove a defect is when it is injected Student best recognizes the defect Student understands what program does Student is most likely to fix the defect correctly The minimum fix time is when students fix their own defects make the fix shortly after the defect was injected

Personal Defect Management PSP2 and PSP2.1 Objectives: to introduce design and code reviews and methods for improving the quality of reviews Using PSP students conduct personal design and code reviews use a defined review process design their own checklists use a review script and follow the checklists When PSP2 is used thoroughly there are more improvements from design and code reviews than any other process change

Personal Review Process For effective reviews students must do the following: follow the process use a personal checklist designed to find the defects they make devise a review strategy and use it review one product component at a time check for one topic at a time treat each check as a certification that the product is free of this defect

Quality Management Summary The objective of a high quality software process is early defect removal PSP code and design reviews help produce quality products without increasing development costs PSP data shows that personal reviews produce significant improvements in quality

Q & A

Exercise P333: 4., 5.

The Future Your career as a Software Tester Chapter 21 The Future Your career as a Software Tester

Big Space of Personal Development 测试人员 测试主管 项目经理

Q & A

End Thanks !