1. Dillon: CSE470: QUALITY ASSURANCE1 Software Qualities Maintainer User Customer Good Documentation Readable Code Good Design Low Cost Portability Increased productivity Reliability Correctness Efficiency Functionality Ease of use Ease of learning

2. Dillon: CSE470: QUALITY ASSURANCE2 Software Quality Assurance l Use of analysis to validate artifacts »requirements, designs, code, test plans l Technical reviews l Document reviews l Compliance to standards l Control of changes

3. Dillon: CSE470: QUALITY ASSURANCE3 Costs of Poor Quality l Increased time to find and fix problems l Increased cost to distribute modifications l Increased customer support l Product liability l Failure in the market place

4. Dillon: CSE470: QUALITY ASSURANCE4 QA Terminology l Correctness l Reliability l Testing l Debugging l Failure l Fault l Error l Verification l Validation l V&V

5. Dillon: CSE470: QUALITY ASSURANCE5 Terminology l Correctness: artifact is consistent with its specification »Specification could be wrong or incomplete »Rarely is software known to be correct »Rarely is the specification correct l Reliability: probability that the software is correct »Statistical measure based on past performance –e.g., mean time to failure

6. Dillon: CSE470: QUALITY ASSURANCE6 More terminology l Testing: entails executing the software on selected test cases »Evaluate the results (oracle) »Evaluate the performance »Evaluate the ease of use l Common Wisdom: Testing reveals bugs but does not guarantee the absence of bugs »How should you select test cases? »How do you know when to stop testing?

7. Dillon: CSE470: QUALITY ASSURANCE7 More terminology l Failure: an erroneous result »incorrect outputs/response for given inputs/stimuli »fails to meet real-time constraints l Error: incorrect concept »may cause failures if not corrected l Fault: the cause of one or more failures »discovered after release

8. Dillon: CSE470: QUALITY ASSURANCE8 More terminology l Debugging: the process of finding the cause of a “bug” and a way to fix it »w/o introducing additional bugs! l Verification: process of proving, using mathematical reasoning, that a program is “correct” »is expensive and is not always possible »is not foolproof

9. Dillon: CSE470: QUALITY ASSURANCE9 More terminology l Validation: process associated with showing that the software peforms reasonably well l V & V: verification & validation? »more typically equated with validation

10. Dillon: CSE470: QUALITY ASSURANCE10 Many different kinds of testing l Unit testing: test individual components »test stubs simulate called components »test harness simulates “outer” context and maintains stubs l Integration testing: combine components and test them »follows build plan l System testing: test whole system

11. Dillon: CSE470: QUALITY ASSURANCE11 More kinds of testing l Acceptance testing: testing to determine if the product is acceptable l Regression testing: retesting after the system has been modified »determine “old” test cases that must be re- executed »determine what new test cases are required

12. Dillon: CSE470: QUALITY ASSURANCE12 More kinds of testing l Black box / functional testing: »testing based on specifications l White box / structural testing: »testing based on looking at the artifact l Both black box and white box testing are needed

13. Dillon: CSE470: QUALITY ASSURANCE13 Testing is hard work l Typically 50% of software development effort goes into testing »up to 85% for life-critical software l How to identify “good” test cases? »high probability of finding a new error l Objective is to find errors »test case is “successful” if it finds a new error

14. Dillon: CSE470: QUALITY ASSURANCE14 Testing is hard work (cont.) l Psychologically difficult for a programmer to test his/her own code thoroughly l Exhaustive testing requires testing all combinations of input values »Sorting an array of size 10 containing integers in the range 1.. 10 has 10! combinations (3,628,800 cases)

15. Dillon: CSE470: QUALITY ASSURANCE15 Testing l CAN: »Uncover errors »Show specifications are met for specific test cases »Be an indication of overall reliability l CANNOT: »Prove that a program is error-free

16. Dillon: CSE470: QUALITY ASSURANCE16 Testing Principles l Tests should be traceable to requirements l Tests should be planned long before testing begins l Exhaustive testing is not possible »80% of all errors typically occur in 20% of the modules »test cases should be chosen to maximize likelihood of finding an error

17. Dillon: CSE470: QUALITY ASSURANCE17 Testing Principles (cont.) l Testing should be done by someone other than the developers »Developers do original testing »SQA does independent testing –usually black box testing l Automated testing tools should be used »Reduce testing costs »Reduce likelihood of human error

18. Dillon: CSE470: QUALITY ASSURANCE18 Testability l Simple software is easier to test »minimize coupling, maximize cohesion l Output is sufficient to determine correct behavior l Performs its own tests for internal errors »raises meaningful exceptions l All code is reachable l Independent modules can be tested in isolation l Documentation is complete and accurate

19. Dillon: CSE470: QUALITY ASSURANCE19 Quality is an on-going concern l You can’t build quality into a system after the fact l Quality should be a consideration during every phase of development l Plan for testing / validation in all phases »requirements -> functional test cases »design -> functional and structural test cases »code -> enhanced func & struc test cases »maintenance -> further enhanced funct & struc test cases

20. Dillon: CSE470: QUALITY ASSURANCE20 Debugging l Find the cause of a failure and fix it »an art, not a science l Debugging is difficult because »symptom may appear long after the fault occurs »symptom may be difficult to reproduce »symptom may be intermittent l Unit testing helps localize errors

21. Dillon: CSE470: QUALITY ASSURANCE21 Software Reviews l Individuals read and comment on the software artifacts l Very human intensive l Overriding evidence shows that it »improves quality and productivity »reduces cost l It is usually one of the first activities to be dropped when schedules get tight

22. Dillon: CSE470: QUALITY ASSURANCE22 Software Reviews (cont.) l Applicable to all software artifacts »code inspections »requirements and design reviews »walk-throughs l Recent research shows that »particular kind of review, size of team, etc. doesn’t matter »need at least one good, dedicated person doing the review

23. Dillon: CSE470: QUALITY ASSURANCE23 Typical Review Team l Developer -- presents the material l Moderator -- keeps the review on track »makes sure everyone abides by the process l Secretary -- takes minutes, documents problems found l Optionally »Apprentice -- learning about the project »Domain expert -- familiar with the domain and can verify assumptions

24. Dillon: CSE470: QUALITY ASSURANCE24 Software Review Guidelines l Review the artifact »don’t attack the developer l Stick to an agenda l Limit debate l Identify problems, not solutions l Keep accurate notes l Establish and follow evaluation guidelines l Limit number of participants

25. Dillon: CSE470: QUALITY ASSURANCE25 Technical Review Guidelines (cont.) l Prepare beforehand »both developers and reviewers l Allocate resources for reviews »people and time l Possible outcomes »accept product as is »reject product outright »accept product provisionally

26. Dillon: CSE470: QUALITY ASSURANCE26 Sample evaluation Guidelines: Code Inspection l Has the design been correctly translated to code? l Are the language features used appropriately? l Are coding standards followed? l Are documentation standards followed? l Are there misspellings or typos? l Are the comments accurate and unambiguous?

27. Dillon: CSE470: QUALITY ASSURANCE27 Sample evaluation Guidelines: Code Inspection (cont.) l Are data types and declarations appropriate? l Are all constants correct? l Are all variables initialized before being used? l Are there overly complex conditions? l Is there unreachable code? l Are there obvious inefficiencies?

28. Dillon: CSE470: QUALITY ASSURANCE28 SQA Summary l U.S. software costs $200 billion/year l Need to »improve software quality »reduce costs –V&V is over 50% of the cost l Improving V&V should reduce costs significantly while improving quality