1. OHT 9.1 Galin, SQA from theory to implementation © Pearson Education Limited 2004 Chapter 9.3 Software Testing Strategies

2. OHT 9.2 Galin, SQA from theory to implementation © Pearson Education Limited 2004 Definitions and objectives Software testing strategies Software test classifications White box testing Data processing and calculation correctness tests Correctness tests and path coverage Correctness tests and line coverage McCabe’s cyclomatic complexity metrics Software qualification and reusability testing Advantages and disadvantages of white box testing  Black box testing Equivalence classes for output correctness tests Other operation factor testing classes Revision factor testing classes Transition factor testing classes Advantages and disadvantages of black box testing Chapter 9.3 – Software Testing Strategies

3. OHT 9.3 Galin, SQA from theory to implementation © Pearson Education Limited 2004 Introductory Thoughts Black Box Testing: Use to perform ‘correctness tests’ and more Some of these white box tests, such as maintainability tests and calculation correctness testing. Bottom line: BB testing canNOT substitute for WB testing. Will look at: –Equivalence class testing and number of test cases required –Performance methodology for other classes of BB tests –Advantages and disadvantages of BB testing.

4. OHT 9.4 Galin, SQA from theory to implementation © Pearson Education Limited 2004 Equivalence Classes for Output Correctness Correctness tests take the greater part of testing resources. One good approach to an improved choice of tests is the equivalence class partitioning. –Increases testing efficiency –An Equivalence Class (EC) is a carefully devised limited set of input variable values that essentially produce the same output results using many more ‘equivalent’ tests. – We have single values, ranges, etc. to consider –We have valid ECs and invalid ECs

5. OHT 9.5 Galin, SQA from theory to implementation © Pearson Education Limited 2004 Equivalence Classes for Output Correctness Design tests such that each valid EC and each invalid EC are included in at least one test case. Test cases are defined separately for each type of EC. For EC class testing, we include ECs in EC Test Cases. –Can have a number of valid EC tests in a single test case. For Invalid EC testing, we assign a single test case to each new invalid EC test. –Can, in this way, distinguish results of each invalid EC.

6. OHT 9.6 Galin, SQA from theory to implementation © Pearson Education Limited 2004 Equivalence Classes for Output Correctness Bottom line here: by using ECs, far fewer test cases needed than might otherwise be needed Test cases are based on the specifications not on code Better results are obtained in testing boundary values of ECs as well. Very important to test boundaries. Rich source of very common errors. More later on this.

7. OHT 9.7 Galin, SQA from theory to implementation © Pearson Education Limited 2004 Equivalence Classes for Output Correctness Boundary Class testing example: (home made) Reading a file: (sequential) –Boundary issues: Can’t open the file File is empty (no records) Read beyond end of file File has one record –Standard EC test here: once you process two records, you are likely good to go. Stack –Stack empty; stack full. These are the boundary conditions. –Other processing - normal Always, always always test the boundaries!!!

8. OHT 9.8 Galin, SQA from theory to implementation © Pearson Education Limited 2004 Test Cases and Boundary Values Formally, one test case per equivalence class should be enough. If, however, have range of values, border values are very error prone. –Example: Range of acceptable values: 1-10. –Midrange test (no problemo) any number 2-9; –Boundary value tests: test 1 and test 10. Test for mid range, lower and upper bound values!! Let’s review the principles again for Equivalence Class partitioning…

9. OHT 9.9 Galin, SQA from theory to implementation © Pearson Education Limited 2004 A black box method aimed at increasing the efficiency of testing and, at the same time, improving coverage of potential error conditions.

10. OHT 9.10 Galin, SQA from theory to implementation © Pearson Education Limited 2004 An equivalence class (EC) is a set of input variable values that produce the same output results or that are processed identically. –Again, consider a range of acceptable values: say, 1 – 10. –Tests with inputs 2 through 9 should produce the same result. –No need for more than one test with input value 2 to 9. EC boundaries are defined by a single numeric or alphabetic value, a group of numeric or alphabetic values, a range of values, and so on. –If range is 1-10, boundaries would be 1 and 10.

11. OHT 9.11 Galin, SQA from theory to implementation © Pearson Education Limited 2004 An EC with only valid states is defined as a "valid EC," whereas an EC that contains only invalid states is defined as the "invalid EC." –By definition! In cases where a program's input is provided by several variables, valid and invalid ECs should be defined for each variable.

12. OHT 9.12 Galin, SQA from theory to implementation © Pearson Education Limited 2004 According to the equivalence class partitioning method: Each valid EC and each invalid EC are included in at least one test case. –Note: equivalence CLASS! –For an acceptable range of input integers is 1-10, we need at least one test with a value in the range 1-10. –Boundaries: 1 and 10 –Invalid EC classes might be EC with values 10, another one - negative numbers, …. Definition of test cases is done separately for the valid and invalid ECs. –Note: Run tests for invalid equivalence classes one at a time.

13. OHT 9.13 Galin, SQA from theory to implementation © Pearson Education Limited 2004 According to the equivalence class partitioning method: In defining a test case for the valid ECs, we try to cover as many as possible “new” ECs in that same test case. –Valid ranges: 1-10; 30-50. –Include both range EC values in same EC test. In defining invalid ECs, assign one test case to each “new” invalid EC. Run separately. A test case with more than one invalid EC may not allow the tester to distinguish between the program’s separate reactions to each of the invalid ECs. Test cases are added as long as there are uncovered ECs.

14. OHT 9.14 Galin, SQA from theory to implementation © Pearson Education Limited 2004 Operation Factor Testing Cases Other kinds of tests include : –Correctnes s Documentation tests Availability (reaction time tests) –Reliability Reliability tests –Efficiency Stress tests (load and durability tests) –Integrit y Software system security tests –Usability Training usability tests Operational usability tests

15. OHT 9.15 Galin, SQA from theory to implementation © Pearson Education Limited 2004 Operation Factor Testing Cases in more Detail Kinds of tests : –Correctnes s Documentation tests –So often neglected –Erroneous user manuals and operators manuals –Functional Descriptions »Overview describes system. Useful for many purposes –Installation Manual »How to install and interface with equipment; for COTS, contains customization instructions –User Manual »How to use various functions, recovery instructions; documented help… –Program Maintenance Manual »Bug corrections, changes, program structure, some algorithms… Availability (reaction time tests) –Critically important for online applications. –Response time –Critical for real time systems –Load and stress testing critical

16. OHT 9.16 Galin, SQA from theory to implementation © Pearson Education Limited 2004 More Black Box Tests – Reliability Reliability tests –Difficult to test –Need to run under various and regular workload conditions –Often simulated environment and loads –Sometimes: MTBF; MTTR –Efficiency Stress tests (load and durability tests) –Load tests are related to the functional performance under max load conditions –E.g. Hits per minute, number of simultaneous users. –Usually simulate more load than specifications cite. Durability tests – –extreme operating conditions, temperature, humidity, rough conditions, think aircraft instrumentation and guidance…sudden electrical ‘jumps’ communications cutoff (battery backup? …) –Can see we are not going ‘in the Box.’ –

17. OHT 9.17 Galin, SQA from theory to implementation © Pearson Education Limited 2004 Still more BB Tests Other kinds of tests include : – Integrit y – read this… Software system security tests –Usability Training and Operational usability tests –Test the operator’s productivity in using or trying to use the application. »Observation and his/her productivity »Can be quantitative and qualitative.

18. OHT 9.18 Galin, SQA from theory to implementation © Pearson Education Limited 2004 Other Tests Maintainability tests –Remember: More than 60% of a system’s operational time is usually done in a maintenance mode! –Other figures suggest higher rates! Overall manuals should include one that addresses: system structure, programmer maintenance and an operator’s manual.

19. OHT 9.19 Galin, SQA from theory to implementation © Pearson Education Limited 2004 Advantages:  Allows us to carry out the majority of testing classes, most of which can be implemented solely by black box tests, i.e. load tests and availability tests.  For testing classes that can be carried out by both white and black box tests, black box testing requires fewer resources. Disadvantages:  Possibility that coincidental aggregation of several errors will produce the correct response for a test case, and prevent error detection.  Absence of control of line coverage. There is no easy way to specify the parameters of the test cases required to improve coverage. * Impossibility of testing the quality of coding and its strict adherence to the coding standards. Advantages and Disadvantages of Black Box Testing

20. OHT 9.20 Galin, SQA from theory to implementation © Pearson Education Limited 2004 Homework Questions 9.1 and 9.4 –Due Wednesday, 13 Nov at 4pm

21. OHT 9.21 Galin, SQA from theory to implementation © Pearson Education Limited 2004 Team Presentation Discuss statement, path, and branch coverage measures. Discuss: static testing and dynamic testing with examples Discuss black box testing and white box testing with examples Discuss verification and validation and acceptance testing. Give examples