1. Software Testing

2. Software Quality Assurance Overarching term Time consuming (40% to 90% of dev effort) Includes –Verification: Building the product right, i.e. follow process, meet specifications –Validation: Building the right product, i.e. its what the customer needs –Testing: Create test suite, run program and compare actual and expected results

3. Quality Assurance Methods Prototyping Walkthroughs Inspections Proofs Code Testing –Black box test case selection –White/Glass box test case selection (structural)

4. Code Testing Unit: Creator tests new unit on correct and incorrect input Integration: –top down with stubs –bottom up with drivers Product: Documentation, Functionality, Performance, Robustness Acceptance: by user representatives Maintenance: “regression” testing

5. Test Case Selection Critical usually impossible to test all cases black box –suppose 5 types of commissions and 7 types of discounts –suppose 20 input factors of 4 possible values each white box –consider the number of paths possible

6. Test Selection Methodologies – Black Box Symbolic execution –input A, B instead of 4, 6 etc. Random –based on known probability distribution of input Functional (testing to specs) –check function –partition input space into equivalence classes …

7. Input Equivalence Classes All elements of an equivalence class considered equal for testing purposes For example, specs say “if the input temperature is between 40 and 50 output Warning A” then the classes are 50 Always emphasize boundaries, therefore –5, 0, 1, 20, 39, 40, 41, 45, 49, 50, 51, 75, 100 might be a good set of test cases

8. Input Equivalence Classes Example: Read in, sort, and output a list of up to 50 positive integers … what’s a good set of test cases? Note: could also have “output equivalence classes” Note: apply same reasoning when designing/coding also

9. Black Box Limitations Specs say “input three integers and print out whether or not they are all the same integer”

10. Test Selection Methodologies – White Box Program instrumentation –trace statements –assert statements Structural testing –statement coverage –branch coverage –path coverage –definition/use path coverage

11. Test Selection Methodologies – White Box Partition testing –partition input space based on program statements Mutation testing –mutate program –determine if test suite catches mutation

12. Glass box limitations Consider previous code example –test case 1: input 1, 2, 3 –test case 2: input 2, 2, 2 We have complete path coverage but do not find error Cannot discover “missing” path, for example x = n / d should be if ( d != 0) x = n / d else …

13. When to stop testing? Software Reliability Models –calendar time –execution time Use historical information … carefully “CIA” approach –seed errors –test –compare