1. The Software Development Life Cycle: An Overview Presented by Maxwell Drew and Dan Kaiser Southwest State University Computer Science Program

2. Last Time Introduction to the Principles of Testing The Testing Process Schwan’s Development Standards MSF Implementation and Testing RUP Implementation and Testing

3. Session 7: Testing and Deployment Brief review of the testing process Dynamic Testing Methods Static Testing Methods Deployment in MSF Deployment in RUP

4. Overall Goal Our overall goal is still to validate and verify the software. Validation "Are we building the right product?" Verification "Are we building the product right?”

5. The V-model of Development

6. Acceptance Tests Pilot test: install on experimental basis Alpha test: in-house test Beta test: customer pilot Parallel testing: new system operates in parallel with old system

7. Integration Testing Strategies Strategies covered Top-down testing Bottom-up testing Thread testing Stress testing Back-to-back testing

8. Dynamic verification Concerned with exercising and observing product behavior (testing) Static verification Concerned with analysis of the static system representation to discover problems Dynamic vs Static Verification

9. Static and Dynamic V&V

10. Methods of Dynamic V&V Black-box testing Structural testing (AKA White-box or Glass-box testing)

11. Defect testing The objective of defect testing is to discover defects in programs A successful defect test is a test which causes a program to behave in an anomalous way Tests show the presence not the absence of defects

12. Test data Inputs which have been devised to test the system Test cases Inputs to test the system and the predicted outputs from these inputs if the system operates according to its specification Test data and test cases

13. The defect testing process

14. Black-box testing Approach to testing where the program is considered as a ‘black-box’ The program test cases are based on the system specification Test planning can begin early in the software process

15. Black-box testing

16. Partition system inputs and outputs into ‘equivalence sets’ If input is a 5-digit integer between 10,000 and 99,999, then equivalence partitions are 100,000 Choose test cases at the boundary of these sets 00000, 09999, 10000, 99999, 100001 Equivalence partitioning

17. Equivalence partitions

18. Search routine specification procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : BOOLEAN; L: ELEM_INDEX) ; Pre-condition -- the array has at least one element T[FIRST] <= T[LAST ] Post-condition -- the element is found and is referenced by L ( Found and T[L] = Key) or -- the element is not in the array ( not Found and not (exists i, FIRST >= i <= LAST, T[i] = Key ))

19. Inputs which conform to the pre-conditions Inputs where a pre-condition does not hold Inputs where the key element is a member of the array Inputs where the key element is not a member of the array Search Routine - Input Partitions

20. Testing Guidelines (Arrays) Test software with arrays which have only a single value Use arrays of different sizes in different tests Derive tests so that the first, middle and last elements of the array are accessed Test with arrays of zero length (if allowed by programming language)

21. Search routine - input partitions

22. Search Routine - Test Cases

23. Sometime called white-box or glass- box testing Derivation of test cases according to program structure. Knowledge of the program is used to identify additional test cases Objective is to exercise all program statements (not all path combinations) Structural testing

24. White-box testing

25. Binary Search void Binary_search (elem key, elem T[ ], int size, bool &found, int &L) { int bott, top, mid ; bott = 0 ; top = size -1 ; L = ( top + bott ) / 2 ; if (T[L] == key) found = true ; else found = false ; while (bott <=top && !found){ mid = top + bott / 2 ; if ( T [mid] == key ){ found = true; L = mid ; } else if (T [mid] < key ) bott = mid + 1 ; else top = mid-1 ; } //end while } //binary_search

26. Pre-conditions satisfied, key element in array Pre-conditions satisfied, key element not in array Pre-conditions unsatisfied, key element in array Pre-conditions unsatisfied, key element not in array Input array has a single value Input array has an even number of values Input array has an odd number of values Binary Search - Equiv. Partitions

27. Binary search equiv. partitions

28. Binary search - test cases

29. Binary Search void Binary_search (elem key, elem T[ ], int size, bool &found, int &L) { int bott, top, mid ; bott = 0 ; top = size -1 ; L = ( top + bott ) / 2 ; if (T[L] == key) found = true ; else found = false ; while (bott <=top && !found){ mid = top + bott / 2 ; if ( T [mid] == key ){ found = true; L = mid ; } else if (T [mid] < key ) bott = mid + 1 ; else top = mid-1 ; } //end while } //binary_search

30. Binary Search Flow Graph

31. 1, 2, 3, 4, 12, 13 1, 2, 3, 5, 6, 11, 2, 12, 13 1, 2, 3, 5, 7, 8, 10, 11, 2, 12, 13 1, 2, 3, 5, 7, 9, 10, 11, 2, 12, 13 Test cases should be derived so that all of these paths are executed A dynamic program analyzer may be used to check that paths have been executed Independent paths

32. The number of tests necessary to test all control statements equals the cyclomatic complexity Cyclomatic complexity equals number of conditions in a program + 1 Can be calculated from the number of nodes (N) and the number of edges (E) in the flow graph Complexity = E - N + 2 Cyclomatic complexity

33. Static Verification Verifying the conformance of a software system to its specification without executing the code

34. Static Verification Involves analyses of source text by humans or software Can be carried out on ANY documents produced as part of the software process Discovers errors early in the software process Usually more cost-effective than testing for defect detection at the unit and module level Allows defect detection to be combined with other quality checks

35. Static Verification Effectiveness More than 60% of program errors can be detected by informal program inspections More than 90% of program errors may be detectable using more rigorous mathematical program verification The error detection process is not confused by the existence of previous errors

36. Program Inspections Formalized approach to document reviews Intended explicitly for defect DETECTION (not correction) Defects may be logical errors, anomalies in the code that might indicate an erroneous condition (e.g. an un-initialized variable) or non-compliance with standards

37. Fagan’s Inspection Pre-conditions A precise specification must be available Team members must be familiar with the organization standards Syntactically correct code must be available An error checklist should be prepared Management must accept that inspection will increase costs early in the software process Management must not use inspections for staff appraisal

38. The inspection process

39. Inspection procedure System overview presented to inspection team Code and associated documents are distributed to inspection team in advance Inspection takes place and discovered errors are noted Modifications are made to repair discovered errors Re-inspection may or may not be required

40. Inspection Teams Made up of at least 4 members Author of the code being inspected Reader who reads the code to the team Inspector who finds errors, omissions and inconsistencies Moderator who chairs the meeting and notes discovered errors Other roles are Scribe and Chief moderator

41. Inspection rate 500 statements/hour during overview 125 source statement/hour during individual preparation 90-125 statements/hour can be inspected Inspection is therefore an expensive process Inspecting 500 lines costs about 40 staff/hours effort = $$$$$

42. Inspection checklists Checklist of common errors should be used to drive the inspection Error checklist is programming language dependent The 'weaker' the type checking, the larger the checklist Examples: Initialization, Constant naming, loop termination, array bounds, etc.

43. Table 8.2. Typical inspection preparation and meeting times. Development artifactPreparation timeMeeting time Requirements document25 pages per hour12 pages per hour Functional specification45 pages per hour15 pages per hour Logic specification50 pages per hour20 pages per hour Source code150 lines of code per hour75 lines of code per hour User documents35 pages per hour20 pages per hour Table 8.3. Faults found during discovery activities. Discovery activityFaults found per thousand lines of code Requirements review2.5 Design review5.0 Code inspection10.0 Integration test3.0 Acceptance test2.0

44. Mathematically-based Verification Verification is based on mathematical arguments which demonstrate that a program is consistent with its specification Programming language semantics must be formally defined The program must be formally specified

45. Program Proving Rigorous mathematical proofs that a program meets its specification are long and difficult to produce Some programs cannot be proved because they use constructs such as interrupts. These may be necessary for real-time performance The cost of developing a program proof is so high that it is not practical to use this technique in the vast majority of software projects

46. Program Verification Arguments Less formal, mathematical arguments can increase confidence in a program's conformance to its specification Must demonstrate that a program conforms to its specification Must demonstrate that a program will terminate

47. Axiomatic approach Define pre and post conditions for the program or routine Demonstrate by logical argument that the application of the code logically leads from the pre to the post-condition Demonstrate that the program will always terminate

48. The name is derived from the 'Cleanroom' process in semiconductor fabrication. The philosophy is defect avoidance rather than defect removal Software development process based on: Incremental development Formal specification. Static verification using correctness arguments Statistical testing to determine program reliability. Cleanroom

49. The Cleanroom Process

50. Testing Effectiveness Experimentation found black-box testing to be more effective than structural testing in discovering defects Static code reviewing was less expensive and more effective in discovering program faults

51. Table 8.5. Fault discovery percentages by fault origin. Discovery techniqueRequirementsDesignCodingDocumentation Prototyping4035 15 Requirements review401505 Design review1555015 Code inspection20406525 Unit testing15200 Table 8.6. Effectiveness of fault discovery techniques. (Jones 1991) Requirements faults Design faultsCode faultsDocumentation faults Reviews FairExcellent Good Prototypes GoodFair Not applicable Testing Poor GoodFair Correctness proofs Poor Fair

52. When to Stop Testing Fault seeding Assumption

53. When to Stop Testing Confidence in the software C = confidence S = number of seeded faults N = number of faults claimed (0 = no faults) n = number of actual faults discovered

54. Questions?