1. Testing the Programs CS4311 – Spring 2008 Software engineering, theory and practice, S. Pfleeger, Prentice Hall ed. Object-oriented and classical software engineering, S. Schach, McGraw Hill ed. Software Engineering: a Practitioner’s Approach, Pressman, McGraw Hill ed. Software Requirements: Objects, Functions, and States, Davis, Prentice Hall ed.

2. Plan Types and classification of faults Purpose of testing Types and levels of testing Test planning and implementation standards White-box and black-box testing Strategies for defining test cases When to stop testing Integration testing strategies

3. Software Fault and Failures Ideal case: each time we run our program it works Real case: not the above A is an incorrect intermediate state that may be entered during program execution A of the system is the result of one or more in some aspect of the system

4. Reasons for Failure Wrong specifications, missing requirements Requirements cannot be met Faulty design (system design or code design) Error in the program, algorithm not properly implemented, etc…

5. Verification Approaches for Implementation Non execution-based   Code inspection  Correctness proofs Automated theorem proving Testing

6. Informal Approaches Examining the code  code review  Experts review code and documentation  Experts look for misunderstanding, inconsistencies, other faults Code walkthrough  Programmer presents the code to the review team  Team comments on correctness  Informal process  Focus on finding faults but not necessary fixing them

7. Formal Approach-1 Code inspection (Fagan 1976)  Similar to a walkthrough but more formal  The review team checks the code and documentation against a prepared list of concerns  5 steps: Preparation Inspection Rework Follow-up

8. Formal Approach-2 Rule: if more than 5% of the material inspected has been reworked, then the team will go through another iteration In a 6000-line business data processing application, 93% of faults were found during inspections Formalism of inspections is better over walkthroughs

9. Code Correctness View the code as a statement of logical flow Test the correctness of the expression Automated techniques  Symbolic execution  Formal proof techniques

10. Symbolic Execution Involves simulated execution of the code using symbols instead of data variables The program execution is viewed as a sequence of states depending on input The program is correct if for each input state generates the correct output state

11. Automated Theorem Proving Idea: develop a tool that takes as input input data and conditions output data and conditions lines of code for the components to be tested Tool either provides a proof of the correctness of the component or a counter-example Very appealing, but the perfect theorem prover is an undecidable problem

12. Formal Proof Techniques int k,s; int y[n]; k=0; s=0; while(k<n) { s=s+y[k]; k=k+1; } Post-condition: s contains the sum of all elements of the array y Proof of correctness is done by induction on n

13. Advantages and Disadvantages of Formal Techniques Logic formalism helps discovering algorithmic faults in the code But The logical description of the code can be difficult to obtain and much longer than the code Does not identify faults in design, interfaces with other components etc… We are not guaranteed that the proof itself is correct

14. What is Testing? We could make sure our program does what it is supposed to do by proving the program is correct However, program correctness is the reverse of testing The goal of testing is to prove the existence of a fault “Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence” (Dijkstra, 1972)

15. Specifications Requirements DesignCode Environment Support DocumentationOther Requirements or Specifications Functionality HW interface SW interface User interface Functional description (Inter)-process Communications Data definition Module design Logic Description Error Checking Standards Logic Computation Data handling Module Interface/ Implementation Standards Test HW Test SW Integration SW Development tools Missing Unclear Wrong Changed Better Way Mode:Why? Type: What? Origin: Where? Hewlett-Packard’s fault classification

16. Views of Software-1 Textual   Measures: program lengths, frequency and occurrence of tokens Syntactic  Code treated as syntactic elements defined by grammar  Measures: statement counts, function calls, frequency of variable use Control flow  Analysis of program execution relative to execution order of program elements and control flow relations  Identify paths through system and to assist in testing

17. Views of Software-2 Data flow  Analysis of program relative to data access behavior and data flow relations  DF graphs used to examine variable definitions and references Computation flow  Program viewed as collection of computations treated as a trace of data states produced in response to input  Fault seeding and mutation analysis is used to analyze this view Functional   Functions denoted as input/output pairs

18. Types of Tests Performance testing  Determine the performance of the system such as execution time associated with various parts of the code Stress testing  Designed to break software module and determine strengths and limitations Functional testing  Exercise code with nominal inputs for which expected values are available  Structural testing  Aimed at exercising internal logic of software system 

19. Levels of Software Testing-1 Unit/Component testing  Verify implementation of software element  Ensure logic is complete and correct; works as designed  Integration testing  Combine software elements and test until the entire system has been integrated  Ensure design objectives satisfied  Trace each test to high-level design

20. Levels of Software Testing-2 System testing   Ensure software as a complete entity complies with operational requirements  Trace test to system requirements Acceptance testing  Determine if test results satisfy acceptance criteria of project stakeholder  Ensure objectives of stakeholders satisfied  Trace each test to stakeholder requirements Installation testing  Perform testing with application installed on its target platform

21. Overview: Test planning Design test cases Write test cases Test test cases Execute test cases Evaluate test result

22. Unit Testing Focus: elementary components of the software   Modules  Module packages Approaches (depends on testing goals)  Black-box testing  White-box testing

23. What is Black-box Testing? No access to the internal or logical structure Determine if given input produces expected output. Input Output

24. Advantage and Disadvantage Advantage: Disadvantage: in most real-world applications, impossible to test all possible inputs Need to define an efficient strategy to limit the drawback of black-box testing

25. Types of Black-box Testing Boundary value analysis Syntax-driven Decision-table Cause-effect

26. Equivalence Class Testing Partition test cases into classes such that: Every possible input belongs to one of the classes If we demonstrate a fault in the code for a given input, we should demonstrate a fault with any other input from the same class (with a high probability) Classes are identified by looking at boundary values for the variables of the application

27. Example 1 If (x > y) then S 1 else S 2 Equivalence classes for values of x and y: x ≤ y x > y

28. Example 2 The program specification states that the program accepts 4 to 10 inputs which are five-digit integers greater than 10,000. What are the partitions?

29. Example 2-Partitions 4 ≤ x ≤ 10 10,000 ≤ y ≤ 99,999 x < 4x > 10 y <10,000 y > 99,999 Number of input values Input values

30. Boundary Value Analysis Examines boundaries of equivalence classes Test cases are on or just to one side of the boundary of an equivalence class Consider (r 1, r 2 ) given as an input or output specification  Equivalence classes would yield values x such as: x ≤ r 1, r 1 < x < r 2, x ≥ r 2  Boundary value analysis: x = r 1 –1, x = r 1, x = r 1 +1, x = r 2 –1, x = r 2, x = r 2 +1, r 1 < x < r 2

31. Exercise Given the following method specification: /** Returns true iff key is contained in values. */ //@ requires values != null; //@ ensures \result == (\exist int i; i >= 0 && i < values.length; //@ values[i] == key); public static boolean contains(int[] values, int key) Do boundary analysis to select a reasonable set of test cases.

32. Answer values empty not empty key not in array key in array first last Test suite: (,, )

33. Syntax-Driven Testing Applicable to systems that are defined by a given grammar, e.g., formula interpretation, file input, compilers Generate test cases such that each production rule is applied at least once.

34. Example: Syntax-Driven Testing ::= + | - | ::= * | / | ::= | ( ) ::= a|b|c…|z Test Cases: a + b * c ((w – v)/x) – y d + e +.. + t + u + z

35. Example: Syntax-Driven Testing ::= + | - | ::= * | / | ::= | ( ) ::= a|b|c…|z Test Cases: a + b * c ((w – v)/x) – y d + e +.. + t + u + z

36. Example: Syntax-Driven Testing ::= + | - | ::= * | / | ::= | ( ) ::= a|b|c…|z Test Cases: a + b * c ((w – v)/x) – y d + e +.. + t + u + z

37. Example: Syntax-Driven Testing ::= + | - | ::= * | / | ::= | ( ) ::= a|b|c…|z Test Cases: a + b * c ((w – v)/x) – y d + e +.. + t + u + z

38. Example: Syntax-Driven Testing ::= + | - | ::= * | / | ::= | ( ) ::= a|b|c…|z Test Cases: a + b * c ((w – v)/x) – y d + e +.. + t + u + z

39. Decision Table-Based Testing Applicable for requirements in if-then  if C 1 and C 2 and…and C n then A k Number of columns: 2ⁿ (based on conditions) Number of rows: n + m, where m is number of actions For each set of conditions, there is a corresponding action

40. Example: Text Editor Functions: copy, paste, boldface, underline, and select. Copy selected00001111 Paste selected00110011 Text Highlighted01010101 Copy text1 Paste text1

41. Cause-Effect Graphs Decision table-based drawback  Test all possible cases  Results in combinatorial explosion Cause-effect graphs  are inputs  are outputs  Create cause-effect graph  Identify dependencies limiting the number of tested combinations

42. Example Consider an ATM Causes: o C1: Command is credit o C2: Command is debit o C3: Account number is valid o C4: Transaction amount is valid Effects: o E1: Print “invalid command” o E2: Print “invalid account number” o E3: Print “debit amount not valid” o E4: Debit account o E5: Credit account

43. Example (Cont.) The graph tells which causes affect which effects. C1 C2 C3 C4 E1 E2 E3 E4 E5 \/ /\ ¬ ¬ ¬ Causes: C1: Command is credit C2: Command is debit C3: Account number is valid C4: Transaction amount is valid Effects: E1: Print “invalid command” E2: Print “invalid account number” E3: Print “debit amount not valid” E4: Debit account E5: Credit account

44. Example C1 C2 C3 C4 E1 E2 E3 E4 E5 or and ¬ ¬ ¬ Graph determines the Test cases by tracing Back the truth values of Selected cause. C101XX1 C20X11X C3X0111 C4XX011 E1 ` 10000 E201000 E300100 E400010 E500001

45. Example (Cont.) Graph determines the test cases by tracing back the truth values of selected cause. C101XX1 C20X11X C3X0111 C4XX011 E110000 E201000 E300100 E400010 E500001 C1 C2 C3 C4 E1 E2 E3 E4 E5 \/ /\ ¬ ¬ ¬ For E1: (C1  C2)   E1  (C1  C2)    E1  C1   C2  E1

46. Example (Cont.) Graph determines the test cases by tracing back the truth values of selected cause. C101XX1 C20X11X C3X0111 C4XX011 E110000 E201000 E300100 E400010 E500001 C1 C2 C3 C4 E1 E2 E3 E4 E5 \/ /\ ¬ ¬ ¬ For E2: C1   C3