1. Chapter 23 전통적인 애플리케이션 테스팅 Testing Conventional Applications 임현승 강원대학교 Revised from the slides by Roger S. Pressman and Bruce R. Maxim for the book “Software Engineering: A Practitioner’s Approach, 8/e”

2. What is a “Good” Test? A good test has a high probability of finding an error A good test is not redundant. A good test should be “best of breed”, among similar tests, one that has the highest likelihood of uncovering a whole class of errors A good test should be neither too simple nor too complex, and should be executed separately 2

3. Test Case Design "Bugs lurk in corners and congregate at boundaries..." Boris Beizer OBJECTIVE CRITERIA CONSTRAINT to uncover errors in a complete manner with a minimum of effort and time 3

4. Exhaustive Testing There are 10 14 possible paths! If we execute one test per millisecond, it would take 3,170 years to test this program!! loop < 24 X 4

5. Selective Testing loop < 24 X Selected path 5

6. Internal and External Views Any engineered product (and most other things) can be tested in one of two ways: Black-box testing: knowing the specified function that a product has been designed to perform, tests can be conducted that demonstrate each function is fully operational while at the same time searching for errors in each function; White-box testing: knowing the internal workings of a product, tests can be conducted to ensure that "all gears mesh," that is, internal operations are performed according to specifications and all internal components have been adequately exercised. 6

7. Software Testing Methods Strategies white-box methods black-box methods 7

8. Software Testing Finding the largest number among three positive numbers A, B, and C Black box Black-box testing White-box testing 8

9. Software Testing Black-box testing IDTest cases 1 A is the largest number… 2 B is the largest number… 3 C is the largest number… 4 A is a negative number… 5 A and B are the same number… 6 B is a prime number… 7 … 9

10. Software Testing White-box testing IDTest cases 1(1->2->3->5->6->8) 2(1->2->3->5->7->8) 3(1->2->4->5->6->8) 4(1->2->4->5->7->8) 10

11. White-Box Testing Also known as glass-box testing and structural testing Uses the control structure to derive test cases.... our goal is to ensure that all statements and conditions have been executed at least once... 11

12. Why Cover? Logic errors and incorrect assumptions are inversely proportional to a path's execution probability We often believe that a path is not likely to be executed; in fact, reality is often counter intuitive Typographical errors are random; it's likely that untested paths will contain some errors 12

13. Basis Path Testing First proposed by Tom McCabe [McC76] The test-case designer derives a logical complexity measure (e.g., cyclomatic complexity) of a procedural design (e.g., flow graph) and uses this measure as a guide for defining a basis set of execution paths (i.e., consisting of independent paths). Test cases derived to exercise the basis set are guaranteed to execute every statement in the program at least once during testing. 13

14. Flow Graph Representation of control flow of a program Depicts logical control flow 14

15. Flowchart and Flow Graph In (b) each flow graph node represents one or more procedural statements. A sequence of process boxes and a decision diamond can map into a single node. An edge must terminate at a node, even if the node does not represent any procedural statements. Areas bounded by edges and nodes are called regions. 15

16. Compound Conditions A separate node is created for each of the conditions a and b in the statement IF a OR b. Each node that contains a condition is called a predicate node. 16

17. Cyclomatic Complexity A software matric that provides an upper bound for the number of tests that must be conducted to ensure that all statements have been executed at least once. Cyclomatic complexity V(G) for a flow graph G is defined as: Number of regions of G V(G) = E – N + 2 where E and N are respectively the number of edges and nodes V(G) = P + 1 where P is the number of predicate nodes For the graph on the right, V(G) = 4 17

18. Cyclomatic Complexity A number of industry studies have indicated that the higher V(G), the higher the probability or errors. number of modules modules in this range are more error prone V(G) 18

19. Independent Paths Any path through the program that introduces at least one new set of processing statements (i.e., a new node) or a new condition (i.e., a new edge) We derive independent paths from cyclomatic complexity. Since V(G) = 4, there are four paths: Path 1: 1-11 Path 2: 1-2-3-4-5-10-1-11 Path 3: 1-2-3-6-8-9-10-1-11 Path 4: 1-2-3-6-7-9-10-1-11 For example, the path 1-2-3-4-5-10-1-2-3-6-8-9-10-1-11 is not considered to be an independent path. Finally, we derive test cases to exercise these paths. 19

20. Notes on Basis Path Testing You don't need a flow graph (or flowchart), but the picture will help when you trace program paths Count each simple logical test, compound tests count as two or more Basis path testing should be applied to critical modules 20

21. Deriving Test Cases Summarizing: 1.Using the design or code as a foundation, draw a corresponding flow graph. 2.Determine the cyclomatic complexity of the resultant flow graph. 3.Determine a basis set of linearly independent paths. 4.Prepare test cases that will force execution of each path in the basis set. Data should be chosen so that conditions at the predicate nodes are appropriately set as each path is tested. 21

22. Example: Procedure average 22

23. Complexity and Independent Paths The cyclomatic complexity V(G): V(G) = 6 regions V(G) = 17 edges – 13 nodes + 2 = 6 V(G) = 5 predicate nodes + 1 = 6 A basis set of independent paths: Path 1: 1-2-10-11-13 Path 2: 1-2-10-12-13 Path 3: 1-2-3-10-11-13 Path 4: 1-2-3-4-5-8-9-2-… Path 5: 1-2-3-4-5-6-8-9-2-… Path 6: 1-2-3-4-5-6-7-8-9-2-... The ellipsis (…) indicates any path through the remainder of the control structure is acceptable 23

24. Graph Matrices Useful for developing a software tool that assists in basis path testing. A graph matrix is a square matrix whose size (i.e., number of rows and columns) is equal to the number of nodes on a flow graph. Each row and column corresponds to an identified node, and matrix entries correspond to connections (an edge) between nodes. By adding a link weight to each matrix entry, the graph matrix can become a powerful tool for evaluating program control structure during testing. E.g., the probability that a link (edge) will be executed. E.g., the memory required during traversal of a link 24

25. An Example Graph Matrix 25

26. Control Structure Testing Basis path testing is simple and highly effective, but is not sufficient in itself. Condition testing — a test-case design method that exercises the logical conditions contained in a program module Data flow testing — selects test paths of a program according to the locations of definitions and uses of variables in the program Loop testing — focuses exclusively on the validity of loop constructs 26

27. Loop Testing 27

28. Testing Simple Loops The following set of tests can be applied to simple loops, where n is the maximum number of allowable passes 1.skip the loop entirely 2.only one pass through the loop 3.two passes through the loop 4.m passes through the loop m < n 5.(n-1), n, and (n+1) passes through the loop 28

29. Testing Nested Loops 1.Start at the innermost loop. Set all outer loops to their minimum iteration parameter values. 2.Conduct simple loop tests for the innermost loop while holding the outer loops at their minimum iteration parameter values. Add other tests for out-of-range or excluded values. 3.Work outward, conducting tests for the next loop, but keeping all other outer loops at minimum values and other nested loops to “typical values”. 4.Continue until all loops have been tested. 29

30. Testing Concatenated Loops Concatenated loops if the loops are independent of one another then treat each as a simple loop else treat as nested loops endif For example, if the final loop counter value of loop 1 is used to initialize loop 2, then they are dependent. Unstructured loops Whenever possible, this class of loops should be redesigned. 30

31. Black-Box Testing Also known as behavioral testing or functional testing Focuses on the functional requirements of the software and derives sets of input conditions that will fully exercise all functional requirementsrequirementsevents input output 31

32. Black-Box Testing Tests are designed to answer the following questions: How is functional validity tested? How is system behavior and performance tested? What classes of input will make good test cases? Is the system particularly sensitive to certain input values? How are the boundaries of a data class isolated? What data rates and data volume can the system tolerate? What effect will specific combinations of data have on system operation? Approaches Graph-based testing methods Equivalence partitioning Boundary value analysis Orthogonal array testing 32

33. Graph-Based Methods To understand the objects (in a broad sense, e.g., data objects, modules, OO elements) that are modeled in software and the relationships that connect these objects Begins by creating a graph of important objects and their relationships and then devising a series of tests that will cover the graph so that each object and relationship is exercised and errors are uncovered. (a) graph notation, (b) example for a word-processing app 33

34. Equivalence Partitioning Divides the input domain of a program into equivalent classes of data from which tests cases can be derived (using representative values of each equivalence class). An equivalence class is a set of objects that are linked by a reflexive, symmetric and transitive relation. From the input domain, we define valid and invalid equivalence classes. Valid data: user supplied commands, responses to system prompts, file names, computational data— physical parameters, bounding values, initiation values—, output data formatting, responses to error messages, graphical data (e.g., mouse picks) Invalid data: data outside bounds of the program, physically impossible data 34

35. Example: Equivalence Partitioning User requirements We have a test whose score is between 0 and 100. Given a test score, display the corresponding grade as follows: ScoreGrade 90 points – 100 pointsA 80 points – 89 pointsB 70 points – 79 pointsC 0 points – 69 pointsF 35

36. Example: Equivalence Partitioning Invalid Partition Valid partition Test cases 12345 Score range Invalid0 – 6970 – 7970 – 8990 – 100 Input value -1050758595 Expected result Warning message FCBA Actual result Warning message FCBA 36

37. Boundary Value Analysis (BVA) A complementary test-case design technique to equivalence partitioning As a greater number of errors occurs at the boundaries of the input domain, BVA selects test cases at the “edges” of a equivalence class. 37

38. Example: BVA Continuing from the equivalence partitioning example Invalid Partition Valid partition Invalid Partition Test cases 12345 Score range InvalidValid Invalid Input value 090100101 Expected result Warning message FAA Actual result Warning message FAA 38

39. Orthogonal Array Testing Used when the number of input parameters is small and the values that each of the parameters may take are clearly bounded E.g., three input parameters taking on three discrete values each (3 3 = 27 possible test cases) Orthogonal array testing provides good test coverage with far fewer test cases than the exhaustive strategy 39

40. L9 Orthogonal Array For each independent variable, all the input value settings appear an equal number of times (input value 1, 2, and 3 appear three times) If we normalize the input value levels, then the inner product of column vectors for variables is zero (thus orthogonal). E.g., by normalizing 1 to -1, 2 to 0, and 3 to 1, the inner product of X and Z column vectors is: (-1 x 1 + (-1) x 0 + (-1) x (-1)) + (0 x 0 + 0 x (-1) + 0 x 1) + (1 x (-1) + 1 x 1 + 1 x 0) = 0 Test case Test parameters XYZ 1113 2122 3131 4212 5221 6233 7311 8323 9332 40

41. Model-Based Testing A black-box testing technique consisting of the following steps: 1.Analyze an existing behavioral model (e.g., UML state diagram) for the software or create one. Recall that a behavioral model indicates how software will respond to external events or stimuli. 2.Traverse the behavioral model and specify the inputs that will force the software to make the transition from state to state. The inputs will trigger events that will cause the transition to occur. 3.Review the behavioral model and note the expected outputs as the software makes the transition from state to state. From input and output pairs, test cases are derived. 4.Execute the test cases. 5.Compare actual and expected results and take corrective action as required. 41

42. Summary The primary objective for test-case design is to derive a set of tests that have the highest likelihood for uncovering errors in software. White-box testing focuses on the program control structure. Test cases are derived to ensure that all statements in the program have been executed at least once during testing and all logical conditions have been exercised. E.g., basis path testing, control structure testing such as loop testing Black-box testing are designed to validate functional requirements without regard to the internal workings of a program. E.g., equivalence partitioning, boundary value analysis, orthogonal array testing, model-based testing 42

43. Backup Slides 43

44. Testability The following characteristics lead to testable software. Operability—the better it works, the more efficiently it can be tested Observability—what you see is what you test Controllability—the better we can control the software, the more the testing can be automated and optimized Decomposability—the software system is built from independent modules that can be tested independently Simplicity—the less there is to test, the more quickly we can test it Stability—the fewer the changes, the fewer the disruptions to testing Understandability—the more information we have, the smarter we will test 44

45. Data Flow Testing The data flow testing method [Fra93] selects test paths of a program according to the locations of definitions and uses of variables in the program. Assume that each statement in a program is assigned a unique statement number and that each function does not modify its parameters or global variables. For a statement with S as its statement number DEF(S) = {X | statement S contains a definition of X} USE(S) = {X | statement S contains a use of X} A definition-use (DU) chain of variable X is of the form [X, S, S'], where S and S' are statement numbers, X is in DEF(S) and USE(S'), and the definition of X in statement S is live at statement S' 45

46. Comparison Testing Used only in situations in which the reliability of software is absolutely critical (e.g., human-related systems) Separate software engineering teams develop independent versions of an application using the same specification Each version can be tested with the same test data to ensure that all provide identical output Then all versions are executed in parallel with real-time comparison of results to ensure consistency 46

47. Software Testing Patterns Testing patterns are described in much the same way as design patterns (Chapter 12). Example: Pattern name: ScenarioTesting Abstract: Once unit and integration tests have been conducted, there is a need to determine whether the software will perform in a manner that satisfies users. The ScenarioTesting pattern describes a technique for exercising the software from the user’s point of view. A failure at this level indicates that the software has failed to meet a user visible requirement. [Kan01] 47