1. Slide (Ch.8) 1 Tian: Software Quality Engineering Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement Jeff Tian Chapter 8. Coverage and Usage Testing Based on Checklists and Partitions Checklist-Based Testing Partitions and Partition Testing Wiley-IEEE/CS Press, 2005Slides V2 (2007) In this chapter, we describe several formal test techniques whose models are based on simple structures such as lists and partitions.

2. (Ch.8) 2 Tian: Software Quality EngineeringSlide Checklists for Testing Ad hoc testing:. "run-and-observe". How to start the run?. Areas/focuses of "observations"?. Implicit checklists may be involved. Explicit checklists:. Function/features (external). Implementation (internal). Standards, etc.. Mixed or combined checklists Wiley-IEEE/CS Press, 2005Slides V2 (2007) The simplest form of testing is to start running the software and make observations, in the hope that it is easy to distinguish between expected and unexpected behavior. Along the same line, software systems are sometimes tested in a similar way to see if some specific problems can be observed or if specific operational condition and input can be handled without resorting to some systematic method. We call these forms of simple and informal testing ad hoc testing. Ad hoc testing is also called random testing in some literature. When ad hoc testing is used repeatedly to test a software system, the testers then need to keep track of what has been done, in order to avoid wasting their time repeating the same tests. In addition, an informal “to- do” list is commonly used to track what needs to be done. Such to-do lists can be a physical list, an online list, or just a mental list. The use of these informal lists forms the rudimentary and implicit checklists, where each item can be checked off when corresponding testing was performed, until every item on the lists is checked off.

3. Slide (Ch.8) 3 Tian: Software Quality Engineering Function Checklists Function/feature (external) checklists:. Black-box in nature. List of major functions expected Example: Table 8.1 (p.105) abnormal termination backup and restore communication co-existence file 1/0 gateway index management installation logging and recovery locking migration stress........................ Wiley-IEEE/CS Press, 2005Slides V2(2007) Functional (black-box) checklists at different levels of abstraction, ranging from major high-level functions for the overall system to specific low-level functions of an individual unit or components. Checklists of system elements (white-box) at different levels, ranging from sub-systems and modules at the high level to individual statements or data items at the low level. Checklists of various structures or features that cut through different elements, such as list of call-pairs, consumers-and- producers for certain resources, modules sharing some common data, etc. These lists are concerned with implementation structures, therefore can be classified as white- box checklists as well.

4. Slide (Ch.8) 4 Tian: Software Quality Engineering Implementation Checklists Implementation (internal) checklists:. White-box in nature. At different levels of abstraction – e.g., lists of modules/components/etc. – statement coverage as covering a list Related: cross-cutting features/structures:. Multiple elements involved.. Examples: call-pairs, diff. parts that cooperate/collaborate/communicate/etc. 0ther checklists:. related to certain properties – e.g., coding standards,. hierarchical list, e.g., refined Table 8.1 Wiley-IEEE/CS Press, 2005Slides V2 (2007) The most commonly used form of multiple checklists is the hierarchical checklist, with each item in a higher-level checklist expandable to a full checklist at a lower level until we stop at a level of detail deemed enough by some criteria.

5. Slide (Ch.8) 5 Tian: Software Quality Engineering Other Checklists Wiley-IEEE/CS Press, 2005Slides V2 (2007) In addition to the use of individual checklists and hierarchical sets of checklists above, various checklists can also be combined in other forms. For example, a coding standard checklist can be combined with a component or unit checklist to make sure that each component follows all the recommended practice in the coding standards. This combination of two checklists forms a two dimensional checklist, or a table with each of its entries to be checked off, as illustrated in Table 8.2. Similarly, higher dimensional checklists can also be used. In addition, mixed checklists that mix the direct list combinations and hierarchies are also possible, but such mixed checklists should be used with care because of the possible confusion involved. Table 8.2

6. (Ch.8) 6 Tian: Software Quality EngineeringSlide Checklists: Assessment Key advantage: simplicity. Possible drawbacks of checklists:. Coverage: need to fill "hole".. Duplication: need to improve efficiency.. Complex interactions not modeled.. Root cause: complexity – contributing to all 3 problems above. Possible solutions:. specialized checklists ⇒ partitions.. alternatives to checklists: FSMs. Wiley-IEEE/CS Press, 2005Slides V2 (2007) One of the important characteristics of these checklists is that they are typically not very specific, particularly those high-level ones. For example, a high-level functional checklist typically lists the major functions to be performed by a software product, but the list items are hardly detailed enough for testers to start a specific test run. The translation from this testing model to the test cases and then to test runs is not a simple matter. It usually involves experienced testers setting up the system and testing environment to execute specific test cases. In addition, repetition of the same test case in a later test run can only be guaranteed with this additional information about setup and environment, but not deduced from the checklist item itself.

7. Slide (Ch.8) 7 Tian: Software Quality Engineering Checklists to Partitions Wiley-IEEE/CSPress, 2005Slides V2 (2007) In the software testing context, partitions and partition-based testing are a special type of checklists and checklist-based testing. Partitions possess some desirable properties that set them apart from general checklists. Therefore, they can be used to better support testing and address some specific problems associated with general checklists in two areas: Better coverage because a partition is collectively exhaustive, and better efficiency because of the use of mutually exclusive partitions. Example of such cases include a decision based on a few input variables of the logical type, each can take a true (T) or false (F) value. Then the number of input combinations would be 2 n for n such input variables. Complete coverage of input combinations are possible as long as n remains small. However, even for simple input-output relations, such as a simple program calculating the root of quadratic equations in the form of ax 2 + bx + c = 0, with the solution for the root r to be: The number of input combinations between the three input variables a, b, and c could be huge. For example, if each of these three variables is represented by a 32 bit floating point number, the number of all possible input value combinations is then

8. Slide (Ch.8) 8 Tian: Software Quality Engineering Partitions: Ideas and Definitions Wiley-IEEE/CS Press, 2005Slides V2 (2007)

9. Slide (Ch.8) 9 Tian: Software Quality Engineering Partition-Based Testing Different types of partition definitions:. membership based partition definitions. properties/relations used in definitions. combinations Basic idea of partition-based testing:. membership/equivalence-class analysis ⇒ defining meaningful partitions. sampling from partitioned subsets for different types of partitions Wiley-IEEE/CS Press, 2005Slides V2(2007) Partitions of some product entities, such as external functions (black-box view), system components (white-box view), etc. The definition of such partitions are typically a simple “member” relation in sets, that is, x ε S for x as a member of the set S. As a concrete example, whether a component belongs to a sub-system or not is easy to decide. The key is to ensure the partitioned subsets truly form a partition of the original set of all entities. That is, they are mutually exclusive and collectively exhaustive.

10. Slide (Ch.9) 10 Tian: Software Quality Engineering Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement Jeff Tian Chapter 9.Boundary Testing Wiley-IEEE/CS Press, 2005 Slides V2 (2007)

11. Slide (Ch.9) 11 Tian: Software Quality Engineering An Overview Wiley-IEEE/CS Press, 2005Slides V2 (2007) Boundary Value Testing (BVT) A type of “black box” functional testing The program is viewed a mathematical “function” The program takes inputs and maps some outputs The internal of the program itself is not considered A technique to generate test cases via considering the inputs to the program The rationale for this focus is that there are experiences from the past, which indicate that errors tends to occur at the “extreme” points Input data Loop iteration Output fields

12. Slide (Ch.9) 12 Tian: Software Quality Engineering Boundary Testing: Overview Wiley-IEEE/CS Press, 2005Slides V2 (2007) A simple Example Consider a program that reads the “age” of students in SSQ and computes the average students age of the class Input (s)  Program  Output (average age) What type of test data would you input to test this program Input (s) to Program 1. How many input data? The answer is  some integer n > 0 2. what values should each input age be Try some typical values 25, 27 or 40 Try some atypical values 150 or 700 When we try the atypical age or wrong age  program may not “handle” it properly Strange answer, but no program termination etc.

13. Slide (Ch.9) 13 Tian: Software Quality Engineering Boundary Testing: Overview Wiley-IEEE/CS Press, 2005Slides V2 (2007) Example: Program Average Number of input data n>0 Lower value is n=1 How big n is? What about 100,000 (just assume) Need to set upper boundary like lower boundary is 1 Input composed of only typical ages Look at the output average; the average is computed either Correctly Incorrectly Input composed atypical or wrong ages Boundary required So that atypical age is other than that boundary (age cap is 1 to 130, for example) Boundaries of the Input 1.At min2. Immediately above min 3. Between min and max (normal)4. Immediately below max 5. At Max

14. Slide (Ch.9) 14 Tian: Software Quality Engineering Boundary Testing: Overview Wiley-IEEE/CS Press, 2005Slides V2 (2007) Single Fault or Independent Fault For the example above, there are 2 “distinct” inputs that are assumed to be independent (single fault) for each other – even though they are somewhat related. Input: n Input: age If they are independent of each other  we can start with looking at 5+5 = 10 sets, but Won’t need all 10 of them Coverage of Input data: nCoverage of Input data: age 1.n=1; age = whatever1. n=3; age = 1, 20, 55 2.n =2; age = whatever2. n = 3; age = 2, 20, 55 3.n =3; age = whatever3. n = 3; age =3, 20, 55 4.n = 99999; age = whatever4. n =3; age = 3, 20, 129 5.n = 100000; age = whatever 5. n = 3; age = 3, 20, 130; Note: There needs to be only 9 test cases for 2 independent variables or inputs In general, there will be (4Z+1) test cases for Z independent inputs

15. Slide (Ch.9) 15 Tian: Software Quality Engineering Boundary Testing: Overview Wiley-IEEE/CS Press, 2005Slides V2 (2007) But n and age are ‘little related’ n dictates the no. of input data that is allowed, but not just the values that age input may take on… For the previous example we would have to further consider the situation where n=x, besides n being between 1 to 100000; but the number of input data is: < x = x > x Thus we need to add more test cases. Other Limitation of Boundary Values What would we do with Boolean variables? True False What about non-numerical variable where the values may be text for an input field? e.g. “163, Garden West Karachi” Do you consider length of the input as a boundary? What about drop-down windows? Is there any boundary?

16. Slide (Ch.9) 16 Tian: Software Quality Engineering Boundary Testing: Overview Wiley-IEEE/CS Press, 2005Slides V2 (2007) Extended Boundary Value (Robustness) Testing This is just an extension of the Boundary values to include: Less than min Greater than max There are 7 cases or values to worry about for each independent variable input The testing of robustness is really a Test of “error” handling Do we anticipate the error situation? Do we issue informative error message? Do we allow some kind of recovery from the error? Note: There need to be only 13 test cases for 2 independent variables or inputs In general, there will be (6n+1) test cases for n independent input Worst-Case testing for Non-independent Variable If the input variable are not independent  need to test all possible combinations of values that the variable may take on For boundary value testing  each of the 5 possible values of variable must iterate through the 5 possible values of the other variable(s) (i.e. n  5 n ) For Robustness Testing  each of the 7 possible values of a variables must iterate thorugh the 7 possible values of the other variable(s) (i.e. n  7 n )

17. Slide (Ch.9) 17 Tian: Software Quality Engineering Boundary Testing: Overview Wiley-IEEE/CS Press, 2005Slides V2 (2007) Hierarchy Boundary value testing on n input:4n +1 Robustness testing of n input:6n+1 Worst case for boundary value: 5 n Worst case for robustness: 7 n Note: Boundary value is a subset of Robustness Worst case for boundary value is subset of worst case of robustness Special Value and Random Testing Special value Testing Based on experience, special knowledge of industry etc. Costly to find the industry “expert” Random Value Testing Based on some random umber generation Generate values within bounds of the boundary or worst case The value of random test has not been clearly justified