Lecture 2 May 21, 2013 Software Testing Research Dr. Sergiy Vilkomir 2013 REU Program at ECU Software Testing - Foundations, Tools, and Applications

Testing can show that defects are present, but cannot prove that there are no defects. Exhaustive testing is impossible 2/38 A restricted number of test cases should be selected There are different ways (testing methods/strategies/criteria/approaches) to select test cases There are no method that guarantee that software is 100% correct

Pair-Wise Pair-Wise: A value of each level for each factor must be combined with a value from every levels for each other factor. In other words, we must cover all possible pairs of values. Pair-wise testing provides a small set of test cases. It is a practical alternative to testing all combinations. Why? One test case covers several pairs ABC Test case greenredblue greenred covers greenblue pairs: redblue

Pair-Wise Pair-Wise: A value of each level for each factor must be combined with a value from every levels for each other factor. ABC red green redgreenred blue redblue green blue red greenred green blue greenbluegreen Exercise: Choose one pair and find a test case which covers this pair

Suppose that a program to be tested requires 3 inputs, one corresponding to each input variable. Each variable can take only one of two distinct values. Considering each input variable as a factor, the total number of factor combinations is 2 3. Let X, Y, and Z denote the three input variables and {X1, X2}, {Y1, Y2}, {Z1, Z2} their respective sets of values. All possible combinations of these three factors follow. Pairwise designs: Example

Now suppose we want to generate tests such that each pair appears in at least one test. There are 12 such pairs: (X1, Y1), (X1, Y2), (X1, Z1), (X1, Z2), (X2, Y1), (X2, Y2), (X2, Z1), (X2, Z2), (Y1, Z1), (Y1, Z2), (Y2, Z1), and (Y2, Z2). The following four combinations cover all pairs: The above design is also known as a pairwise design. It is a balanced design because each value occurs exactly the same number of times. There are several sets of four combinations that cover all 12 pairs. Pairwise designs: Reducing the combinations

Pairwise combinatorial testing Pairwise combination (instead of exhaustive) –Generate combinations that efficiently cover all pairs –Rationale: most failures are triggered by single values or combinations of a few values. Covering pairs reduces the number of test cases, but reveals most faults

Example: Display Control Display ModeLanguageFontsColorScreen size full-graphicsEnglishMinimalMonochromeHand-held text-onlyFrenchStandardColor-mapLaptop limited- bandwidth SpanishDocument- loaded 16-bitFull-size PortugueseTrue-color The total number of combinations – 432 (3x4x3x4x3) test cases

Pairwise combinations: 17 test cases LanguageColorDisplay ModeFontsScreen Size EnglishMonochromeFull-graphicsMinimalHand-held EnglishColor-mapText-onlyStandardFull-size English16-bitLimited-bandwidth-Full-size EnglishTrue-colorText-onlyDocument-loadedLaptop FrenchMonochromeLimited-bandwidthStandardLaptop FrenchColor-mapFull-graphicsDocument-loadedFull-size French16-bitText-onlyMinimal- FrenchTrue-color--Hand-held SpanishMonochrome-Document-loadedFull-size SpanishColor-mapLimited-bandwidthMinimalHand-held Spanish16-bitFull-graphicsStandardLaptop SpanishTrue-colorText-only-Hand-held Portuguese--MonochromeText-only PortugueseColor-map-MinimalLaptop Portuguese16-bitLimited-bandwidthDocument-loadedHand-held PortugueseTrue-colorFull-graphicsMinimalFull-size PortugueseTrue-colorLimited-bandwidthStandardHand-held Exercise: Choose one pair and find a test case which covers this pair

Number of test cases Example 1Example 2Example3 Number of factors235 Number of all combinations Number of pairwise test cases4917 System 1System 2 Number of factors61 29 – 2 levels 17 – 3 levels 15 – 4 levels – 2 levels 39 – 3 levels 1 – 4 levels Number of all combinations7.4 x x Number of pairwise test cases4128

In-class exercise Consider software with input parameters: –Dest: London, Paris, Chicago, NY –M: 1000, 2000, 3000 –Direct: Y, N Create test cases according to pair-wise approach. Number of all possible combinations? Number of pairs? Minimal number of test cases? 4 x 3 x 2 = 24 Dest - M: 4 x 3 = 12 Dest – Direct: 4 x 2 = 8 M – Direct: 3 x 2 = 6 Total: 26 Start with factors with max numbers of levels: Dest - M: 4 x 3 = 12 We need min 12 test cases

In-class exercise –Dest: London, Paris, Chicago, NY –M: 1000, 2000, 3000 –Direct: Y, N DestMDirect t1L t2L t3L t4P t5P t6P t7C t8C t9C t10N t11N t12N

In-class exercise –Dest: London, Paris, Chicago, NY –M: 1000, 2000, 3000 –Direct: Y, N DestMDirect t1L1 t2L2 t3L3 t4P1 t5P2 t6P3 t7C1 t8C2 t9C3 t10N1 t11N2 t12N3

In-class exercise –Dest: London, Paris, Chicago, NY –M: 1000, 2000, 3000 –Direct: Y, N DestMDirect t1L1Y t2L2N t3L3- t4P1- t5P2Y t6P3N t7C1N t8C2- t9C3Y t10N1Y t11N2N t12N3-

t-Wise t-Wise: A value of each level for each group of t factors must be combined. Sometimes: t-way (3-way, 4-way, etc) A natural extension is to require combinations of t values instead of 2 t-wise is expensive and benefits are not clear Example: 4-way 30 parameters 5 values each 3,800 test cases (could be too many)

Combinatorial Methods in Software Testing National Institute of Standards and Technology (NIST) 16/18 Presentation by Rick Kuhn, (NIST) at ECU, March 22, RG/seminars.html

Software Testing Study Software Testing is a part of ECU MSc SE program SENG 6265 Foundations of Software Testing SENG 6270 Software Verification and Validation

Software Testing Research Research - ? New Scientific approach: investigation, analysis, comparison, justification, etc. Publications

Project: Testing of Mobile Applications Android Development Tools Monkeyrunner tools from Android SDK Case study (testing) Robotium Calabash-Android Comparison, analysis Pair-wise approach for mobile testing Cloud Testing of Mobile Systems (CTOMS) framework