CS527: Advanced Topics in Software Engineering (Software Testing and Analysis) Darko Marinov September 18, 2008

Recent Schedule Sep 9: Testing refactoring engines Sep 11: Static analysis Sep 16: Model checking TODAY: Sep 18: Test prioritization Sep 23&25: More on testing –Let me know if you have some preferred topic Project proposal (more info on mailing list) –Your initial project idea will be due on Sep 25 –Proposal will be due the week after

Report Feedback Vilas and I will send you by Monday a summary about the reports we received –My spam filter caught some reports, so we should double-check we got all your reports –We’ll also comment on the quality of reports –These papers were from various areas –Your projects will focus on one area

Paper Today Empirical Studies of Test Case Prioritization in a JUnit Testing Environment by Hyunsook Do, Gregg Rothermel, Alex Kinneer (ISSRE 2004)Empirical Studies of Test Case Prioritization in a JUnit Testing Environment

Brief Summary Motivation: regression testing is important for finding bugs Problem: regression testing is slow Solution: prioritize tests to find bugs faster Evaluation: JUnit tests for 4 Java programs Results: Prioritization helps

Some Terminology Regression testing Test: test case, test input, test data… Test suite: test set, test class… Test (case) prioritization Test (suite) minimization (Regression) test selection Code coverage

Research Questions RQ1: Can test case prioritization improve the rate of fault detection of JUnit test suites? RQ2: How do the three types of information and information use that distinguish prioritization techniques (type of coverage information, use of feedback, use of modification information) impact the effectiveness of prioritization techniques? RQ3: Can test suite granularity (the choice of running testclass level JUnit tests versus test- method level tests) impact the cost-effectiveness of prioritization techniques?

Independent Variables Prioritization technique –Untreated, random, optimal –Block-total, block-addtl –Method-total, method-addtl –Method-diff-total, method-diff-addtl Test Suite Granularity –Class-level –Method-level

Dependent Measure APFD = 1 – (TF1 + … + TFm)/nm + 1/2n n test cases m faults TFi is the index of the first test case to find fault i Why is this between 0 and 100%?

Threats to Validity Internal –Are there bugs in experimental tools? –Are seeded faults the same as real? External –Are programs representative? –Is process representative? Construct –Is APFD an appropriate measure?

Brief Summary of Results RQ1: Can test case prioritization improve the rate of fault detection of JUnit test suites? –Yes RQ2: How do the three types of information and information use that distinguish prioritization techniques (type of coverage information, use of feedback, use of modification information) impact the effectiveness of prioritization techniques? –Coverage little, feedback lot, modification little RQ3: Can test suite granularity (the choice of running testclass level JUnit tests versus test- method level tests) impact the cost-effectiveness of prioritization techniques? –Yes

Questions for Discussion (1) Why no significant improvement when prioritizing on code modification? [DH] Why is test-class level not as good as test- method? [ST] How much the results vary based on how one writes a JUnit test class? [YL] Why random better than untreated? [WX] What new techniques would you propose? How prioritize automatically generated tests?

Questions for Discussion (2) What is hot topic in testing: regression test selection, test suite minimization, test case prioritization, automatic test generation...? What other prioritization techniques exist? How does test case prioritization fit in TDD? Is the extra effectiveness of prioritization techniques worth the extra cost (compared to random ordering)?

Questions for Discussion (3) How to apply this in the real world if we don’t know when all bugs have been found? How Galileo framework works (more detail)? –Questions like this can always lead to a project How to prioritize in integration testing? How to prioritize based on, say, complexity or criticality of test units? What are other regression test performance metrics besides APFD?