CS527 Topics in Software Engineering (Software Testing and Analysis) Darko Marinov September 16, 2010

Schedule First few lectures to help you select projects –Testing: ReAssert, UDITA, Pex, Randoop –Model checking: JPF, CHESS –Today: Regression testing, note: survey paper –Sep 21: MC (static analysis) –Sep 23: REST (GUI testing) –Sep 28: Analysis of code comments? –Testing distributed or real-time systems? –From Oct 7: Your presentations –Choose a date only on this page

Projects By far the most important part of the course –Meet or Sandro and/or me –Our discussion can help you select project/paper You need to explore some topic in depth –“I don't know anything, but I do know that everything is interesting if you go into it deeply enough” Richard Feyman quotequote Mostly for I2CS students: Can use the technology from your work or learn some new –Need to be able to show us some results –Preferably work in groups of 2-3 students

Paper Today Regression Testing Minimization, Selection and Prioritization: A Survey by Shin Yoo and Mark Harman (STVR 2010)Regression Testing Minimization, Selection and Prioritization: A Survey Survey paper –Presents a broad summary of an area –Regression testing Test Suite Minimization Test Case Selection Test Case Prioritization Follows: brief overview, projects, questions

Minimization t1t1 t2t2 tntn t3t3 t n-1 … r1r1 r2r2 r3r3 rkrk … x x x x xx x x x Select a subset of T to cover all requirements

Selection and Prioritization code 1 t1t1 t2t2 tntn t3t3 t n-1 … δ code 2 i1i1 i2i2 i3i3 i n-1 inin selection prioritization t1t1 t3t3 t n-1 …

Future Directions Listed Orchestrating regression testing techniques with test data generation Multi-objective regression testing Problem of test oracle and its cost Consideration of other domains Non-functional testing Tool support –Little text –Big issue in practice –Often ignored in research

Sample Project Ideas (1) [AA], [AS], [CR], [JC], [MK] Writing a test suite for faults vs. to meet requirements [AL] Extend a tool X to generate tests only for modified parts of the code [MG] Compare existing approaches for test selection [MG] Perform a large study on all apps from some open-source code base [HY] Extend or evaluate RETEST [DM]

Sample Project Ideas (2) Develop system based on regression testing minimization techniques to instead prepare sets of non-regression, related test suites that have overlaps in functionality [JN] Integrate test selection/prioritization with version control [QL] Combine several approach (e.g., integer programming and data-flow analysis) [ST] A project on test case minimization would be challenging… (types of experiments in CS/SE)

Sample Project Ideas (3) Study number of redundant or obsolete test cases in open source (how about disabled?) Implement and compare some approchaes –Extend the CDG-based technique Generate test cases that target changes Integrate some of these strategies with capture-replay testing tools Apply these techniques on web applications –Apply in other new domains –Test power consumption of embedded systems

Questions for Discussion (1) Correlation between fault detection and requirement coverage? [AL] How is selection affected by refactoring? [MG] Many techniques are proposed, are there some tools? [MG] What's the conclusion? Is minimization technique useful? [HY] What is minimum hitting set problem in terms of suite minimization? [JN] Regression testing for porting systems to different OS? (Porting testing?) [DM]

Questions for Discussion (2) What to do for multithreaded code? [QL] Can minimization and selection produce same results? [ST] Are redundant test cases a real problem? Doesn’t the need for minimization mean that we generated counter-productive tests? Should we minimize if that can miss faults? How much redundancy to have in testing? What are best practices for regression testing of GUI's?