CS527 Topics in Software Engineering (Software Testing and Analysis) Darko Marinov September 7, 2010

Schedule First few lectures to help you select projects –Previously: Intro, ReAssert, UDITA, Pex –Today: Randoop (test generation) –Sep 9: JPF (model checking), note: journal paper –Sep 14: CHESS (multithreaded testing) –Sep 16: Regression testing, note: survey paper –Sep 21: Static (code) analysis? –Sep 23: GUI testing? –Sep 28: Analysis of code comments? –Your suggestions?

Reports, Project, Presentations Paper reports due for every lecture –4 items for now (fewer after proposals) –Did you get feedback for first report? No more feedback but for projects (and non-ASCII :) Project proposals due September 30 –It’s hard to have a good proposal in a day/week –Start discussion with Sandro and me early Your presentations start from October 7 –Need to choose papers more in advance –Can sign up for slots even now –Bonus given to those who sign up for first lecture

Paper Today Feedback-directed random test generation by Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball (ICSE 2007)Feedback-directed random test generation Slides courtesy of Carlos Pacheco

Paper Overview Problem (Question) –Generate unit tests (with high coverage?) Solution (Result) –Generate sequences of method calls –Random choice of methods and parameters –Publicly available tool for Java (Randoop) Evaluation (Validation) –Data structures (JPF is next lecture) –Checking API contracts –Regression testing (lecture next week)

Questions for Discussion (1) Is there a limit on the length of the sequence generated by Randoop? [ST] Is sequence extension effective when/if different violation results? (e.g. non- determinism) [MK] Can this method also effective for other type of classes such as UI or business logic? [HY] How can we combine systematic testing and random testing? [QL] Why is BFS deemed more preferable to DFS in systematic testing? [DM]

Questions for Discussion (2) Can we deploy symbolic execution when coverage of some methods is small to prioritize some values? [MG] What are some other options to add repetition to the generator? [JN] Tradeoff in random and systematic? [AL] Other applications/domains? How do we know what coverage is good? What are tool limitations? What about methods that change arguments?

Questions for Discussion (3) What is the relation between test coverage and correct code? High coverage and bugs? What is Randoop well suited for? What is Randoop not well suited for? Why is Randoop not used more often? What is the number of false positives? Combining Randoop and Pex (or JPF or X)? –Do they find complementary bugs? Can we parallelize Randoop? Can we use application-specific filters?

Past Questions for Discussion (1) Evaluation considered mostly good, but –Why evaluate on data structures? –Why not compare with more other tools? What kind of bugs were found? –Any feedback from developers on bugs? ISSTA 2008 paper (see optional paper) –When is this technique good/bad? –What to use to complement RANDOOP? Why is this solution better than others?

Past Questions for Discussion (2) What about non-primitive inputs? –Can RANDOOP generate input programs? What about testing business systems? How to reduce # of illegal sequences? How many seeds to use in runs? What are general pros/cons of random testing? Is time a good stopping condition?

Sample Project Ideas First: would Randoop by itself be a good class project (is it too big or too small?) Apply Randoop on some software Extend Randoop, e.g., directed generation –Improve effectiveness or performance Apply random testing in another domain Compare with more tools and techniques Evaluate for regression testing –Assertion failures? False alarms? Repair? Check Randoop page ProjectIdeasProjectIdeas