CS527 Topics in Software Engineering (Software Testing and Analysis) Darko Marinov August 30, 2011

Course Info Everyone on the mailing list and “People” –Hopefully no more registration changes Coursework –PROJECT [40%] Already received some proposals –Presentation [20%] –Participation (reports and discussion) [20%] –Homework assignment(s) [20%] Fair warning: this is NOT easy –Project requires that you explore some topic in great DEPTH

Schedule First few lectures to help you select projects –Last week: Intro, Reading/Writing papers –Today: CHESS (find multithreaded bugs) –Sep 1: IMUnit (writing multithreaded tests) –Sep 6: CAPP & Setac (more concurrent testing) –Sep 8: iComment (analyzing source comments) –Your preferences? (I’ll also read HW0) Software testing, model checking, static analysis Your presentations from Oct 4 –1 or 2 presentations per lecture

(Y)our Paper Reports Start on NEXT Thursday (Sep 6) Four items: 1.One good point 2.One bad point 3.One potential project (help for proposals) 4.One question for discussion Looking for DEPTH –Length is not important; answers that are too long can make it hard to find the depth

Discussion Before/After Meeting Starting the discussion online before the meeting? –Pro: the discussion starts shortly after you read the paper –Con: potential issues with copying ideas or statements for the paper reports –Could use rule that every report needs to address an issue not raised previously Two NO votes so far

Paper Today Iterative Context Bounding for Systematic Testing of Multithreaded Programs Madan Musuvathi & Shaz Qadeer (PLDI 2007)Iterative Context Bounding for Systematic Testing of Multithreaded Programs Slides from a tutorial by Sebastian Burckhardt, Madan Musuvathi & Shaz QadeerSlides –Not only about this specific paper What would you write for the four items: 1.One good point 2.One bad point 3.One potential project 4.One question for discussion

Paper Overview Problem –Testing multithreaded code is hard –Trying all thread interleavings doesn’t scale Solution –Bound the number of preemptive context switches –Tool for Windows: CHESS (stateless, reexecutes) –Evaluation also for ZING (stateful) Evaluation –6 programs of various sizes –Found 14 previously unknown bugs

Old Questions for Discussion, 1 Missing by 12:30: [CR], [AS] Can POR improve CHESS & how much? [JC] How often CHESS goes to infinite loop in state space graph? [MG] Does this paper assume correct use of synchronization variables? [AA] What is the difference between observable and non-observable race? [AA] Better to limit rather than test threading? [HY] How to automatically close open systems? [MK]

Old Questions for Discussion, 2 How to know the size of state space? [AL] Any advantages inserting (removing??) a scheduling point between two syncs? [DM] Why not bound nonpreempting switches? [JN] Can iterative context bounding be applied to message passing programs? [ST] What’s used in industry, stress testing? [QL] Soundness vs. completeness –False positives vs. false negatives –Precision vs. recall Combine CHESS and Pex (or X and Y)?

Old Questions for Discussion, 3 Why not report all accessed to shared data not protected by a synchronization variable? How are data races identified? How are other bugs identified? Assertions? What other heuristics can one use? POR? If you analyze models (ZING), how do you know there is bug in actual code? How to test a complex multithreaded algorithm in mostly sequential code? Extract it out?

Old Questions for Discussion, 4 How does it reduce time to polynomial? What about dynamically changing number of threads in programs? Or dynamically created? Can CHESS handle locks, mutexes, semaphores, etc.? How to identify preemption or nonpreemption in Java code? What are limits of this approach? Is it LOC, 10k, 100k, 1M? Is it # of threads, preemptions? Can it be combined with DFS?

Older Questions for Discussion 1 Which part of JPF/CHESS can be incorrect? Can a compiler be model checked? How difficult is it to set environments? –How to “close a system”? Comparison of testing, model checking, and static analysis When is model checking applicable? What kind of bugs it can find? Why work with language X not Y?

How well does it scale? What about memory requirements? What about time requirements? How can I apply this on my code? What’s recent work in this area? –A lot of work in many group, including mine How to use the tool in practice? How does CHESS/JPF compare states? How good are heuristics? Older Questions for Discussion 2

Old Sample Project Ideas Apply CHESS on some software –Did you read the optional reading/report on JPF? –Compare with more tools and techniques Consider other heuristics for exploration –Message-passing programs? Evolving code? Automate creation of environments Extend JPF implementation of CHESS –Contributed by Igor Andjelkovic, Steven Lauterburg, and Mirko Stojmenovic Reimplement in another model checker Improve debugging of multithreaded code

Next Meeting Thursday, September 1 –Guest presentation by Vilas Jagannath Read a paper (listed on Wiki) –Improved Multithreaded Unit Testing V. Jagannath, M. Gligoric, D. Jin, Q. Luo, G. Rosu, and D. Marinov (ESEC/FSE 2011)Improved Multithreaded Unit Testing –NO report due yet –Consider four points: good, bad, potential project, discussion question