CS527 Topics in Software Engineering (Software Testing and Analysis) Darko Marinov September 9, 2010

Schedule First few lectures to help you select projects –Testing: ReAssert, UDITA, Pex, Randoop –Today: JPF (model checking), note: journal paper –Sep 14: CHESS (multithreaded testing) –Sep 16: Regression testing, note: survey paper –Sep 21: Static (code) analysis? –Sep 23: GUI testing? –Sep 28: Analysis of code comments? –Testing distributed or real-time systems? –Your suggestions?

Paper Today Model Checking Programs by W. Visser, K. Havelund, G. Brat, S. Park and F. Lerda (J-ASE, vol. 10, no. 2, April 2003)Model Checking Programs JPF web site – –First open-source project from NASA Slides courtesy of Peter Mehlitz and Willem VisserSlides

Paper Overview Problem –Model checking of real code Terminology: Systematic testing, state-space exploration Solution –Specialized Java Virtual Machine Supports backtracking, state comparison Many optimizations to make it scale –Publicly available tool (Java PathFinder) Evaluation/applications –Remote Agent Spacecraft Controller –DEOS Avionics Operating System

Questions for Discussion (1) Reports on time but not early: [CR], [AS], [JC] Can an explicit-state model checker really scale to million lines of code? [AA] Tools are top down (execute the program and evaluate it as it runs) but code is written bottom up (unit by unit)? [AL] When is a state revisited? [JN] Can we move some part of search and bytecode interpretation to C? [MG] How to use JPF for sequential programs? [QL]

Questions for Discussion (2) Omitted? [DM] –Reminder: you can omit up to two reports How many kinds of abstraction are available? [ST] Too much work for too little benefit? [HY] –Does it only find bugs that other tools can find? Make JPF distibuted/parallel? [MK] Combine JPF and Radoop (or X and Y)? How can false positives be managed for a better user experience?

Questions for Discussion (3) Level of adoption outside NASA/Research? Have any high profile bugs been found? If JPF a valid approach outside of real-time systems and safety/mission critical software? How do you know the tool is correct? –Especially first version with Java -> Promela –What about differences between JPF and JVM Does it find bugs or prove software correct? What about non-Java languages?

Old Questions for Discussion (1) Which part of JPF can be incorrect? Can a compiler be model checked? How difficult is it to set environments? What to check in sequential programs (some examples are multi-threaded)? What about checking design not code? Comparison of testing, model checking, and static analysis

When is model checking applicable? What kind of bugs it can find? How well does it scale? What about memory requirements? What about time requirements? Why work with Java not C/C++? How can I apply this on my code? Old Questions for Discussion (2)

What’s recent work in this area? –There’s some work in my group if interested How do they compare states? How good are heuristics? How to use the tool in practice? –Hopefully the tutorial is helpful –You can download from NASA servers Problem set based on some tool??? Old Questions for Discussion (3)

Sample Project Ideas Apply JPF on some software –Did you read the optional reading/report? –Compare with more tools and techniques Improve JPF –Add new functionality (e.g., backtrackable time) –Implement more library methods (e.g., reflection) –Patch some bugs –Use for another language (Scala, X10…) –Automate creation of environments Check 427 Final Project ProposalsFinal Project Proposals