1. CS527: (Advanced) Topics in Software Engineering (Software Testing and Analysis) Darko Marinov August 23, 2011

2. Course Overview Graduate seminar on program analysis (for bug finding), emphasis: systematic testing (for concurrent code) Focus on a (research) project –Papers: reading in advance, writing reports, presenting, and discussing –Project: initial proposal, progress report, [final presentation, based on enrollment size], paper One homework assignment (maybe two) to help with projects

3. Administrative Info Meetings: TR 2-3:15pm, 1302 SC Credit: 4 graduate hours Auditors welcome for discussions –Can come to any lecture, mostly self-contained Prerequisites: some software engineering and programming languages

4. Evaluation Grading –Project [40%] –Presentation [20%] –Participation (reports and discussion) [20%] –Homework assignment(s) [20%] Distribution –Grades will be A- centered –No guarantee for A (or even a passing grade)! Project is the most important

5. Project Proposal (due in late September) Progress report (in November) [Presentation (after Thanksgiving Break)] Paper (last day of classes) 15 students who took similar classes published 12 papers based on their projects –I’m happy to help, no co-authorship required

6. Fair Warnings This class will differ from most you take –Seminar style, reading papers –Centered around (research) projects Projects are NOT easy –Require that you explore a topic in great depth –The topic can/should be fairly narrow

7. Repeated Warning Project matters the MOST –If you like open-ended projects, do take this course –If you don’t like open-ended projects, please drop this course now –If you’re unsure, please discuss with me The worst scenario: take the course but realize you don’t like projects

8. Project Overview Testing/analysis of some open-source code –We will use mostly Java this semester You can use C#/C/C++/C--/Clojure/Cecil… http://en.wikipedia.org/wiki/List_of_programming_languages#C http://en.wikipedia.org/wiki/List_of_programming_languages#C Sample topics –Automated unit testing –Test coverage and adequacy criteria –Test-input generation, test oracles –Test maintenance –…

9. Course Communication Wiki http://wiki.engr.illinois.edu/display/cs527fa11 http://wiki.engr.illinois.edu/display/cs527fa11 Mailing list cs527-fa11 AT cs.illinois.edu

10. Personnel Instructor: Darko Marinov –Office: 3116 SC, hours: by appointment –Phone number: 217-265-6117 –NetID: marinov TA: Shin Hwei Tan –NetID: stan6 Please use your netid@illinois.edu email addresses for communication

11. Signup Sheet Name NetID (please write it legibly) Program/Year Group Interests Please also sign up on Wiki (if possible, as there are some delays with netids)

12. This Lecture: Overview Why look for bugs? What are bugs? Where they come from? How to detect them?

13. Some Costly “Bugs” NASA Mars space missions –Priority inversion (2004) –Different metric systems (1999) BMW airbag problems (1999) –Recall of >15000 cars Ariane 5 crash (1996) –Uncaught exception of numerical overflow –http://www.youtube.com/watch?v=kYUrqdUyEpIhttp://www.youtube.com/watch?v=kYUrqdUyEpI Your own (in)famous example?

14. Some “Bugging” Bugs An example bug on my laptop –“Jumping” file after changing properties Put a read-only file on the desktop Change properties: rename and make not read-only Your own favorite example?

15. Economic Impact “The Economic Impact of Inadequate Infrastructure for Software Testing” NIST Report, May 2002 $59.5B annual cost of inadequate software testing infrastructure $22.2B annual potential cost reduction from feasible infrastructure improvements

16. Estimates Extrapolated from two studies (5% of total) –Manufacturing: transportation equipment –Services: financial institutions Number of simplifying assumptions “…should be considered approximations” What is important for you? –Correctness, performance, functionality

17. Terminology Anomaly Bug Crash Defect Error Failure, fault Glitch Hangup Incorrectness J…

18. Dynamic vs. Static Incorrect (observed) behavior –Failure, fault Incorrect (unobserved) state –Error, latent error Incorrect lines of code –Fault, error

19. “Bugs” in IEEE 610.12-1990 Fault –Incorrect lines of code Error –Faults cause incorrect (unobserved) state Failure –Errors cause incorrect (observed) behavior Not used consistently in literature!

20. Correctness Common (partial) properties –Segfaults, uncaught exceptions –Resource leaks –Data races, deadlocks –Statistics based Specific properties –Requirements –Specification

21. Traditional Waterfall Model Requirements Analysis Design Checking Implementation Unit Testing Integration System Testing Maintenance Verification

22. Phases (1) Requirements –Specify what the software should do –Analysis: eliminate/reduce ambiguities, inconsistencies, and incompleteness Design –Specify how the software should work –Split software into modules, write specifications –Checking: check conformance to requirements

23. Phases (2) Implementation –Specify how the modules work –Unit testing: test each module in isolation Integration –Specify how the modules interact –System testing: test module interactions Maintenance –Evolve software as requirements change –Verification: test changes, regression testing

24. Testing Effort Reported to be >50% of development cost [e.g., Beizer 1990] Microsoft: 75% time spent testing –50% testers who spend all time testing –50% developers who spend half time testing

25. When to Test The later a bug is found, the higher the cost –Orders of magnitude increase in later phases –Also the smaller chance of a proper fix Old saying: test often, test early New methodology: test-driven development (write tests before code)

26. Software is Complex Malleable Intangible Abstract Solves complex problems Interacts with other software and hardware Not continuous

27. Software Still Buggy Folklore: 1-10 (residual) bugs per 1000 nbnc lines of code (after testing) Consensus: total correctness impossible to achieve for (complex) software –Risk-driven finding/elimination of bugs –Focus on specific correctness properties

28. Approaches for Detecting Bugs Software testing Model checking (Static) program analysis

29. Software Testing Dynamic approach Run code for some inputs, check outputs Checks correctness for some executions Main questions –Test-input generation –Test-suite adequacy –Test oracles

30. Other Testing Questions Maintenance Selection Minimization Prioritization Augmentation Evaluation Fault Characterization …

31. Model Checking Typically hybrid dynamic/static approach Checks correctness for all executions Some techniques –Explicit-state model checking –Symbolic model checking –Abstraction-based model checking

32. Static Analysis Static approach Checks correctness for all executions Some techniques –Abstract interpretation –Dataflow analysis –Verification-condition generation

33. Comparison Level of automation –Push-button vs. manual Type of bugs found –Hard vs. easy to reproduce –High vs. low probability –Common vs. specific properties Type of bugs (not) found

34. Soundness and Completeness Do we detect all bugs? –Impossible for dynamic analysis Are reported bugs real bugs? –Easy for dynamic analysis Most practical techniques and tools are both unsound and incomplete! –False positives –False negatives

35. Analysis for Performance Static compiler analysis, profiling Must be sound –Correctness of transformation: equivalence Improves execution time Programmer time is more important Programmer productivity –Not only finding bugs

36. Combining Dynamic and Static Dynamic and static analyses equal in limit –Dynamic: try exhaustively all possible inputs –Static: model precisely every possible state Synergistic opportunities –Static analysis can optimize dynamic analysis –Dynamic analysis can focus static analysis –More discussions than results

37. Current Status Testing remains the most widely used approach for finding bugs A lot of recent progress (within last decade) on model checking and static analysis –Model checking: from hardware to software –Static analysis: from sound to practical Vibrant research in the area Gap between research and practice

38. Topics Related to Finding Bugs How to eliminate bugs? –Debugging How to prevent bugs? –Programming language design –Software development processes How to show absence of bugs? –Theorem proving –Model checking, program analysis

39. Next Lecture Thursday, August 25, at 2pm, 1302 SC Texts to read (listed on Wiki) –How to Read an Engineering Research Paper by William G. GriswoldHow to Read an Engineering Research Paper –Writing Good Software Engineering Research Papers by Mary Shaw (ICSE 2003)Writing Good Software Engineering Research Papers If you have read that paper, read on another area You don’t have to write any report now Assignment 0: Modify “People” on Wiki