CS527: (Advanced) Topics in Software Engineering (Software Testing and Analysis) Darko Marinov August 24, 2010

Course Overview Graduate seminar on program analysis (for bug finding), emphasis: systematic testing Focus on a (research) project –Papers: reading in advance, writing reports, presenting, and discussing –Project: initial proposal, progress report, final presentation, paper One homework assignment (maybe two) to help with projects

Administrative Info Meetings: TR 2-3:15pm, 1302 SC Credit: 4 graduate hours Auditors welcome for discussions –Can come to any lecture, mostly self-contained Prerequisites: some software engineering and programming languages

Evaluation Grading –Project [40%] –Presentation [20%] –Participation (reports and discussion) [20%] –Homework assignment(s) [20%] Distribution –Grades will be A- centered –No guarantee for A (or even a passing grade)! Project is the most important

Project Proposal (due in about a month) Progress report (midterm time) Presentation (last weeks of classes) Paper (last day of classes) 10 students who took similar classes published 10 papers based on their projects –I’m happy to help, no co-authorship required

Fair Warnings This class will differ from most you take –Seminar style, reading papers –Centered around (research) projects Projects are NOT easy –Require that you explore a topic in great depth –The topic can/should be fairly narrow

Project Overview Testing/analysis of some open-source code –We will use mostly Java this semester You can use.NET/C# if you’re on Windows Sample topics –Automated unit testing –Test coverage and adequacy criteria –Test-input generation, test oracles –Test maintenance –…

Course Communication Wiki Mailing list cs527 AT cs.illinois.edu

Personnel TA: Sandro Badame –Office: 1218 SC, hours: by appointment –NetID: badame1 Instructor: Darko Marinov –Office: 3116 SC, hours: by appointment –Phone number: –NetID: marinov

Signup Sheet Name NetID (please write it legibly) Program/Year Group Interests Please also sign up on Wiki (if possible, as there are some delays with netids)

This Lecture: Overview Why look for bugs? What are bugs? Where they come from? How to detect them?

Some Costly “Bugs” NASA Mars space missions –Priority inversion (2004) –Different metric systems (1999) BMW airbag problems (1999) –Recall of >15000 cars Ariane 5 crash (1996) –Uncaught exception of numerical overflow – Your own favorite example?

Some “Bugging” Bugs An example bug on my laptop –“Jumping” file after changing properties Put a read-only file on the desktop Change properties: rename and make not read-only Your own favorite example?

Economic Impact “The Economic Impact of Inadequate Infrastructure for Software Testing” NIST Report, May 2002 $59.5B annual cost of inadequate software testing infrastructure $22.2B annual potential cost reduction from feasible infrastructure improvements

Estimates Extrapolated from two studies (5% of total) –Manufacturing: transportation equipment –Services: financial institutions Number of simplifying assumptions “…should be considered approximations” What is important for you? –Correctness, performance, functionality

Terminology Anomaly Bug Crash Defect Error Failure, fault Glitch Hangup Incorrectness J…

Dynamic vs. Static Incorrect (observed) behavior –Failure, fault Incorrect (unobserved) state –Error, latent error Incorrect lines of code –Fault, error

“Bugs” in IEEE Fault –Incorrect lines of code Error –Faults cause incorrect (unobserved) state Failure –Errors cause incorrect (observed) behavior Not used consistently in literature!

Correctness Common (partial) properties –Segfaults, uncaught exceptions –Resource leaks –Data races, deadlocks –Statistics based Specific properties –Requirements –Specification

Traditional Waterfall Model Requirements Analysis Design Checking Implementation Unit Testing Integration System Testing Maintenance Verification

Phases (1) Requirements –Specify what the software should do –Analysis: eliminate/reduce ambiguities, inconsistencies, and incompleteness Design –Specify how the software should work –Split software into modules, write specifications –Checking: check conformance to requirements

Phases (2) Implementation –Specify how the modules work –Unit testing: test each module in isolation Integration –Specify how the modules interact –System testing: test module interactions Maintenance –Evolve software as requirements change –Verification: test changes, regression testing

Testing Effort Reported to be >50% of development cost [e.g., Beizer 1990] Microsoft: 75% time spent testing –50% testers who spend all time testing –50% developers who spend half time testing

When to Test The later a bug is found, the higher the cost –Orders of magnitude increase in later phases –Also the smaller chance of a proper fix Old saying: test often, test early New methodology: test-driven development (write tests before code)

Software is Complex Malleable Intangible Abstract Solves complex problems Interacts with other software and hardware Not continuous

Software Still Buggy Folklore: 1-10 (residual) bugs per 1000 nbnc lines of code (after testing) Consensus: total correctness impossible to achieve for (complex) software –Risk-driven finding/elimination of bugs –Focus on specific correctness properties

Approaches for Detecting Bugs Software testing Model checking (Static) program analysis

Software Testing Dynamic approach Run code for some inputs, check outputs Checks correctness for some executions Main questions –Test-input generation –Test-suite adequacy –Test oracles

Other Testing Questions Maintenance Selection Minimization Prioritization Augmentation Evaluation Fault Characterization …

Model Checking Typically hybrid dynamic/static approach Checks correctness for all executions Some techniques –Explicit-state model checking –Symbolic model checking –Abstraction-based model checking

Static Analysis Static approach Checks correctness for all executions Some techniques –Abstract interpretation –Dataflow analysis –Verification-condition generation

Comparison Level of automation –Push-button vs. manual Type of bugs found –Hard vs. easy to reproduce –High vs. low probability –Common vs. specific properties Type of bugs (not) found

Soundness and Completeness Do we detect all bugs? –Impossible for dynamic analysis Are reported bugs real bugs? –Easy for dynamic analysis Most practical techniques and tools are both unsound and incomplete! –False positives –False negatives

Analysis for Performance Static compiler analysis, profiling Must be sound –Correctness of transformation: equivalence Improves execution time Programmer time is more important Programmer productivity –Not only finding bugs

Combining Dynamic and Static Dynamic and static analyses equal in limit –Dynamic: try exhaustively all possible inputs –Static: model precisely every possible state Synergistic opportunities –Static analysis can optimize dynamic analysis –Dynamic analysis can focus static analysis –More discussions than results

Current Status Testing remains the most widely used approach for finding bugs A lot of recent progress (within last decade) on model checking and static analysis –Model checking: from hardware to software –Static analysis: from sound to practical Vibrant research in the area Gap between research and practice

Topics Related to Finding Bugs How to eliminate bugs? –Debugging How to prevent bugs? –Programming language design –Software development processes How to show absence of bugs? –Theorem proving –Model checking, program analysis

Next Lecture Thursday, August 26, at 2pm, 1302 SC –Guest lecture by Brett Daniel Texts to read (listed on Wiki) –How to Read an Engineering Research Paper by William G. GriswoldHow to Read an Engineering Research Paper –Writing Good Software Engineering Research Papers by Mary Shaw (ICSE 2003)Writing Good Software Engineering Research Papers If you have read that paper, read on another area Assignment 0: Modify “People” on Wiki