cs498dm Software Testing Darko Marinov January 22, 2008

Course Overview Introduction to software testing –Systematic, organized approaches to testing –Based on models and coverage criteria –Testing is not (only) about finding “bugs” –Improve your testing (and development) skills Teaching staff –Insructor: Darko –TA: Vilas

Administrative Info FAQ: Deliverables? –No exams: no final, no midterm –Five problem sets (5*15%) and a project (25%) Project: proposal, two reports, hopefully bug reports Project is on testing a piece of refactoring engine Undergrads must be registered for 3 hours –For more, you can do an independent study –If interested in research on testing, contact me Juniors: ask for permission if you didn’t

“Assignment” Did you sign up on Wiki Did you receive s on the list Did you try out Eclipse/NetBeans? –Which refactorings did you try?

“Prizes” Potential categories –The “buggiest” bug found in the course Hardest to find, most important, realistic case study –Most bugs found or tests generated Not the best measures of testing effort Occasional candies to encourage your participation and discussion in lectures –Used to be a part of grade but not any more

Several Warm-Up Topics Your opinion about the guest lecture? –Do you want to have more of them? My report from the Workshop on Teaching Software Testing (WTST 7) –No technique is perfect –Abstraction/modeling is important Think of various ways that software could break –Testing for testers vs. testing for developers Do you look at source code while testing?

This Lecture: Introduction (Cont’d) Why look for bugs? What are bugs? Where they come from? How to detect them? Topics that will be covered in the course Related topics that will not be covered

“Bugs” in IEEE Fault –Incorrect lines of code Error –Faults cause incorrect (unobserved) state Failure –Errors cause incorrect (observed) behavior Not used consistently in literature!

Correctness and Quality Common (partial) properties –Segfaults, uncaught exceptions –Resource leaks –Data races, deadlocks –Statistics based Specific properties –Requirements –Specification

Traditional Waterfall Model Requirements Analysis Design Checking Implementation Unit Testing Integration System Testing Maintenance Regression Testing We will look at general techniques, applicable in several phases of testing

Phases (1) Requirements –Specify what the software should do –Analysis: eliminate/reduce ambiguities, inconsistencies, and incompleteness Design –Specify how the software should work –Split software into modules, write specifications –Checking: check conformance to requirements, using for example conformance testing

Phases (2) Implementation –Specify how the modules work –Unit testing: test each module in isolation Integration –Specify how the modules interact –Integration testing: test module interactions –System testing: test the entire system Maintenance –Evolve software as requirements change –Regression testing: test changes

Testing Effort Reported to be >50% of development cost [e.g., Beizer 1990] Microsoft: 75% time spent testing –50% testers who spend all time testing –50% developers who spend half time testing

When to Test The later a bug is found, the higher the cost –Orders of magnitude increase in later phases –Also the smaller chance of a proper fix Old saying: test often, test early New methodology: test-driven development (write tests even before writing code)

Software is Complex Malleable Intangible Abstract Solves complex problems Interacts with other software and hardware Not continuous

Software Still Buggy Folklore: 1-10 (residual) faults per 1000 nbnc lines of code (after testing) Consensus: total correctness impossible to achieve for complex software –Risk-driven finding/elimination of faults –Focus on specific correctness properties

Approaches to Detecting Bugs Software testing Model checking (Static) program analysis …

Software Testing Dynamic approach Run code for some inputs, check outputs Checks correctness for some executions Main questions –Test-suite adequacy (coverage criteria) –Test-input generation –Test oracles

Other Testing Questions Selection Minimization Prioritization Augmentation Evaluation Fault Characterization … Testing is not (only) about finding faults!

Current Status Testing remains the most widely used approach to finding bugs –Validation: are we building the right system? –Verification: are we building the system right? Testing is gaining importance with test-first development and increased reliability needs A lot of research on testing (part of mine too) –We’ll look mostly at techniques used in practice

“Schools” of Software Testing Bret Pettichord described four schools –Analytic (a branch of CS/Mathematics) –Factory (a managed process) –Quality (a branch of quality assurance) –Context-Driven (a branch of development) This course focuses on artifacts, not process Do you want a guest speaker from industry?

Topics Related to “Finding Bugs” How to “eliminate bugs” (localize faults)? –Debugging How to “prevent bugs”? –Programming language design –Software development processes How to “show absence of bugs”? –Theorem proving –Model checking, program analysis

Testing Topics to Cover Test coverage and adequacy criteria –Graph, logic, input domains, syntax-based Test-input generation Test oracles Model-based testing Testing software with structural inputs Test automation Testing in your domain of interest?

Your Interests Testing methods and methodology (*2) Writing good/correct test cases (*3) Better/more effective testing (*5) Working with testers/developers (*2) Relationship of testing and development cycle Improve development (*4) Software verification and (test) automation (*2) Job skills (*3) (short-term or long-term?) Please update me as we go through the course

Summary of the Introduction Eliminate bugs to save lives and money “Bugs” may mean faults, errors, failures Several approaches for detection: software testing, model checking, static analysis… Software testing is the most widely used approach for validation and verification –We will cover systematic approaches to testing, based on coverage criteria for various models –Testing is not (only) about revealing faults

Next Lecture Thursday, January 24, 3:30pm, 1304 SC –Example Interactive Testing Session Test some refactoring It should be fun We can learn from mistakes as we go We will learn some testing terminology Assignments –Really sign up on Wiki –Really try out Eclipse and/or NetBeans