cs498st Software Testing Tao Xie

Teaching Staff Instructor: Tao Xie No TA Graders (likely) Email: taoxie@illinois.edu (subject cs498st) Office: SC 4237, 217-244-5931 Office hours: by appointment No TA Graders (likely)

Administrative Info Lectures: TR 3:30pm-4:45pm, 1109 SC Credit: 3 undergraduate hours 3 or 4 graduate hours (a larger project for 4) Prerequisites: recommended software engineering (cs427) and programming languages (cs225, cs421) Consent of instructor

Course Overview Introduction to software testing Systematic, organized approaches to testing Based on models and coverage criteria Testing is not (only) about finding “bugs” Improve your testing (and development) skills Several homework assignments Project: testing some real software C# code No exams

Grading Points Grades Homeworks (70%) Project (30%) A*(90%), B*(80%), C*(70%), D*(60%), F(<60%) For more details, see the syllabus online The instructor may lower the point limits

Software Tools Microsoft Visual Studio 2017 & 2015 Enterprise Edition Available to UIUC students: https://webstore.illinois.edu/shop/product.aspx ?zpid=2917 IntelliTest (Pex) Code Contracts (available for VS 2015) Useful resources: Java vs. C# C# for Java Developers

Project Testing some real software (in C#) Deliverables Proposal (due in three weeks) Progress report (around midterm) Final report (by the grade submission deadline) Pull requests for test code and/or bug reports Please meet me if you cannot contribute publicly

Collaboration You must individually solve homework assignments You canNOT collaborate on anything else (unless explicitly stated that you can!) Do projects in groups, preferably two or three students but can do solo projects Testing is a social activity Communication matters But we are in an education environment so cannot allow all communication

Course Communication Wiki https://wiki.illinois.edu/wiki/display/cs498stfa17/ Main class communication Piazza: https://piazza.com/illinois/fall2017/cs498st/home Instructor: Tao Email: taoxie@illinois.edu Office: SC 4237, 217-244-5931 Office hours: by appointment

Signup Sheet Name Email address (NetID@illinois.edu) Program/Year Interests: what would you like to learn about testing? Experience: what testing did you do?

Optional Textbook “Introduction to Software Testing” by Paul Ammann and Jeff Offutt Cambridge University Press, 2nd Edition, Dec. 2016 http://cs.gmu.edu/~offutt/softwaretest/

This Lecture: Introduction to “Bugs” Why look for bugs? What are bugs? Where they come from? How to detect them?

Some Costly “Bugs” Ariane 5 crash (1996) Your own favorite examples? http://www.youtube.com/watch?v=kYUrqdUyEpI Uncaught exception of numerical overflow “Specifically a 64 bit floating point number relating to the horizontal velocity of the rocket with respect to the platform was converted to a 16 bit signed integer. The number was larger than 32,767, the largest integer storeable in a 16 bit signed integer, and thus the conversion failed.” http://www-users.math.umn.edu/~arnold/disasters/ariane.html Your own favorite examples?

Economic Impact “The Economic Impact of Inadequate Infrastructure for Software Testing” NIST Report, May 2002 $59.5B annual cost of inadequate software testing infrastructure $22.2B annual potential cost reduction from feasible infrastructure improvements

Economic Impact – cont. Is Knight's $440 million glitch the costliest computer bug ever? http://money.cnn.com/2012/08/09/technology/knight- expensive-computer-bug/index.html

New Era…Software itself is changing... Services

How people use software is changing…

How people use software is changing… Individual Isolated Not much data/content generation

How people use software is changing… Individual Isolated Not much data/content generation

How people use software is changing… Social Individual Isolated Collaborative Huge amount of data/artifacts generated anywhere anytime Not much data/content generation

How software is built & operated is changing…

How software is built & operated is changing… Code centric Data pervasive In-lab testing Debugging in the large Experience & gut-feeling Informed decision making Centralized development Distributed development Long product cycle Continuous release … …

Quality Attributes What is important to you? Correctness, performance, functionality, security, …

Some Motivation for Testers A slide from Debra Richardson, a professor at UC Irvine: “Analysis and Testing are Creative” (page 26/48)

https://www.ics.uci.edu/~taylor/ICS221/slides/SA&T-Nov26-2002.pdf

Terminology Anomaly Bug Crash Defect Error Failure, fault Glitch Hang Incorrectness (from a 2009 cs498dm student) Jitter (from a 2012 cs498dm student) Kink (from a 2015 cs498dm student) L…?

Dynamic vs. Static Incorrect (observed) behavior Failure, fault Incorrect (unobserved) state Error, latent error Incorrect lines of code Fault, error

“Bugs” in IEEE 610.12-1990 Fault Error Failure Incorrect lines of code Error Faults cause incorrect (unobserved) state Failure Errors cause incorrect (observed) behavior Not used consistently in literature!

Correctness and Quality Common (partial) properties Segfaults, uncaught exceptions Resource leaks Data races, deadlocks Statistics based Specific properties Requirements Specification

Traditional Waterfall Model Requirements Analysis Design Checking Implementation Unit Testing Integration System Testing Maintenance Regression Testing

Phases (1) Requirements Design Specify what the software should do Analysis: eliminate/reduce ambiguities, inconsistencies, and incompleteness Design Specify how the software should work Split software into modules, write specifications Checking: check conformance to requirements, using for example conformance testing

Phases (2) Implementation Integration Maintenance Specify how the modules work Unit testing: test each module in isolation Integration Specify how the modules interact Integration testing: test module interactions System testing: test the entire system Maintenance Evolve software as requirements change Regression testing: test changes

Testing Effort Reported to be >50% of development cost [e.g., Beizer 1990] Microsoft: 75% time spent testing 50% testers who spend all time testing 50% developers who spend half time testing

When to Test The later a bug is found, the higher the cost Orders of magnitude increase in later phases Also the smaller chance of a proper fix Old saying: test often, test early New methodology: test-driven development (write tests before code)

Software is Complex Malleable Intangible Abstract Solves complex problems Interacts with other software and hardware Not continuous

Software Still Buggy Folklore: 1-10 (residual) bugs per 1000 nbnc lines of code (after testing) Consensus: total correctness impossible to achieve for (complex) software Risk-driven finding/elimination of bugs Focus on specific correctness properties

Approaches for Detecting Bugs Software testing Model checking (Static) program analysis

Software Testing Dynamic approach Run code for some inputs, check outputs Checks correctness for some executions Main questions Test-input generation Test-suite adequacy Test oracles

Other Testing Questions Maintenance Selection Minimization Prioritization Augmentation Evaluation Fault Characterization …

Current Status Testing remains the most widely used approach for finding bugs Testing is gaining importance with test-first development and increased reliability needs A lot of recent progress (within last decade) on model checking and static analysis Model checking: from hardware to software Static analysis: from sound to practical Vibrant research in the area Gap between research and practice

Topics Related to Finding Bugs How to eliminate bugs? Debugging How to prevent bugs? Programming language design Software development processes How to show absence of bugs? Theorem proving Model checking, program analysis

Testing Topics to Cover Test coverage and adequacy criteria Graph, logic, input domains, syntax-based Test-input generation Test oracles Regression testing Test automation Testing in your domain of interest?

Summary of the Introduction Eliminate bugs to save lives and money “Bugs” may mean faults, errors, failures Several approaches for detection: software testing, model checking, static analysis… Software testing is the most widely used approach for validation and verification We will cover systematic approaches to testing, based on coverage criteria for various models Testing is not (only) about revealing faults