Paul Ammann & Jeff Offutt http://www.cs.gmu.edu/~offutt/softwaretest/ Introduction to Software Testing Chapter 9.2 Challenges in Testing Software – Software Testability Paul Ammann & Jeff Offutt http://www.cs.gmu.edu/~offutt/softwaretest/

Chapter 9 Outline Software Testability Testing for Emergent Properties: Safety and Security Software Testability Test Criteria and the Future of Software Testing Software Testability Paul Ammann and Jeff Offutt, Introduction to Software Testing, Cambridge University Press, Cambridge, UK, ISBN 0-52188-038-1, 2008 Introduction to Software Testing (Ch 9.2) © Ammann & Offutt 2

Testability Overview Testability is distinct from software testing General definition : An estimate or measurement of a conditional probability – assuming that a software artifact contains a fault, how likely is it that testing will reveal that fault ? What do we do with testability estimates ? Pay more attention to components with low testability – code reviews, formal analysis, stronger test criteria Modify low testability components to increase their testability We have more confidence in components with high testability Risk analysis – low testability components represent risk that management needs to be aware of Introduction to Software Testing (Ch 9.2) © Ammann & Offutt

Model of Testability If a program has a fault, how difficult will it be to find a test that causes a failure ? Simple model Out failure causing Inputs P Testability = | failure causing | | Input | X 100 % Impractical to measure Introduction to Software Testing (Ch 9.2) © Ammann & Offutt

Approximating Testability Testability can be approximated with the RIP model and mutation Given a location X in a program P P: entry X exit R = % inputs from some distribution that reach X Induce faults (mutants) I = % inputs that cause a fault to infect (average over N faults) P = % infected states that propagate to output Sensitivity (X) = R * I * P Testability (P) = F (Sensitivity (X)), for all X in P Introduction to Software Testing (Ch 9.2) © Ammann & Offutt

Issues in Approximating Testability Reasonable input distribution ? How to induce faults ? What faults ? How to measure propagation ? Expensive! Information hiding reduces propagation Assertion checking can be used to increase testability Introduction to Software Testing (Ch 9.2) © Ammann & Offutt

Testability of OO Software Information hiding decreases testability State information is saved in instance variables No direct access to instance variables Inheritance compounds the problem Instance variables are defined in ancestor classes – harder to get to These are primarily issues of observability Increasing observability in OO software : Require additional get ( ) methods – must be done during development Use Java reflection to access internal variables – hard to interpret the data values This is an area of ongoing research Introduction to Software Testing (Ch 9.2) © Ammann & Offutt

Testability of Web Applications Both controllability and observability are very low User interface and most software components distributed on different computers Server software may be distributed even further Most communication is through message passing Much of the inter-component communication goes through the client Stateless HTTP messages State is kept in an unusual combination of technologies Cookies and session objects Testability in web applications is still very much an open research area Introduction to Software Testing (Ch 9.2) © Ammann & Offutt

How best to measure testability is still an open research problem Testability Summary Testability can give valuable information to testers, managers and developers Testability is often thought of as combining two characteristics of software Controllability and observability Measuring testability is a very technical task How best to measure testability is still an open research problem Introduction to Software Testing (Ch 9.2) © Ammann & Offutt