Efficient Techniques for Software Testing Jeff Lei April 13, 2007

Outline Software Testing Concurrency Testing –Testing Concurrent Programs –Testing Synchronization Components T-Way Interaction Testing Other Work –Model Checking, Protocol Validation, Pervasive Computing

Software Engineering Software has become pervasive in modern society –Directly contributes to quality of life –Malfunctions cost billions of dollars every year, and have severe consequences in a safe-critical environment All about building better software in better ways, especially for large-scale development –Requirements, design, coding, testing, maintenance, configuration, documentation, deployment, and etc.

Software Testing A dynamic approach to detecting software faults –Alternatively, static analysis can be performed, which is however often intractable Involves sampling the input space, running the test object, and observing the runtime behavior Perhaps the most widely used approach in practice –Labor intensive, and often consumes more than 50% of development cost

Research Problems How to detect as many faults as possible spend with as little effort as possible? –How to generate a good set of test inputs? –How to perform actual test executions? –How to evaluate test outcomes?

Outline Software Testing Concurrency Testing –Testing Concurrent Programs –Testing Synchronization Components T-Way Interaction Testing Other Work –Model Checking, Protocol Validation

Concurrent Programs Consists of multiple threads of execution that proceed in parallel –multithreaded programs, multi-process (or distributed) programs Advantages: –Better resource utilization, increased computation efficiency, and providing simplified solutions for many problem domains Notoriously difficult to test due to their non- deterministic behavior

What Is Unique Non-determinism –Multiple executions with the same data input might display different behaviors Synchronization –How to ensure that concurrent events are exercised in an expected order? Communication –How to exchange information between different threads?

Concurrency Testing Strategies Non-deterministic testing - execute the same program with the same input for multiple times: –easy, but inefficient, and some errors cannot be detected Deterministic testing - select a set of test sequences and then force them to be exercised: –can detect more subtle errors, but requires additional effort for test sequence selection and runtime control State exploration – explore the state space in a systematic manner –suffers the well-known state explosion problem –state representation is difficult for programs written in a full- fledged programming language

Reachability Testing Combines non-deterministic and deterministic testing –Generating test sequences dynamically, without constructing any static model –Dealing with partial orders directly, and thus no redundant interleavings! Can also be considered as a state exploration technique –Exercising every possible synchronization sequence of a program with a given input

The Framework 1.Execute a program P with a given input non- deterministically to collect a trace Q 2.Identify the race conditions in Q and compute its race variants 3.For each variant, conduct a prefix-based test run to collect a new trace Q’ 4.Repeat 2, 3, and 4 for each newly collected trace Q’.

Example

Main Results Developed three reachability testing algorithms –Exhaustive RT, Combinatorial RT, and Random RT Built a reachability testing tool called RichTest –Portable implementation, applicable to most common synchronization constructs Conducted an empirical evaluation –Performed significantly better than VeriSoft (from Bell Labs) Publications: STVR07a, TSE06, ISSRE05

Dinning Philosophers

Outline Software Testing Concurrency Testing –Testing Concurrent Programs –Testing Synchronization Components T-Way Interaction Testing Other Work –Model Checking, Protocol Validation

Synchronization Components Synchronization logic is often encapsulated into one or more components Synchronization components need to be tested separately –Obtain necessary confidence prior to integration –helps to reduce complexity by incremental testing A different problem than testing complete programs –The ways in which a component could be used is typically unknown

Java Monitor A high-level synchronization construct that supports data encapsulation and information hiding –A Java class that defines one or more synchronized methods Mutual exclusion automatically enforced by the Java runtime Conditional synchronization implemented by the programmer, using wait and notify operations

Java Monitor Semantics Consists of three components: an entry queue, a critical section, a waiting queue A thread must enter the CS before it executes a synchronized method; it has to wait in the entry queue if the CS is not empty The wait and notify operations can only be executed inside the CS –wait: puts the current thread in the waiting queue –notify: awakens one thread in the waiting queue

Example class BoundedBuffer { private int fullSlots=0; private int capacity = 0; private int[] buffer = null; private int in = 0, out = 0; public BoundedBuffer(int bufferCapacity) { 1. capacity = bufferCapacity; 2. buffer = new int[capacity]; } public synchronized void deposit (int value) { 3. while (fullSlots == capacity) { 4. try { wait(); } catch (InterruptedException ex) {} } 5. buffer[in] = value; 6. in = (in + 1) % capacity; 7. if (fullSlots++ == 0) { 8. notifyAll(); } public synchronized int withdraw () { 9. int value = 0; 10. while (fullSlots == 0) { 11. try { wait(); } catch (InterruptedException ex) {} } 12. value = buffer[out]; 13. out = (out + 1) % capacity; 14. if (fullSlots-- == capacity) { 15. notifyAll(); } 16. return value; }

Graphic View Critical Section Entry Queue W Condition Queue D Assume that the buffer is initially empty.

Research Questions What is unique to testing Java monitors? –Used to synchronize, and thus typically accessed by, multiple threads simultaneously Can existing work on OO testing apply? –OO testing methods typically assume a single- threaded test driver How to effectively test Java monitors?

Technical Challenges To simulate possible scenarios in which a Java monitor can be accessed, multiple threads need to be created –Some faults can only be detected by a certain minimum number of threads. But how many? –Multiple threads can display non-deterministic behavior. How to describe and control thread behavior?

A State Exploration-Based Approach Explores the state space of a monitor as an open system –Threads introduced on-the-fly, and as needed, to simulate race conditions –State abstraction used to ensure termination and to control state explosion –Thread behavior is controlled at finer granularity than method calls

Main Results Developed an algorithm that implements the state exploration-based approach Built a prototype tool, called MonitorExplorer, and conducted an empirical evaluation –Empirical results suggest that this approach can be very effective Ongoing work: Extend to general monitors, and general synchronization components Publication: ISSRE06

Outline Software Testing Concurrency Testing –Testing Concurrent Programs –Testing Synchronization Components T-Way Interaction Testing Other Work –Model Checking, Protocol Validation, Pervasive Computing

Combinatorial Testing Creates tests by combining different parameter values –The input space of a test object is represented as a set of parameters and their values Advantages: light specification, test input generation can be fully automated, requires no access to source code How to deal with the combinatorial explosion problem?

T-Way Interaction Testing Motivation: Not every parameter contributes to every fault –Many faults can be exposed by interactions involving a few parameters Given any t parameters, every combination of values of these parameters be covered by at least one test –Allows faults triggered by at most t parameters to be detected Makes an excellent trade-off between test effort and test coverage

T-Way Testing - Example P1 P2 P P1 P2 P Three parameters, each with values 0 and 1 2-way testing exhaustive testing

State of the Art Greedy construction: Each step tries to cover as many combinations as possible –Involves explicit enumeration of all possible combinations Algebraic Construction: test sets are constructed using pre-defined rules Most approaches focus on 2-way (or pairwise) testing

Strategy In-Parameter-Order Input: A set of parameters and values; Output: a t-way test set Construct a t-way test set for the first t parameters Extend the test set to cover each of the remaining parameters one by one –Horizontal growth - extends each existing test by adding one value for the new parameter –Vertical growth – adds new tests, if needed, to make the test set complete

Example (a) P1 P2 P (b) P1 P2 P3 P P1 P2 P3 P * * (c) Horizontal growthVertical growth Four parameters: P1, P2, P3, and P4 P1, P2, and P3 have 2 values P4 has 3 values 3-way test set

Main Results Developed two algorithms, IPOG and IPOG- D, for general t-way testing –Has the lowest algorithmic complexity among the existing algorithms Built a t-way testing tool called FireEye, and conducted an empirical evaluation –Performed significantly better than existing tools, e.g., ITCH, TConfig, Jenny, TestVector Publications: TSE02, ECBS07, STVR07b

TCAS Example t- way FireEyeITCHJennyTConfigTVG SizeTimeSizeTimeSizeTimeSizeTimeSizeTime >1 hour >12 hour >21 hour NA>1 day NA>1 day NA>1 day NA>1 day TCAS: Seven 2-value parameters, two 3-value parameters, one 4-value parameter, two 10-value parameters

Outline Software Testing Concurrency Testing –Testing Concurrent Programs –Testing Synchronization Components T-Way Interaction Testing Other Work –Model Checking, Protocol Validation, Pervasive Computing

Other Work Model checking: How to deal with the state explosion problem? –True concurrency model vs interleaving-based concurrency model –Publication: FM05 Protocol validation: A more specific problem than model checking –Allows multiple threads to proceed simulatenously during state exploration –Publication: CJ06 Pervasive computing –Context-awareness: How to derive high level events from a sequence of low level events? –Power-constraints: How to reduce the power consumption on small mobile devices?

References [STVR07a] Y. Lei, R. Carver, R. Kacker, and D. Kung. A Combinatorial Testing Strategy for Concurrent Programs. Accepted for publication in Software Testing, Verification, and Reliability, [STVR07b] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, J. Lawrence. Two deterministic strategies for multi-way software testing. Software Testing, Verification, and Reliability, pending review. [ECBS07] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, J. Lawrence. A general strategy for multi- way software testing. Int’l Conf. on Engineering of Computer-Based Systems, [ISSRE06] Yu Lei, R. H. Carver, D. Kung, V. Gupta, M. Hernandez. A State Exploration- Based Approach to Testing Java Monitors. Int’l Symposium Software Reliability Engineering, , [CJ06] Q. Ye, Y. Lei, and D. Kung. A Blocking-Based Approach to Protocol Validation, The Computer Journal, 49: , [TSE06] Y. Lei and R. Carver, Reachability testing of concurrent programs, IEEE Transactions On Software Engineering, 32(6): , [FM05] Y. Lei and P. Iyer. An Approach to Unfolding Asynchronous Communication Protocols, Proc. 13th Intl. Symp. on Formal Methods, pp , [TSE02] K. C. Tai and Y. Lei, "A test generation strategy for pairwise testing", IEEE Transactions on Software Engineering, Vol. 28, No. 1, pp , Jan