Taxonomy of Test Oracles Mudit Agrawal Course: Introduction to Software Testing, Fall ‘06 12/3/2018 Software Testing, Fall 2006

Motivation Shortcomings of automated tests (as compared to human ‘eye-ball oracle’) Factors influencing test oracles Environmental (private, global) variables State (initial and present) of the program Dependence on test case What kind of oracle(s) are best suited for which applications? 12/3/2018

Contents Mutating Automated Tests – Challenges for test oracles Generations of Automations and Oracles Challenges for Oracles Types of Oracles Conclusion 12/3/2018

Automated Tests Advantages Limitations: No intervention needed after launching tests Automatically sets up and/or records relevant test environment Evaluates actual against expected results Reports Analysis of pass/fail Limitations: Less likely to cover latent defects Doesn’t do anything different each time it runs Oracle is not ‘complete’ 12/3/2018

12/3/2018

Limited by Test Oracle ‘Predicting’ and comparing the results Playing twenty questions with all the questions written down in advance! Has to be influenced by Data Program State Configuration of the system environment 12/3/2018

Testing with the Oracle Source: Douglas Hoffman, SQM, LLC 12/3/2018

Generations of Automation: First Generation Automation Automate existing tests by creating equivalent exercises Self verifying tests Hard Coded oracles Limitations Handling negative test cases 12/3/2018

Second Generation Automation Automated Oracles Emphasis on expected results – more exhaustive input variation, coverage Increasing Frequency Intensity Duration of automated test activities (load testing) 12/3/2018

Second Generation continued… Random selection among alternatives Partial domain coverage Dependence of system state on previous test cases Mechanism to determine whether SUT’s behavior is expected Pseudo random number generator Test recovery 12/3/2018

Third Generation Automation Take into account knowledge and visibility Software instrumentation Multi-threaded tests Fall back compares Using other oracles 12/3/2018

Third Generation continued… Heuristic Oracles Fuzzy comparisons, approximations Diagnostics Looks for errors Performs additional tests based on the specific type of error encountered 12/3/2018

Challenges for Oracles Independence and completeness – difficult to achieve both Independence from Algorithms Sub-programs, libs Platforms OS Completeness in form of information Comparing computed functions, screen navigations and asynchronous event handling Speed of predictions Time of Execution of oracle 12/3/2018

Challenges for Oracle continued… Better an oracle is, more complex it becomes Comprehensive oracles make up for long test cases (DART paper) More it predicts, more dependent it is on SUT More likely for it to contain the same fault 12/3/2018

Challenges for Oracle continued… Legimitate oracle – an oracle that produces accurate rather than estimated outputs Generates results based on the formal specs What if formal specs are wrong? Very few errors cause noticeable abnormal test termination 12/3/2018

12/3/2018 Source: An automated oracle for software testing

IORL – Input/Output Req. Lang. Graphics based Language Optimal representation between the informal requirements and the target code 12/3/2018

Oracles for different scenarios Transducers That read an input sequence and produce and output sequence Logical correspondence between I/O structures e.g. native file format to HTML conversions in web applications Solution – CFGs System translates a formal specs of I/O files into an automated oracle 12/3/2018

Embedded Assertion Languages [Oracles for different scenarios] Asserts! Problems: Non-local assertions Asserts for pre/post condition pairs with a procedure as a whole e.g. asserts for each method that modifies the object state State caching Saving parts or all of ‘before’ values 12/3/2018

Embedded Assertion Languages [Oracles for different scenarios] Auxiliary variables Quantification 12/3/2018

Extrinsic Interface Contracts [Oracles for different scenarios] Instead of inserting asserts within the program, checkable specs are kept separate from the implementation Extrinsic specs are written in notations Less tightly coupled with target programming language Useful when source-code need not be touched 12/3/2018

Pure Specification Languages [Oracles for different scenarios] Problem with older approach: specs were not pure Z and object-Z are model based specification languages Describe intended behavior using familiar mathematical objects Free of the constraints of the language 12/3/2018

Trace Checking [Oracles for different scenarios] Uses a partial trace of events Such a trace can be checked by an oracle derived from formal specs of externally observable behavior 12/3/2018

Types of Oracles Categorized based on Oracle-Outputs True Oracle Faithfully reproduces all relevant results Uses independent platform, processes, compilers, code etc. Lesser commonality => more confidence in the correctness of results e.g. sin(x) [problem: how is ‘all inputs’ defined? 12/3/2018

Types of Oracles continued… Stochastic Oracle Statistically random input selection Error prone areas of the software are no more or less likely to be encountered sin() – pseudo random generator to select input values 12/3/2018

Types of Oracles continued… Heuristic Oracle Reproduces selected results for the SUT Remaining values are checked based on heuristics No exact comparison Sampling Values are selected using some criteria (not random) e.g. boundary values, midpoints, maxima, minima 12/3/2018

Types of Oracles continued… Consistent Oracle Uses results from one test run as the Oracle for subsequent runs Evaluating the effects of changes from one revision to another 12/3/2018

Comparison 12/3/2018

Who wants to be a Millionaire? 12/3/2018

Which category best describes GUI Testing? a. Heuristic b. Trace Checking c. Transducers d. None 12/3/2018

Thanks! 12/3/2018