A Review of Software Testing - P David Coward Reprinted: Information and Software Technology; Vol. 30, No. 3 April 1988 Software Engineering: The Development Process, Vol 1, Chapter 7 Presented By: Andrew Diemer Software Engineering II – EEL 6883

Aim of paper No guarantee that software meets functional requirements No guarantee that software meets functional requirements Introduces software testing techniques Introduces software testing techniques

Needs Software is to be correct Software is to be correct – What does this mean  It often means the program matches the specifications. Problem with specification Problem with specification – Specification could be wrong

Needs If this happens then the correctness is measured by the software meeting the user requirements If this happens then the correctness is measured by the software meeting the user requirements

Needs Testing Testing – Why test  Tests may have not been adequate enough Asses the performance of the tasks Asses the performance of the tasks

Terminology Verification –vs- Validation Verification –vs- Validation Verification Verification – Ensures correctness from phase to phase of the software life cycle process – Formal proofs of correctness

Terminology Validation Validation – Checks software against requirements  Executes software with test data Author uses testing and checking instead of verification and validation Author uses testing and checking instead of verification and validation

Categories of Testing Two categories of testing: Two categories of testing: – Functional – Non-functional

Functional Testing Functional Functional – Addresses to see if the program obtains the correct output  It is normally used when testing a modified or new program

Functional Testing Regression Testing Regression Testing – Tests following modification – Tests to see if the unchanging functions have indeed changed

Non-functional Requirements Style Style Documentation standards Documentation standards Response times Response times Legal obligations Legal obligations

Situation testing Two situations testing can fall under: Two situations testing can fall under: – Testing which finds faults in the software – Testing which does NOT find faults in the software

Situation testing Finding faults Finding faults – Destructive process  more probing Not finding faults Not finding faults – miss inherent faults  too gentle

Questions How much testing is needed? How much testing is needed? Confidence in testing? Confidence in testing? Ignore faults? Ignore faults? – Which ones are important? Are there more faults? Are there more faults? What is the purpose of this testing? What is the purpose of this testing?

Testing Strategies Functional -vs- Structural Functional -vs- Structural Static -vs- Dynamic analysis Static -vs- Dynamic analysis

Strategy starting points Specification Specification – It makes known the required functions – Asses to see if they are provided – Functional testing

Strategy starting points Software Software – Tests the structure of the system – Structural testing – Functions are included into the system but are NOT required – Example: accessing a database that has not been asked by the user

Functional testing Identify the functions which the software is expected to perform Identify the functions which the software is expected to perform Creating test data that will check to see if these functions are performed by the software Creating test data that will check to see if these functions are performed by the software Does NOT matter how the program performs these functions Does NOT matter how the program performs these functions

Functional testing Rules may be applied to uncover the functions Rules may be applied to uncover the functions Functional testing methods of formal documentation that includes descriptions of faults that correlate to each part of the design and the design features themselves Functional testing methods of formal documentation that includes descriptions of faults that correlate to each part of the design and the design features themselves

Functional testing Isolation of these particular properties of each function should take place Isolation of these particular properties of each function should take place Fault class associations Fault class associations Black box approach Black box approach Testers have an understanding of what the output should be Testers have an understanding of what the output should be

Functional testing Oracle Oracle – An expert on what the outcome of a program will be for a particular test When might the oracle approach not work? When might the oracle approach not work? – Simulation testing  Only provides a “range of values”

Structural testing Testing is based on the detailed design rather than the functions required by the program Testing is based on the detailed design rather than the functions required by the program

Structural testing Two approaches for this testing Two approaches for this testing – First and most common is to execute the program with test cases – Second is symbolic execution  Functions of the program are compared to the required functions for congruency

Structural testing May require single path or percentage testing May require single path or percentage testing Research has been conducted to find out what the minimum amount of testing would be to ensure a degree of reliability Research has been conducted to find out what the minimum amount of testing would be to ensure a degree of reliability

Structural testing Measure of reliability Measure of reliability – All statements should be executed at least once – All branches should be executed at least once – All linear code sequence and jumps in the program should be executed at least once

Structural testing Measure of reliability (cont.) Measure of reliability (cont.) – Best approach would be the exhaustive approach in which every path is tested

Structural testing Problems with the exhaustive approach Problems with the exhaustive approach – Extensive number of paths – Multiple combinations constitutes multiple conditions – Infeasible paths  Contradictions of predicates at conditional statements

Structural testing Path issues Path issues – There is a path for a loop not executing, executing once, and executing multiple of times – Control loops determine the number of paths

Structural testing Path issues Path issues – Known as the “level-i” path or island code – Island code  A series of lines of code, following a program termination, which is not the destination of a transfer control from somewhere else in the program

Structural testing Path issues Path issues – When does island code occur?  When failing to delete redundant code after maintenance

Static analysis Does NOT involve execution of software with data but involves the use of constraints on the input and output data sets mathematically on software components Does NOT involve execution of software with data but involves the use of constraints on the input and output data sets mathematically on software components Examples of static analysis would be program proving and symbolic execution Examples of static analysis would be program proving and symbolic execution

Static analysis Symbolic execution Symbolic execution – Use symbolic values for variables instead of numeric or string values

Dynamic analysis Relies on program statements which make calls to analysis routines Relies on program statements which make calls to analysis routines They record the frequency of execution of elements of the program They record the frequency of execution of elements of the program

Dynamic analysis Act as a liaison between functional and structural testing Act as a liaison between functional and structural testing Test cases monitored dynamically, then structurally tested to see what idle code is left by previous tests Test cases monitored dynamically, then structurally tested to see what idle code is left by previous tests Shows functions the program should perform Shows functions the program should perform

Classification of Techniques There are three classifications: There are three classifications: Static – Structural Static – Structural – Symbolic execution – Partition analysis – Program proving – Anomaly analysis

Classification of Techniques Dynamic - Functional Dynamic - Functional – Domain testing – Random testing – Adaptive perturbation – Cause-effect graphing

Classification of Techniques Dynamic - Structural Dynamic - Structural – Domain and computational testing – Automatic test data generation – Mutation analysis

Classification of Techniques Dynamic - Structural Dynamic - Structural – Domain and computational testing – Automatic test data generation – Mutation analysis

Symbolic execution Non traditional approach Non traditional approach – traditional is the execution requires that a selection of paths through a program is exercised by a set of test classes Produces a set of expressions, one per output variable Produces a set of expressions, one per output variable

Symbolic execution Usually a program executes using input data values with the output resulting in a set of actual values Usually a program executes using input data values with the output resulting in a set of actual values Use of flow-graphs Use of flow-graphs – Each branch contains decision points (directed graph) – Branch predictions are produced

Symbolic execution Use of top down approach Use of top down approach During the top down traversal, the input variable is given a symbol in place of an actual value During the top down traversal, the input variable is given a symbol in place of an actual value A problem is in the iterations A problem is in the iterations As mentioned before, no executing loop, executing once, and then executing multiple times. As mentioned before, no executing loop, executing once, and then executing multiple times.

Partition analysis Uses symbolic execution to find sub domains of the input domain Uses symbolic execution to find sub domains of the input domain Path conditions are used to find them Path conditions are used to find them Input domains that cannot be allocated to a sub domain infer a fault Input domains that cannot be allocated to a sub domain infer a fault

Partition analysis Specifications need to be written at a higher lever Specifications need to be written at a higher lever

Program proving At the beginning and end of the procedure, place a mathematical method assertions At the beginning and end of the procedure, place a mathematical method assertions Similar to symbolic execution Similar to symbolic execution Neither of them execute actual data and both examine source code Neither of them execute actual data and both examine source code

Program proving Tries to come up with a proof that encompasses all possible iterations Tries to come up with a proof that encompasses all possible iterations Program proving steps: Program proving steps: – Construct a program – Examine the program and insert mathematical assertions at the beginning and end of procedures

Program proving Program proving steps (cont): Program proving steps (cont): – Determine whether the code between each pair of start and end assertions will achieve the end assertion given the start assertion – If the code reaches the end assertion then the block has been proven

Program proving DeMillo says that proofs can only be stated as acceptable and not correct DeMillo says that proofs can only be stated as acceptable and not correct His acceptance is determined by a gathering of people who cannot find fault with the proof His acceptance is determined by a gathering of people who cannot find fault with the proof

Program proving The larger the audience, the more confidence in the software The larger the audience, the more confidence in the software Total correctness means loops will terminate Total correctness means loops will terminate

Anomaly analysis Two levels of anomalies: Two levels of anomalies: – made by the compiler (language syntax) – problems that are not necessarily wrong by the programming language

Anomaly analysis Some anomalies are: Some anomalies are: – Unexecutable code – Array boundaries – Initializing variables wrong – Unused labels and variables – Traversing in and out of loops

Anomaly analysis Produce flow-graphs Produce flow-graphs Determine infeasible paths Determine infeasible paths Some use data-flow analysis Some use data-flow analysis – Where the input values turn into intermediate, which then turn into output values

Anomaly analysis Some data-flow anomalies: Some data-flow anomalies: – Assigning values to variables which are not used – Using variables that have not been assigned previously to a value – Re-assigning of a variable without making use of a previous variable – Indicates possible faults

Domain testing Based upon informal classifications of the requirements Based upon informal classifications of the requirements Test cases executed and compared against the expected to determine whether faults have been detected Test cases executed and compared against the expected to determine whether faults have been detected

Random testing Produces data without reference to code or specification Produces data without reference to code or specification Random number generation is used Random number generation is used Main problem is there is no complete coverage guarantee Main problem is there is no complete coverage guarantee

Random testing Key is to examine small subsets Key is to examine small subsets If followed it will give you a high branch coverage success rate If followed it will give you a high branch coverage success rate

Adaptive perturbation testing Concerns with assessing the effectiveness of sets of test cases Concerns with assessing the effectiveness of sets of test cases Used to generate further test cases for effectiveness Used to generate further test cases for effectiveness

Adaptive perturbation testing Optimization is reached when routines find the best value to replace the discarded value so the number of assertions is maximized Optimization is reached when routines find the best value to replace the discarded value so the number of assertions is maximized Process is replaced until the violated assertions are maximized to the limit Process is replaced until the violated assertions are maximized to the limit

Cause-effect graphing Power comes from the input combinations used by logic (AND, OR, NOT, etc) Power comes from the input combinations used by logic (AND, OR, NOT, etc)

Cause-effect graphing Five steps: Five steps: – Divide into workable pieces – Identify cause and effect – Represent a graph to link cause and effect semantics

Cause-effect graphing Five steps (cont): Five steps (cont): – Annotate to show impossible combinations and effects – Convert the graph into a limited- entity decision table It helps identify small test cases It helps identify small test cases

Domain and computational testing Based upon selecting test cases Based upon selecting test cases Assignment statements are used for deciding computational paths Assignment statements are used for deciding computational paths

Domain and computational testing Paths considered: Paths considered: – Path computation  Set of algebraic expressions, one for each output variable, in terms of input variables and constraints – Path condition  A joining of constraints of a path

Domain and computational testing Paths considered (cont): Paths considered (cont): – Path domain  Set of input values that satisfy the path condition – Empty path  Infeasible paths and cannot execute A path that follows errors is a computation error A path that follows errors is a computation error

Automatic test data generation Courage comes with covering metrics Courage comes with covering metrics Contradictory paths are infeasible Contradictory paths are infeasible Needs detailed specification to achieve this testing Needs detailed specification to achieve this testing Formal specifications may provide fundamental help Formal specifications may provide fundamental help

Mutation analysis Concerns the quality of sets of test data Concerns the quality of sets of test data Uses the program to test the test data Uses the program to test the test data This means the original and mutant program are tested using the same test data This means the original and mutant program are tested using the same test data

Mutation analysis The two outputs are compared The two outputs are compared If the mutant output is different from the original output, then the mutant is of little value If the mutant output is different from the original output, then the mutant is of little value If the two outputs are the same, then the problem is there has been a change If the two outputs are the same, then the problem is there has been a change

Mutation analysis The mutant is then said to be live The mutant is then said to be live Ratios are then taken (dead/alive) Ratios are then taken (dead/alive) High ratio of live mutants equals poor test data High ratio of live mutants equals poor test data If this happens then more tests need to be run until the ratio goes down If this happens then more tests need to be run until the ratio goes down

Conclusion I thought this paper was thorough I thought this paper was thorough This paper gave good examples (compartmentalized) This paper gave good examples (compartmentalized) I thought this paper was a little out of date I thought this paper was a little out of date