1. Analyzing Regression Test Selection Techniques
-presented by Xuan Lin

2. Outline Introduction Concepts and Assumptions Analysis Framework
Examples Techniques
Conculsion and Discussion

3. Outline Introduction Concepts and Assumptions Analysis Framework
Examples Techniques
Conculsion and Discussion

4. Introduction What is Regression Testing? -Everybody knows …
Retest-all strategy VS. Test Selection
Notions: P,P’,S,S’,T,T’,

5. Typical Selective Retest Process
1. Select T'≤T, a set of tests to execute on P'
Regression Test Selection Problem
2.Test P’ with T’, to establish the correctness of P’ with respect to T’
Test Suite Execution Problem
3.If necessary, create T’’, a set of new functional or structural tests for P’
Coverage Identification Problem
4.Test P’ with T’’, to establish the correctness of P’ with respect to T’’
Test Suite Execution Problem
5.Create T’’’, a new test suite and test history for P’, from T, T’ and T’’
Test Suite Maintenance

6. Typical Selective Retest Process
1. Select T'≤T, a set of tests to execute on P'
Regression Test Selection Problem
2.Test P’ with T’, to establish the correctness of P’ with respect to T’
Test Suite Execution Problem
3.If necessary, create T’’, a set of new functional or structural tests for P’
Coverage Identification Problem
4.Test P’ with T’’, to establish the correctness of P’ with respect to T’’
Test Suite Execution Problem
5.Create T’’’, a new test suite and test history for P’, from T, T’ and T’’
Test Suite Maintenance

7. Test Selection Techniques
Specification-based VS. Code-based
Three distinct goals of code-based test selection techniques
-Coverage Techniques
Compare and Evaluation !!!
-Minimization Techniques
-Safe Techniques

8. Outline Introduction Concepts and Assumptions Analysis Framework
Examples Techniques
Conculsion and Discussion

9. Concepts and Assumptions
Fault-realing for P’: cause P’ to fail
-No Effective procedure by which to find tests in T
that are fault-realing for P’ [1]
-Under certain conditions, a technique can select a
Superset of the set of fault-revealing for P
Modification-revealing: casue the outputs of P and P’ to differ.

10. Concepts and Assumptions
Modification-revealing = Fault-revealing ???
P-Correct-for-T Assumption: For each test t in T, when P was tested with t, P halted and produced the correct output
Obsolet-Test-Identification Assumption: There is an effective procedure for determining, for each test in t, whether t is obsolete for P’.
Test t is obsolete for P’ if and only if t either specifies an input to P’ that, according to S’, is invalid for P’, or t specifies an invalid input-output relation for P’

11. Concepts and Assumptions
Up to now, we can find the fault-revealing test cases by:
1. Run our procedure for identifying obsolete test in T.
2. Remove them.
3. Find the modification-revealing test cases.
- In the set of non-obsolete test cases, modification-revealing=fault-revealing

12. Concepts and Assumptions
Obsolete
Fault-Revealing
Nonbsolete
???
Fault-Revealing
Modification-Revealing

13. Concepts and Assumptions
Modification-traversing: a test t is modification-traversing for P and P’ if and only if it (a) executes new or modified code in P’, or (b) formerly executed code that has since been deleted

14. Concepts and Assumptions
Obsolete
Fault-Revealing
Nonbsolete
Fault-Revealing
Modification-Revealing
???
Modification-Traversing

15. Concepts and Assumptions
Controlled Regression Testing Assumption: when P’ is tested with t, we hold all factors that might infuence the output of P’, except for the code in P’, constant with respect to their states when we tested P with t.

16. Why We Need Define These Concepts and Assumptions?
Evaluate test selection techniques in terms of their ablities to select and avoid discarding fault-revealing tests.
Three classes can be used to distinguish techniques even CRTA is not satisfied.
Coverage techniques may omit tests from T’ that may reveal faults in P’

17. Outline Introduction Concepts and Assumptions Analysis Framework
Examples Techniques
Conculsion and Discussion

18. Analysis Framework
Incusiveness
Precision
Efficiency
Generality

19. Analysis Framework-Inclusiveness

20. Analysis Framework-Inclusiveness
There is no algorithm to determine the inclusiveness!
However…
We can prove M is safe.
We can prove M is not safe.
We can compare techinques in terms of inclusiveness
We can experiment to approximate

21. Analysis Framework-Precision

22. Analysis Framework-Precision
There is no algorithm to determine the precision!
However…
We can compare techinques in terms of precision.
We can prove M is not precise
We can show M is precise.
We can experiment to compare.

23. Analysis Framework-Efficiency
Time & Space
Cost of selecting T’ < the cost of running T-T’
Three Factors
1.preliminary phase vs. critical phase
2.automatability
3. calculation informatin on program modifications
4.ability to handle multiple modifications

24. Analysis Framework-Generality
Should function for some identifiable and practical class of program
Should handle realistic program modifications
Should be independent of assumptions about testing or maintenance enviroments.
Should be independent of particular program analysis tools
Should support intraprocedural or interprocedural test selection

25. Analysis Framework-Tradeoffs
Precision vs. Efficiency
- both safe and unsafe
Inclusiveness vs.Efficiency
-not safe
Generality vs. Inclusiveness, Efficiency or Precision
Multiple modication vs. Efficiency

26. Outline Introduction Concepts and Assumptions Analysis Framework
Examples Techniques
Conculsion and Discussion

27. Refresh… Obsolete Fault-Revealing Nonbsolete Fault-Revealing
Modification-Revealing
Modification-Traversing

28. Depiction of inclusiveness and precision

29. Retest-all

30. Optimum

31. Examples: Dataflow Caculate d-u pairs for both P and P’
Identify and select d-u pairs that are new in, or modified for P’
Some techniques also select deleted d-u pairs
Incremental / Nonincremental

32. Examples: Dataflow-Inclusion
Not safe

33. Examples: Dataflow-Precision
Not precise

34. Examples: Dataflow

35. Examples: Dataflow-Effiency
Incremental- O(|T|*|P’|*|P’|)
Nonincremental-

36. Examples: Dataflow-Generity
Applied to procedural programs generally.
Function for all program changes except those that do not alter d-u association
Some techiques applied to intraprocedural programs while others applied to interprocedural programs
Incremental approach requires incremental dataflow analysis tools.

37. Examples: Graph Walk Techniques
Build CFG for P and P’
Collects traces for tests with CFG edges.
Performs synchronous depth-first traversals of the two graphs, selects those are not lexically identical.

38. Examples: Graph Walk Techniques-Inclusiveness
[1] shows that for controlled regression testing, the techniques will select all modification-traversing test.
So, it is safe.

39. Examples: Graph Walk Techniques-Precision
Not precise
Multiply-visisted-node

40. Examples: Graph Walk Techniques

41. In practice

42. Improved version

43. Examples: Graph Walk Techniques-Efficiency
Generally:
Property not hold[1]:

44. Examples: Graph Walk Techniques-Generality
Apply to procedural languages generally
All type of modifications
Both interprocedure and intraprocedure
No assumption on test suite or coverage
Require tools for constructing dataflow and tools for dataflow analysis

45. Examples: Path Analysis
Takes set of program paths in P’ expressed as an algebraic expression
Manipulates the expression to get a set of cycle-free exemplar paths.
Compare such paths in P with P’
Tests that traverse modified exemplar paths will be selected

46. Examples: Path Analysis-Inclusiveness
Selects only modified paths and omits the cancel and new paths.
Not safe.

47. Examples: Path Analysis-Precision
It will select all the test cases that are modification-traversing and execute modified exemplar paths.

48. Examples: Path Analysis

49. Examples: Path Analysis-Efficiency
Exponential in |P| and |P’|

50. Examples: Path Analysis-Generality
Assumption: low-level program designs are depicted by language-independent algebraic representations.
Does not handle test cases for additions or deletings of code.
Does not require any coverage criterian or test generation technique.
Require tool for collecting traces at the statement level.

51. Conclusions
Framework for evaluating regression test selection technique that classifies techniques in terms of inclusiveness, precision, efficiency, and generality.
Several test selection techiques are evaluated

52. Reference
[1]G.Rotherel. Efficient, Effective Regression Testing Using Safe TestSelection Techniques.