1. An Extensive Study of Static Regression Test Selection in Modern Software Evolution
Owolabi Legunsen, Farah Hariri, August Shi,
Yafeng Lu, Lingming Zhang, and Darko Marinov
CS 498ST Lecture
09/21/2017
University of Illinois at Urbana Champaign
Hello everyone, my name is Owolabi and today, I will be presenting “An extensive study of static regression test selection in modern software evolution”.
This is work that I did together with Farah, August, Yafeng, Lingming and Darko.
We are affiliated with the University of Illinois and the University of Texas at Dallas.
CCF , CCF ,
CCF , CCF ,
CCF

2. Regression Testing A T1 T1 B T2 T2 C C T3 T3 D T4 T4 E F
In regression testing, all tests are run after every change to ensure that code changes did not break existing functionality.
To illustrate, let’s consider the following example that I will use in the rest of my talk. Suppose that a codebase consists of entities A through F, which have the dependencies shown by these arrows. For example, the arrow between A and B means that A depends on B. Further, there are Tests T1 through T4 for this code. Now, let’s say that a change is made to F. In regression testing, all the tests are rerun.
The problem with regression testing is that it can be slow, especially when there are many tests. T4 T4 E F

3. Recall Regression Testing Pros & Cons
Automated testing after every change
Early detection of faults
Ensures faults that are fixed stay fixed
Cons:
Long time to run (N is very large!)
Expensive to maintain
Potentially unreliable (flaky tests)

4. Regression Testing
Rerun tests to ensure that code changes did not break existing functionality A T1 T1 B T2 T2 C C T3 T3 D
In regression testing, all tests are run after every change to ensure that code changes did not break existing functionality.
To illustrate, let’s consider the following example that I will use in the rest of my talk. Suppose that a codebase consists of entities A through F, which have the dependencies shown by these arrows. For example, the arrow between A and B means that A depends on B. Further, there are Tests T1 through T4 for this code. Now, let’s say that a change is made to F. In regression testing, all the tests are rerun.
The problem with regression testing is that it can be slow, especially when there are many tests. T4 T4 E F
Problem: Regression testing can be very slow! (many tests)

5. Regression Testing can be SLOW!
August Shi told you about Regression Testing at Microsoft
Tests for a single product at CompanyA takes 7 days!
Let’s look at some examples from open-source

6. Regression Testing can be SLOW! (2)
In Parallel
In Sequence

7. Regression Testing can be SLOW! (3)
In Parallel
In Sequence

8. Discussion
What is your own personal experience with slow regression testing?
What would be your response if your tests took 5 hours to run after every change?

9. Today’s Lecture
Speeding up Regression Testing through Regression Test Selection (RTS)
Discuss FSE 2016 paper in which we evaluated various approaches to RTS
Demonstrate Open-source RTS tool, STARTS (

10. Regression Test Selection (RTS)
Speed up regression testing by rerunning only tests that are affected by code changes A
Quiz: Which classes can change behavior due to change in C? T1 B T2 C C
Quiz: Which test classes can need to be rerun? T3 D T4 E
The goal of regression test selection, RTS, is to speed up regression testing by rerunning only tests that are affected by code changes.
Given our previous example and the change to F, an RTS technique will first find that both A and B depend on F and then select to rerun only T1 and T1, which are the only tests that are affected by the changes.
Finding dependencies among classes can be done statically or dynamically.
In this paper, we studied static RTS techniques and compared them with a state-of-the-art dynamic RTS technique. F

11. Regression Test Selection (RTS)
Speed up regression testing by rerunning only tests that are affected by code changes A A T1 T1 B B T2 T2 C C T3 D
Finding dependencies can be done statically or dynamically
The goal of regression test selection, RTS, is to speed up regression testing by rerunning only tests that are affected by code changes.
Given our previous example and the change to F, an RTS technique will first find that both A and B depend on F and then select to rerun only T1 and T1, which are the only tests that are affected by the changes.
Finding dependencies among classes can be done statically or dynamically.
In this paper, we studied static RTS techniques and compared them with a state-of-the-art dynamic RTS technique. T4 E F
This paper: we studied static RTS approaches and compared with state-of-the-art dynamic RTS

12. Motivation for our Study
Dynamic RTS has been getting adopted recently
Ekstazi ( is used in several projects
Clover ( is an open-source code coverage tool that also does RTS
Dynamic RTS may not always be applicable
Instrumentation costs can be high
Dependencies may be incomplete, e.g., due to non-determinism
Static RTS was proposed previously but not evaluated at scale on modern software
We conducted this study for the following reasons:
Although dynamic RTS has been gaining adoption recently, it has some limitations that make it to not always be applicable. For example dynamic RTS relies on instrumentation which can be costly and the dependencies it captures may be incomplete in cases where the code under test exhibits nondeterminism.
In addition, although static RTS was proposed over two decades ago, it was not evaluated at scale on modern software systems.

13. How RTS works
Find
Dependencies
Code + Tests
Dependencies
Analyze
Dependencies
Changes
Affected Tests
Before I go into details about our study, let me first say more about RTS in general.
There are two inputs to an RTS technique: the code plus tests and the changes between two versions of the codebase. The output are the tests that are affected by the changes. First, an RTS technique finds the dependencies among the various entities in the codebase. Then, using the changes, these dependencies are analyzed in order to find the affected tests.
Note that an affected test is one that can behave differently after the code changes and a test is affected if any of its dependencies changed.
An affected test can behave differently due to code changes
A test is affected if any of its dependencies changed

14. Finding and Analyzing Dependencies
Dependencies: entities that can affect test behavior A
Finding Dependencies:
T1 depends on A, B, C, D
T2 depends on B, C
T3 depends on E
T4 depends on D, E, F T1 B T2 C C T3 D
A test’s dependencies are any entities that can affect test behavior.
To illustrate the finding and analyzing of dependencies in an RTS technique, let us consider again our example, where F is the only change. An RTS technique first finds the dependencies for each test. In this case, it finds that T1 depends on A, B, C and F, that T2 depends on B and F, and so on. Note that each test depends on itself.
Now, given the information that only F changed, these dependencies are analyzed to find which tests depend on F, resulting in T1 and T2 being returned as the affected tests. T4 E
Quiz: Did we capture all the dependencies? F

15. Finding and Analyzing Dependencies
Dependencies: entities that can affect test behavior A
Finding Dependencies:
T1 depends on A, B, C, D, T1
T2 depends on B, C, T2
T3 depends on E, T3
T4 depends on D, E, F, T4
Analyzing Dependencies:
T1 & T2 are affected T1 B T2 C C T3 D
A test’s dependencies are any entities that can affect test behavior.
To illustrate the finding and analyzing of dependencies in an RTS technique, let us consider again our example, where F is the only change. An RTS technique first finds the dependencies for each test. In this case, it finds that T1 depends on A, B, C and F, that T2 depends on B and F, and so on. Note that each test depends on itself.
Now, given the information that only F changed, these dependencies are analyzed to find which tests depend on F, resulting in T1 and T2 being returned as the affected tests. T4 E F

16. Important RTS Considerations
End-to-end time of RTS must be less than time to run all tests
Run All Tests
Time Savings
Find Dependencies
Analyze
Run Affected Tests
End-to-End Time for RTS
There are certain considerations that are very important for any RTS technique.
First, the end-to-end time for any RTS technique must be less than the time to simply rerun all the tests. Suppose that this big bar represents the time to run all tests. The time for RTS is shown below as the sum of the time to find and analyze dependencies as we as to run the affected tests. This is the end-to-end time for RTS, and the goal is to keep this end-to-end time as small as possible to maximize the time savings.
There are two other important considerations. First, RTS is “safe” if it selects to rerun all affected tests, and “precise” if it selects to “rerun” only affected tests.
RTS is safe if it selects to rerun all affected tests
RTS is precise if it selects to rerun only affected tests

17. RTS Techniques Evaluated
Finding dependencies can be done dynamically or statically
Dependencies can be at different levels of granularity, e.g., methods, classes, jar files, etc.
In this paper, we compare these approaches:
Class-Level Dynamic
Class-Level Static
Method-Level Static
End-to-End Time
Safety
Precision ? ? ?
Like I mentioned before, finding dependencies can be done statically or dynamically. In addition, dependencies can be at different levels of granularity such as methods, classes, jar files and so on. In this paper, we evaluated and compared three RTS techniques in terms of End-to-end time, safety and precision: a class-level dynamic technique, a class-level static technique and a method-level static technique.
We found the method-level RTS technique performed rather poorly, compared with the class-level techniques and I will not be discussing it in the rest of my talk. Please see our paper for the details.
See details on method-level RTS in paper

18. Class-Level Dynamic RTS (Ekstazi[1])
Find Dependencies: dynamically track classes used while running each test class
Changes: classes whose .class files differ
Analyze Dependencies: select test classes for which any of its dependencies changed
The class-level RTS technique that we evaluated is Ekstazi, which was proposed by Gligoric et. al. in their ISSTA 15 paper. To find dependencies, Ekstazi dynamically tracks what classes are used while running each test class. Ekstazi computes as changed, any whose bytecode changed since the previous version and Ekstazi selects to rerun test classes for which at least one dependency changed.
[1] M. Gligoric, L. Eloussi, and D. Marinov. Practical Regression Test Selection with Dynamic File Dependencies. ISSTA 2015

19. Class-Level STAtic RTS (STARTS[2])
First, statically build a class dependency graph
Each class has an edge to direct parents and referenced classes
Find Dependencies: classes reachable from test classes in the graph
Changes: computed in same way as Ekstazi
Analyze Dependencies: select test classes that reach a changed class in the graph
The static class-level RTS technique that we evaluated works in the following way: First, it builds a class dependency graph in which each class has an edge to its it’s direct parents and any classes that it references. Specifically, the dependency graph is an IRG, as described by Orso et. al., in their FSE 2004 paper. Once the IRG is constructed, dependencies are computed as all classes that are reachable from each test class in the IRG. Changes are computed the same way as in Ekstazi and the tests that are selected to be rerun are those that can reach a changed class in the IRG.
[2]

20. Variants of RTS Techniques studied
We studied 12 RTS techniques in total
2 variants of the static/dynamic class-level RTS
Offline: pre-compute dependencies before changes are known
Online: compute dependencies after changes are known
8 variants of static method-level RTS technique
Going into more details about our study, we studied 12 RTS techniques in total.
We studied 2 variants each of the class-level RTS techniques: an offline variant and an online variant. In the offline variant, dependencies are precomputed from the old version, before changes are known. That way, once the changes are known, it is much faster to analyze the dependencies and find the affected tests. However, in the online variant, dependencies are computed on the fly, after the changes are known.
We also studied 8 variants of the method-level RTS technique and you can find the details in our paper.
The main reason for online offline is performance
See details on method-level RTS in paper

21. Research Questions
RQ1: How do RTS techniques compare w.r.t. number of tests selected?
RQ2: How do RTS techniques compare w.r.t. end-to-end time?
RQ3: How do static RTS techniques compare with class- level dynamic RTS in terms of precision and safety?
RQ4: How do variants of MethSRTS influence the cost/safety trade-offs?
We asked four research questions in our evaluation.
First, we wanted to know how the different techniques compared in terms of the number of tests selected. Second we wanted to know how the techniques compared in terms of their end-to-end time. Third, we wanted to know how the static RTS techniques compared with dynamic RTS in terms of safety and finally, we wanted to know how the variants of the method-level static RTS influenced the cost/safety tradeoff.
In this talk, I will only discuss the first three RQs, please see our paper for the results concerning RQ4…
See answer to RQ4 in paper

22. Experimental Setup 22 open-source projects from ASF and GitHub
Single-module Maven projects with JUnit4 tests
Project sizes: from 2 kLOC to 185 kLOC
985 revisions of these 22 projects
Selection criteria: subset of latest 100 commits
Compile successfully
All tests pass
Ekstazi runs successfully
We evaluated all these RTS techniques on 22 open-source projects selected from the Apache Software Foundation and from Github. These were all single-module Maven projects with Junit4 tests and the project sizes ranged from two thousand lines of code to 185 thousand lines of code.
We used a combined total of 985 revisions of these 22 projects and these revisions were selected in the following way. We started from the latest 100 commits in each project and selected the subset of revisions that satisfied three criteria: compilation was successful, all tests pass and Ekstazi runs successfully.

23. Ekstazi selects fewer tests
RQ1: Tests Selected
This plot shows the percentage of tests selected by static class-level RTS and Ekstazi, relative to rerunning all tests in all versions of the 22 projects in our study. The red boxes represents Ekstazi and the blue boxes represents the static class-level RTS technique.
As expected, for most of these projects, Ekstazi selects fewer tests than class-level SRTS, when comparing the means, as shown by the direction of these arrows.
On average across all these projects, Ekstazi selects 20.6% of tests while the static class-level RTS selected 29.4% of tests.
Ekstazi selects fewer tests
than STARTS
20.6% vs. 29.4%

24. RQ2: End-to-End Time
For RQ2, this table shows the min, max, and average end-to-end times as percentages relative to the time to running all tests in all 22 projects.
We’ve sorted the projects in decreasing order of relative end-to-end time. We can see in these cases, both static and dynamic RTS were slower than simply rerunning all the tests. Running all the tests in these projects took very little time so RTS overhead slowed down the testing process.
If we compare averages, Ekstazi and ClassSRTS have very similar end-to-end times, which is notable when it is considered that ClassSRTS selected more tests, on average than Ekstazi.

25. RQ3: Safety and Precision
Safety and precision were measured against Ekstazi
Safety violation: STARTS misses Ekstazi-selected tests:
Precision violation: STARTS selects tests that Ekstazi does not:
𝑆𝑎𝑓𝑒𝑡𝑦𝑉𝑖𝑜𝑙𝑎𝑡𝑖𝑜𝑛= |𝐸 \ 𝑆| |𝐸 ∪𝑆|
𝐸=𝑡𝑒𝑠𝑡𝑠 𝑠𝑒𝑙𝑒𝑐𝑡𝑒𝑑 𝑏𝑦 𝐸𝑘𝑠𝑡𝑎𝑧𝑖
𝑆=𝑡𝑒𝑠𝑡𝑠 𝑠𝑒𝑙𝑒𝑐𝑡𝑒𝑑 𝑏𝑦 𝑆𝑇𝐴𝑅𝑇𝑆
For RQ3, we measured the safety and precision of ClassSRTS relative to Ekstazi. Specifically, we measured “safety violation”, which penalized ClassSRTS for missing tests that Ekstazi selects. We also measured “precision violation, which penalized ClassSRTS for selecting tests that Ekstazi did not select.
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛𝑉𝑖𝑜𝑙𝑎𝑡𝑖𝑜𝑛= |𝑆 \ 𝐸| |𝐸 ∪𝑆|

26. RQ3: Safety and Precision
𝑆𝑎𝑓𝑒𝑡𝑦𝑉𝑖𝑜𝑙𝑎𝑡𝑖𝑜𝑛= |𝐸 \ 𝑆| |𝐸 ∪𝑆|
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛𝑉𝑖𝑜𝑙𝑎𝑡𝑖𝑜𝑛= |𝑆 \ 𝐸| |𝐸 ∪𝑆|
Here are the results. In this table, the revs column shows the percentage of revisions where there is a violation (compared to Ekstazi), while the min, max and avg percentages refer to the degree of such violations when they occur.
We can see that there are very few revisions with safety violations, and we also see the average percentage of safety violations in those revisions. Overall ClassSRTS was only unsafe in 0.2% of the revisions and the average safety violation was 6.8%.
Here’s what the numbers looked like for precision violations. ClassSRTS was imprecise in 33% of the revisions and the overall average precision violation was 42.9%

27. Reflection caused all Safety Violations
Example simplified from Apache commons-math
STARTS misses this edge because it is not aware of reflection
AbstractIntegrator
InterpolatorTest
AbstractInterpolatorTest
Integrator
name = interpolatorName.replaceAll("Interpolator", "Integrator");
Class clz = (Class) Class.forName(name);
i = clz.getConstructor(…).newInstance(field, field.getOne());
Now that we’ve seen that ClassSRTS incurred safety and precision violations, lets look into *why* violations happened. The reason for safety violations in our experiments was reflection. Consider the following safety violation from apache commons-math. We show only the relevant portion of the class dependency graph here, and we see that InterpolatorTest depends on AbstractInterpolatorTest and that Integrator depends on AbstractIntegrator, which is a changed class. Given this change, ClassSRTS fails to select to rerun InterpolatorTest because it misses this edge. What is happening is that AbstractIntegrator uses reflection to dynamically create instances of Integrator, so there is a dynamic dependency. However, as ClassSRTS is not aware of reflection, it misses this edge.

28. Since this paper was accepted …
We open-sourced STARTS
STARTS now handles multi-module Maven projects
STARTS now finds dependencies from bytecode much faster
We are making STARTS safer with respect to reflection
We are evaluating STARTS on larger software systems
me if interested in tool paper describing STARTS implementation

29. Quick STARTS Demo Video Demo: https://youtu.be/PCNtk8jphrM
README shows how to add STARTS to a project
ADME.md

30. Conclusions
We performed the first, large-scale empirical study of static RTS and its comparison with dynamic RTS
At the class level, we found static RTS (STARTS) comparable with state-of-the-art dynamic RTS (Ekstazi)
Similar end-to-end times
STARTS had very few safety violations
Method-level static RTS requires more work to be usable
To conclude, the work that I just presented is the first study of static RTS and its comparison with dynamic RTS.
We found class-level static RTS to be comparable with class-level dynamic RTS in terms of end-to-end time, while incurring very few safety violations.
However, at the method level, static RTS performed poorly and requires more work to be usable.
We are now working on making static RTS safe with respect to reflection.
If you have any questions, feel free to me at this address. I’ll be happy to take questions now.

31. STARTS Projects that you can do
Estimate test-running time for “mvn starts:select”
Fix integration tests on Windows and add more (>40) integration tests
Make STARTS analysis incremental and faster. Currently on each revision, STARTS
parses entire application code to find dependencies
builds dependency graph from scratch
finds dependencies for all tests

32. STARTS Projects that you can do (2)
Implement Maven goals in STARTS for online mode and offline mode and evaluate on many projects
Evaluate STARTS vs. Ekstazi on at least 10 projects where tests take at least 1 hour to run
Make STARTS work on Travis and shadow many projects
Add STARTS support for other Build Systems: Gradle, Bazel