1. Automatic Detection of Previously-Unseen Application States for Deployment Environment Testing and Analysis Chris Murphy, Moses Vaughan, Waseem Ilahi, Gail Kaiser Columbia University

2. 2 Chris Murphy, Columbia University Overview Deployment environment testing and analysis can find defects not found prior to release Approaches potentially suffer from high overhead We seek to increase the efficiency by only running tests/analysis in previously-unseen application states

3. 3 Chris Murphy, Columbia University In Vivo Testing [Murphy et al. ICST’09] Automatically conduct tests as the software is running in the deployment environment  Unit tests, integration tests, etc.  “Runtime assertions with side effects” Looks for defects in previously untested application states

4. 4 Chris Murphy, Columbia University In Vivo test Function f is about to be executed with input x in state S Create a sandbox for the test Execute f(x) Program continues Execute INVtest_f(x) in state S Report violations

5. 5 Chris Murphy, Columbia University Research Question Can the approach be made more efficient by only running tests and performing analysis in application states that the program has not seen before? Yes, assuming we can quickly determine whether the current state has not previously been seen

6. 6 Chris Murphy, Columbia University Broader Impact Other approaches may also use data about deployment environment execution  Monitoring: Anomaly detection  Profiling: Path coverage, line coverage, etc.  Fault localization All of these may benefit if analysis is done only in previously-unseen states

7. 7 Chris Murphy, Columbia University Analysis Overhead of standard approach  T standard = N * t i Overhead in previously unseen states  T unseen = N * DSP * (t d + t u + t i ) Overhead in previously seen states  T seen = N * (1-DSP) * t d Find DSP so that T unseen + T seen ≤ T standard Number of tests Time per test Distinct State Percentage Time to determine if state has been seen before Time to update list of seen states

8. 8 Chris Murphy, Columbia University Analysis Results To be more efficient, we need:  DSP ≤ (t i – t d ) / (t i + t u ) If t d and t u are much less than t i the right side of the inequality approaches 1 This means that even if nearly all states are distinct, this new approach will still be more efficient

9. 9 Chris Murphy, Columbia University Implementation Issues How do we define the “state”? How do we represent the state? How can we quickly determine whether a state has previously been seen? Does the previous analysis hold in the real world?

10. 10 Chris Murphy, Columbia University Defining “State” We define “state” as “the values of all variables that are in scope at a given execution point” For the purposes of In Vivo Testing, this can be further refined to “the values of all variables on which a function depends that are in scope at the start of the function execution”

11. 11 Chris Murphy, Columbia University Example We can statically determine that function f1 depends on: parameters p1, p2, p3 ; and global variables a and b

12. 12 Chris Murphy, Columbia University Representing States Given our definition, the “state” is simply a map between variable names and values We want to avoid element-wise comparison We want to avoid false positives and false negatives a = 4 b = 3 … a = 2 b = 5 … a = 1 b = 8 … a = 2 b = 7 …

13. 13 Chris Murphy, Columbia University Cantor Function Goal: Give each state a distinct value Hashing function that assigns a distinct number to a pair of numbers [Royden 1988] f(k1,k2) = (1/2)(k1+k2)(k1+k2+1) + k2 Can be used recursively over a set of numerical values

14. 14 Chris Murphy, Columbia University Tracking Execution States Even if each state has its own representation, how can we quickly determine whether a state has been seen before? Hashtable: O(n) in worst case Bloom filter: O(1), but allows for false positives 27189133742618 ?

15. 15 Chris Murphy, Columbia University Judy Array Scalable, space efficient, speed efficient  Developed by Hewlett-Packard in 2001 Highly-optimized 256-ary prefix tree data structure Lookups are O(log 256 n) Now we can quickly detect whether a state has already been seen

16. 16 Chris Murphy, Columbia University Automated Process 1. Statically analyze the source code to determine which parts of the state the function depends on 2. Create code that uses Cantor function to represent the state and Judy Array to determine whether it had already been seen 3. Generate instrumentation as normal, with call to function created in Step 2

17. 17 Chris Murphy, Columbia University Evaluation In practice, is sometimes running the instrumentation (i.e., only in previously unseen states) really more efficient than always doing it? Target: In Vivo Testing implementation in C Sieve of Eratosthenes program run with 100 inputs, using varying percentages of distinct states ranging from:  0%, i.e. all values are the same  100%, i.e. all values are different

18. 18 Chris Murphy, Columbia University Results

19. 19 Chris Murphy, Columbia University Limitations & Future Work Memory cost Upper bound of Cantor function Representation of complex objects Coordinating globally-unseen states

20. Automatic Detection of Previously-Unseen Application States for Deployment Environment Testing and Analysis Chris Murphy Columbia University cmurphy@cs.columbia.edu

21. 21 Chris Murphy, Columbia University Related Work Reducing overhead of runtime monitoring  Static analysis to remove unnecessary instrumentation [Yong & Horwitz, 2005]  Fast cases vs. slow cases [Liblit et al., 2003] State representation for anomaly detection  [Baah et al., 2006]  [Hangal & Lam, 2002]

22. 22 Chris Murphy, Columbia University