1 VLDB 2006, Seoul Mapping a Moving Landscape by Mining Mountains of Logs Automated Generation of a Dependency Model for HUG’s Clinical System Mirko Steinle, EPFL and HUG Karl Aberer, EPFL Sarunas Girdzijauskas, EPFL Alexander Lamb, HUG

2 VLDB 2006, Seoul Overview Background – Dependency Models Approaches –L1: Analyzing general service activity –L2: Analyzing user sessions –L3: Analyzing textual content Evaluation Conclusion

3 VLDB 2006, Seoul Background - A Moving Landscape Distributed clinical system of University Hospital Geneva (HUG) –2000 beds, 4500 PCs, records accessed per day Relevant features –Communication is web service based Service Directory: about 50 service groups –Centralized Logging System with a standard XML format 10 Mio log messages/day, 1 TeraByte/year –Quite homogeneous infrastructure Severe Availability Requirements (24 x 7 x 365) ➱ Need for automated support for problem diagnosis

4 VLDB 2006, Seoul Dependency Model Service Orientation allows for easy reuse and integration, but has resulted into a complex dependency structure Dependency model is not clear –DM difficult to obtain, impossible to keep up-to-date manually –Infrastructure for manual documentation of the dependency structure is available, but not used …

5 VLDB 2006, Seoul Goal - Automated Dependency Model Goal: Automated creation of a model of the system’s dependency structure (DM) –Non-intrusive and low-cost –Focus on invocation dependencies between high-level objects Applications –Support for Fault Localization Algorithms –Prediction of Impact of Management Operations –Support for Architectural Decisions –Detection of Abnormal Behavior “you don’t want to interrupt a surgery because of DB maintenance”

6 VLDB 2006, Seoul Possible Approaches Static approaches –Capture dependencies at “compile time” by scanning configuration files, code etc. Dynamic approaches –Capture dependencies at runtime –Approaches include: Code instrumentation (standards like JMX or ARM exist but are not yet applied broadly) Middleware instrumentation (eg. request tagging) Active perturbation of system operation Time series analysis of activity measures, eg. using Neural Networks, (network communication, cpu usage, …) [Ensel02] Generality Accuracy & Precision

7 VLDB 2006, Seoul State of the Art Research –Focuses on how to exploit a dependency model, little work on how to obtain it –No generally applicable solution providing sufficiently correct dependency models seems to exist Commercial Products –Most focus on low-level objects and visualization –(Few) existing dynamic approaches: high configuration effort!

8 VLDB 2006, Seoul Overview Background – Dependency Models Approaches –L1: Analyzing general service activity –L2: Analyzing user sessions –L3: Analyzing textual content Evaluation Conclusion

9 VLDB 2006, Seoul Technique L1: Logs as a General Activity Measure Key idea –Activity of dependent objects is likely to be correlated in some sense –Use logs as an activity measure Earlier work –Neural networks on CPU usage, traffic volume, … [Ensel02] –Drawback: supervised training Our approach –statistical approach (no training) –inspired by [LM04] (“Mining Temporal Patterns without Predefined Time Windows”)

10 VLDB 2006, Seoul Statistical Approach Tests for association of spatial point processes –Compare the typical distance of a random point R in time to the closest timestamp of a log from B, to the one of a timestamp of a log from A Approach –Obtain distances by sampling from R and A –Determine median for distances A-B and R-B –If median for A-B lower than for R-B → correlation/dependence –Use confidence intervals

11 VLDB 2006, Seoul Example confidence interval for median of x 1,…,x n : median falls with probability 95% into this interval, interval [x j, x k ] s.t. B n,½ (k-1)- B n,½ (j-1) > 0.95

12 VLDB 2006, Seoul Observations for L1 Observations from preliminary experimental evaluation –True dependencies found, but clearly incomplete –Few “random” errors –However, correlation also if no invocation dependency exists Limit analysis to shorter time windows –Eliminate common dependency on time Transitive dependencySimultaneous use

13 VLDB 2006, Seoul Technique L2: Logs in a User Session One main difficulty is heavy parallelism in system ➱ execution sequences get overshadowed Reconstruct user sessions ➱ eliminates parallelism due to multiple users Then, adapt a procedure from NLP [Evert04] Two independent steps 1.Extraction of consecutive log-source pairs [APP i, APP j ] and creation of contingency tables 2.Statistical test for association on these tables

14 VLDB 2006, Seoul Construction of Contingency Table Session Log Bigrams (u, v) Contingency table for A-B u = Au ≠ A v = B 11 v ≠ B 01 (A,B) - (B,C) - (C,B)

15 VLDB 2006, Seoul Expected vs. Observed Frequencies Expected frequencies under the hypothesis that u and v are statistically independent

16 VLDB 2006, Seoul Statistical Test for Association Log-likelihood test (Dunning) Works well for heavily skewed tables (O 11 << N) For an excellent discussion of statistical tests for correlation see [Evert04]

17 VLDB 2006, Seoul Observations for L2 Observations from preliminary experimental evaluation –Many true dependencies found –Interestingly, a few similar errors as in L1 transitivity and simultaneous use –Main problem only a small subset of logs can be assigned to a session, and many interactions can thus not be observed

18 VLDB 2006, Seoul Technique L3: Exploiting Textual Content in Logs Observation –Invocation of a remote service is typically logged by the caller –One could identify such logs and process log content to find callee The other way round –Find logs mentioning directory entry contents for a given service –Infer a dependency of the log’s source, the caller, on the service Example: service s calls notify on server myserver ●Possible content of free text in log entry Invoke externalService [fct [notify] server [myserver.hguge:9999/myurl]] or (DPINOTIFICATION) notify ($myparams)

19 VLDB 2006, Seoul Overview Background – Dependency Models Approaches –L1: Analyzing general service activity –L2: Analyzing user sessions –L3: Analyzing textual content Evaluation Conclusion

20 VLDB 2006, Seoul Experiments on Logs: Setting Test data: 56.8 Mio logs from 1 week Reference model (RM) –Created with help of more than a dozen system experts and developers –178 dependencies out of 1431 possible dependencies (54 services) Strategy 1.Validate L1, L2 and L3 against static reference model 2.Validate L1 and L2 against L3 and study influence of load

21 VLDB 2006, Seoul Experiment: Validation against RM L level CI: [0.63, 0.73] L level CI: [0.71, 0.78] L level CI: [0.93, 0.96] True Positives detected Small classification error for L1 –about 2% in negative case False Positives (FP) for L1 –transitive and simultaneous use (e.g. administrative patient data and laboratory results) True Positives detected FP for L2 –asynchronous communication Sessions in L2 –only 10% of all logs can be assigned to a session True Positives detected –10 False Negatives on the whole week

22 VLDB 2006, Seoul Experiment: Influence of Load on Detection Realizations of dependency relationships computed with L3 Percentage of False Positives is not influenced by load CI for linear factors L1: [-0.284, ] L2: [-0.025, 0.002]

23 VLDB 2006, Seoul Overview Background – Dependency Models Approaches –L1: Analyzing general service activity –L2: Analyzing user sessions –L3: Analyzing textual content Evaluation Conclusion

24 VLDB 2006, Seoul Comparison of Log-based Approaches L3. Logs as TextL2. Logs in SessionsL1. Logs as Activity Measure Accuracy and Precision of Result  Concurrency  Correlation Implementation and Maintenance  Parametrization Performance and security impact  Required Structure and Content of Logs (Scope)  Service directory  Session info  Only source and timestamp All techniques can be implemented in linear complexity w.r.t. #logs Invocation direction  functional dependency direction Solution for HUG –Centralized logging system ➱ little effort for log-based methods –L3 is a viable solution

25 VLDB 2006, Seoul Conclusion Three new approaches to use logs for DM generation with a large scope All have been shown to discover useful dependency information in real-world environment Seems to be first study on use of logs and first real- world experiment for DM generation Sniffing –Applicable for web service oriented systems Simple and efficient solution for HUG