1. CISC 879 - Machine Learning for Solving Systems Problems Presented by: Suman Chander B Dept of Computer & Information Sciences University of Delaware Automatic Software Fault Diagnosis by Exploiting Application Signatures

2. CISC 879 - Machine Learning for Solving Systems Problems Motivation Application problem diagnosis in complex Enterprise environments large number of possible causes, most of the failures due to runtime interactions with the system environment Troubleshooting these problems requires extensive experience and time

3. CISC 879 - Machine Learning for Solving Systems Problems Overview Present a black box approach to diagnose several application faults Application signatures Approach to detect application faults Provide detail on tool design and implementation Evaluate effectiveness of the tool to correct fault behaviour of an application Case studies to support the ideology

4. CISC 879 - Machine Learning for Solving Systems Problems Application Behaviour Factors aiding in capturing application behaviour: System Calls Signals Environment variables Resource Limits Access Control Collecting and keeping history information help in finding the root cause of problem in quick time.

5. CISC 879 - Machine Learning for Solving Systems Problems Building Signatures... Choice of attributes – using “test of goodness test” using KS-test

6. CISC 879 - Machine Learning for Solving Systems Problems Signatures for system calls...

7. CISC 879 - Machine Learning for Solving Systems Problems handling multiple processes... Data is collected for each process separately Relations between systems calls will be correctly reflected after separating interleaving system calls Some specific attributes (eg. Signals, UID) are specific to a process For multithreaded applications – data collection and signatures are built separately for each thread Current approach does not handle user-level threads

8. CISC 879 - Machine Learning for Solving Systems Problems Tool Design System Architecture

9. CISC 879 - Machine Learning for Solving Systems Problems One...Application Tracer... Tracer tool force executes target application e.g. ‘tracer application_program’ Low overheads is crucial Uses p-trace interface for building signatures for system calls Some runtime behaviours (environment variables, resource limit, user id, etc) are not relevant to system calls

10. CISC 879 - Machine Learning for Solving Systems Problems Two...Signature Bank...

11. CISC 879 - Machine Learning for Solving Systems Problems Three...Fault Diagnosis... Classifier tool provides root cause for deviation from normal behaviour: Access the signature bank for normal traces Compare with faulty trace obtained Determine the root cause for this fault Provide information to user with diagnosis

12. CISC 879 - Machine Learning for Solving Systems Problems Case Studies

13. CISC 879 - Machine Learning for Solving Systems Problems Testing with Apache... For testing the tool with Apache, WebStone 2.5 is used WebStone 2.5 is free benchmarking tool for web servers Signature bank was built from performing operations ten times each to generate corresponding traces Example: Faulty execution of write system call Unable to write into log file Root Cause: Error Number EFBIG indicating that file is too large

14. CISC 879 - Machine Learning for Solving Systems Problems Testing with Apache...

15. CISC 879 - Machine Learning for Solving Systems Problems Observation 1 Comparison showing change in size of trace over 45 minute period 6.3 MB space contains recording of nearly 11 million system call invocations

16. CISC 879 - Machine Learning for Solving Systems Problems Observation 2 Comparison of change in size of trace file and signature bank based on the number of traces run Signature bank grows slow as redundant data are merged

17. CISC 879 - Machine Learning for Solving Systems Problems From other tests... CVS Average slowdown – 29.6% Collected 26 traces ranging from 0.1 MB to 1.6 MB Recorded signature bank is 6.5 MB consisting of about 1.8 million system calls PostgreSQL Average slowdown – 15.7% Collected traces ranging from 0.6 MB to 2.1 MB Recorded signature bank is 3.2 MB

18. CISC 879 - Machine Learning for Solving Systems Problems Limiting false positives First cause is related to KS-test Second cause relates to the fact that Signature bank cannot cover all normal variations of the attributes Aggregating more traces would complete the bank and reduce false positives gradually

19. CISC 879 - Machine Learning for Solving Systems Problems Performance measure Majority is due to information collection and trace file updating when a system call happens Overheads that occur: Switching from kernel to tracer and back both at system call entry and exit Retrieving system call number, return value and related attributes Looking up user stack to get its content Improvement obtained by modifying ptrace code with addition of primitives PTRACE_SETBATCHSIZE and PTRACE_READBUFFER

20. CISC 879 - Machine Learning for Solving Systems Problems Improvement...

21. CISC 879 - Machine Learning for Solving Systems Problems Limitations Labelling of application execution trace as faulty Manual indication required Conservative approach in capturing amount of information needed for trace More analysis required to identify minimum required set of data that will provide higher accuracy in detecting problems Results are limited from exploring few case studies

22. CISC 879 - Machine Learning for Solving Systems Problems THANK YOU

23. CISC 879 - Machine Learning for Solving Systems Problems Slide Title First Bullet