1. Ocasta: Clustering Configuration Settings for Error Recovery Zhen Huang, David Lie Department of Electrical and Computer Engineering University of Toronto

2. Outline Motivation Design Evaluation User Study Conclusion 2

3. Configuration errors Configuration errors are a leading cause of computer failures. Fixing a configuration error has two essential steps: – Identifying configuration settings that cause the error – Changing the faulty settings with values that fix the error 3

4. Configuration errors Two categories of configuration errors: – Single-setting configuration errors require changing one configuration setting to fix. – Multi-setting configuration errors require changing more than one configuration settings, related configuration settings, to fix. Example multi-setting configuration for Acrobat Reader 4 InlineAutoComplete AutoCompleteOnTab

5. Configuration errors Two categories of configuration errors: – Single-setting configuration errors require changing one configuration setting to fix. – Multi-setting configuration errors require changing more than one configuration settings, related configuration settings, to fix. Example multi-setting configuration for Acrobat Reader 5 InlineAutoComplete: false AutoCompleteOnTab : false

6. Configuration errors Two categories of configuration errors: – Single-setting configuration errors require changing one configuration setting to fix. – Multi-setting configuration errors require changing more than one configuration settings, related configuration settings, to fix. Example multi-setting configuration for Acrobat Reader 6 InlineAutoComplete: true AutoCompleteOnTab : false

7. Configuration errors Two categories of configuration errors: – Single-setting configuration errors require changing one configuration setting to fix. – Multi-setting configuration errors require changing more than one configuration settings, related configuration settings, to fix. Example multi-setting configuration for Acrobat Reader 7 InlineAutoComplete: true AutoCompleteOnTab : true

8. Multi-setting configuration errors A significant number of configuration errors (14.9% - 34.7%) are multi-setting configuration errors [1]. However, previous techniques [2,3] cannot fix multi-setting configuration errors. Ocasta is designed to fix both single-setting and multi-setting configuration errors. 1. “An empirical study on configuration errors in commercial and open source systems”, SOSP 2011 2. “Automated Diagnosis of Software Configuration Errors”, ICSE 2013 3. “Automating configuration troubleshooting with dynamic information flow analysis,” OSDI 2010 8

9. Ocasta Ocasta works in two modes: – Recording mode: record access to configuration settings – Recovery mode: fixing configuration errors with rollbacks access history of configuration settings RECORDING MODE Search for a fix Identify related configuration settings trial to execute RECOVERY MODE candidate fixes 9

10. Ocasta Ocasta works in two modes: – Recording mode: record access to configuration settings – Recovery mode: fixing configuration errors with rollbacks access history of configuration settings RECORDING MODE Search for a fix Identify related configuration settings trial to execute RECOVERY MODE candidate fixes 10

11. Ocasta Ocasta works in two modes: – Recording mode: record access to configuration settings – Recovery mode: fixing configuration errors with rollbacks access history of configuration settings RECORDING MODE Search for a fix Identify related configuration settings trial to execute RECOVERY MODE candidate fixes 11

12. Ocasta Ocasta works in two modes: – Recording mode: record access to configuration settings – Recovery mode: fixing configuration errors with rollbacks access history of configuration settings RECORDING MODE Search for a fix Identify related configuration settings trial to execute RECOVERY MODE candidate fixes 12

13. Ocasta Ocasta works in two modes: – Recording mode: record access to configuration settings – Recovery mode: fixing configuration errors with rollbacks access history of configuration settings RECORDING MODE Search for a fix Identify related configuration settings trial to execute RECOVERY MODE candidate fixes 13

14. Recording access to configuration settings Ocasta records application’s access to configuration settings into a database. – Provides input for identifying related configuration settings – Supplies historical values of configuration settings for rollback recovery Ocasta treats application as black-box and intercepts access to configuration settings. – Windows registry API calls – GConf API calls – File read and write 14

15. Identify related configuration settings We observe that related configuration settings are more likely to be modified together than unrelated settings. InlineAutoComplete AutoCompleteOnTab MaxMRUCnt UseCustomPaper 0500100015002000 Time (second) Example access history for Acrobat Reader 15

16. Identify related configuration settings We define a correlation metric between every pair of two settings to identify settings that are often modified together, but rarely modified individually. Ocasta uses hierarchical agglomerative clustering algorithm to cluster related configuration settings using correlation metric. 16 Modified Together Total Modifications

17. Automated Search Execute trial Take original screenshot Rollback a cluster of related settings Execute trial Take screenshot Select a cluster to rollback 17 Undo rollback

18. Automated Search 18 Execute trial Take original screenshot Rollback a cluster of related settings Execute trial Take screenshot Select a cluster to rollback Undo rollback

19. Automated Search 19 Execute trial Take original screenshot Rollback a cluster of related settings Execute trial Take screenshot Select a cluster to rollback Undo rollback

20. Automated Search 20 Execute trial Take original screenshot Rollback a cluster of related settings Execute trial Take screenshot Select a cluster to rollback Undo rollback Configuration Error

21. Automated Search 21 Execute trial Take original screenshot Rollback a cluster of related settings Execute trial Take screenshot Select a cluster to rollback Undo rollback

22. Automated Search 1. InlineAutoComplete AutoCompleteOnTab 2. MaxMRUCnt 3. UseCustomPaper 22 Execute trial Take original screenshot Rollback a cluster of related settings Execute trial Take screenshot Select a cluster to rollback Undo rollback

23. Automated Search InlineAutoComplete AutoCompleteOnTab 2013-09-03: false, false 2013-10-12: true, false 2013-12-02: false, false 2014-02-25: true, true 23 Execute trial Take original screenshot Rollback a cluster of related settings Execute trial Take screenshot Select a cluster to rollback Undo rollback

24. Automated Search 24 Execute trial Take original screenshot Rollback a cluster of related settings Execute trial Take screenshot Select a cluster to rollback Undo rollback

25. Automated Search 25 Execute trial Take original screenshot Rollback a cluster of related settings Execute trial Take screenshot Select a cluster to rollback Undo rollback

26. Automated Search 26 Execute trial Take original screenshot Rollback a cluster of related settings Execute trial Take screenshot Select a cluster to rollback Undo rollback

27. Automated Search 27 Execute trial Take original screenshot Rollback a cluster of related settings Execute trial Take screenshot Select a cluster to rollback Undo rollback 1. InlineAutoComplete AutoCompleteOnTab 2. MaxMRUCnt 3. UseCustomPaper

28. Recovery of configuration errors Users view screenshots to select which fix to apply. 28 ErrorFix

29. Evaluation We installed Ocasta on 24 Linux and 5 Windows desktop computers for several months. We evaluated the accuracy of Ocasta’s clustering algorithm, the effectiveness and performance of Ocasta’s error recovery for 11 Linux and Windows desktop applications. – mail client: MS OUTLOOK, Evolution – web browser: Internet Explorer, Chrome browser – document editor/viewer: MS Word, GNOME Edit, Acrobat reader – image editor/viewer: MS Paint, GNOME Image Viewer – media player: Windows Media Player – Windows shell: Explorer 29

30. Accuracy of clustering algorithm Ocasta identified 1,005 clusters for the 11 applications, involving 1,871 settings. Ocasta has a 88.6% accuracy of identifying multi-setting clusters. – We manually examined all 255 multi-setting clusters. 30

31. Configuration errors used in evaluation We use 16 real-world configuration errors from literature, online forums, FAQ and configuration document. – Complex real-world applications: MS Outlook, MS Word, Internet Explorer, Chrome, Acrobat reader – Variety of complex and simple errors: List of recently opened documents missing User is unable to save any document Menu bar missing for certain PDF documents 31

32. Performance of fixing configuration errors The configuration errors were simulated by being injected into the recorded access history, 14 days before the end of the history. Ocasta successfully fixed all 16 errors. – Without clustering, 5 of them cannot be fixed. – For 14 errors, user only needs to examine less than 5 screenshots. – All errors were fixed within 7 minutes. 32

33. User study We performed a user study on 19 participants with various backgrounds to evaluate Ocasta. Participants first used Ocasta to fix some configuration errors and then tried to manually fix them. To limit the length of user study, we used 4 simple configuration errors among the 16 configuration errors. 33

34. User study We use a scale of 1 to 5 indicating from very easy to very difficult. 74% of the participants rated creating trials as very easy. 80% of the participants rated selecting the correct screenshots as very easy. 34

35. Conclusion Ocasta accurately identifies 88.6% of clusters on average. Ocasta fixes all 16 real-world configuration errors, 5 of them are multi-setting configuration errors. Our user study shows that it is easy and efficient to use Ocasta to fix configuration errors. 35

36. Thank You! contact: z.huang@mail.utoronto.caz.huang@mail.utoronto.ca demo: http://youtu.be/aRvJlTj-0F0http://youtu.be/aRvJlTj-0F0 36