1. EGEE-III INFSO-RI-222667 Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks Design of an Expert System for Enhancing Grid Fault Detection based on Grid Monitoring Data Gerhild Maier March 2 nd 2008

2. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Mining Job Monitoring Data Gerhild Maier 2 Outline  problem description  approach association rule mining design of an expert system  current status, example  outlook and summary

3. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Mining Job Monitoring Data Gerhild Maier 3 Problem Description  Dashboard database: a lot of information about jobs  Dashboard monitoring tools: find faulty Grid components  exit codes  detect error source underneath the exit codes  fast, to solve problems quickly  automatization

4. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Mining Job Monitoring Data Gerhild Maier4 Approach  combine machine created knowledge with human knowledge to an expert system  Association Rule Mining on Monitoring Data (machine created knowledge)  Human Knowledge (the rule interpretation)‏  Expert System

5. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Mining Job Monitoring Data Gerhild Maier 5 Association Rule Mining (1/2)‏  rule: set of items  item: attribute-value pair  support: s% of the data includes all items  confidence: c% of the data including the antecedent also include the consequent  lift: measurement of interestingness lift = support(AB)/(support(A)*support(B))‏ RULE {ERROR=8001}{user=AB, ce=red.unl.edu} (0.367/100.000/11.330)‏ QUALITY (s%/c%/lift)‏ consequent‏ antecedent‏

6. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Mining Job Monitoring Data Gerhild Maier 6 Association Rule Mining (2/2)‏ Find frequent item set Create association rules Remove not interesting rules … rule 1 rule 2... rule n rule 1 rule 2 … rule k item set 1 item set 2 item set n Apriori Algorithm Pruning the rules Set of association rules Job Monitoring Information of the Dashboard Database

7. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Mining Job Monitoring Data Gerhild Maier 7 Expert System (1/2)‏  a program solving problems like an expert  example: decision support system to detect a problem... Did you plug in the printer? → yes Did you install a driver? → no...  2 components: 1.knowledge base: collection of human expert knowledge in a problem domain 2.inference engine: defines how to use the knowledge

8. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Expert System (2/2)‏ Mining Job Monitoring Data Gerhild Maier 8 using the ES maintaining the ES building the ES

9. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Mining Job Monitoring Data Gerhild Maier 9 QAOES - Input  QAOES = Quick Analysis Of Error Sources  time range: last 12 hours, last 24 hours  support: –minimum number of jobs –low number → many rules → long runtime  confidence: –significance of the rule –high percentage → good rules  background: 8 job attributes site, ce, queue, worker node dataset user, application exit code

10. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Mining Job Monitoring Data Gerhild Maier 10 QAOES - Output  list of rules, with quality measures: support, confidence, lift  association rules: interesting dependencies of job attributes unusual patterns in the dataset  link to dashboard job summary page  example: CMS analysis jobs from 12 hours: 30761 min 100 jobs => support = 0.26 % confidence: 90% runtime: 5 min number of rules: 7

11. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Mining Job Monitoring Data Gerhild Maier 11 QAOES - Output

12. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Mining Job Monitoring Data Gerhild Maier 12 Output Verification (1/2)‏ one user has problems on different sites

13. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Mining Job Monitoring Data Gerhild Maier 13 Output Verification (2/2)‏ one user has problems with different datasets

14. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667  user has problems on different site, with different datasets → problem in his code?  exit code 60xxx → stage out problem → problem with the storage element?  …  collection of rule interpretations  rule generalization  input for the knowledge base Mining Job Monitoring Data Gerhild Maier 14 Output Interpretation

15. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Mining Job Monitoring Data Gerhild Maier 15 Outlook  continuous adaptation of the association rule mining parameters  building the knowledge base  development of the inference engine

16. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Mining Job Monitoring Data Gerhild Maier 16 Summary  building the Expert System  Association Rule mining completed  collecting Human Knowledge  web interface currently deployed for analysing CMS analysis jobs  QAOES easy to adapt to different VOs job data

17. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Mining Job Monitoring Data Gerhild Maier 17 Links and References  QAOES: http://dashb-cms-mining-devel.cern.ch/dashboard/request.py/rules  Twiki: https://twiki.cern.ch/twiki//bin/view/ArdaGrid/AutomaticFaultDetection  Association Rule Mining: article: Mining Association Rules between Sets of Items in Large Databases, Agrawal R, Imielinski T, Swami AN.  Pruning Association Rules: article: Efficient Statistical Pruning of Association Rules, Alan Ableson, Janice Glasgow  Expert Systems: book: Introduction to Expert Systems, Peter Jackson