1. Application of Maximum Entropy Principle to software failure prediction Wu Ji Software Engineering Institute BeiHang University

2. Agenda Introduction Problem and focus Method and models Results Conclusions

3. Introduction Failure prediction is one of the key problems for software quality (reliability) estimation. Generally, failure prediction can be defined as y = f(x). – y is failure related variable – x is the foundation on which prediction works As far as we know, x has been set as: –Software execution time  reliability growth prediction –Software execution trace  anomaly detection

4. Introduction (cont.) Reliability has been a big concern for high reliability requirement (HRR) software. Reliability engineering has very high cost. Reliability testing is seldom done for the software without HRR. Anomaly detection is usually implemented as a built-in module of software.

5. Introduction (cont.) Generally, all managers are striving for high quality. What does manager really care for failure prediction? –Given an usage scenario, if software can survive? How to predict software failure from input is still a new problem.

6. Problem and focus How to predict failure from software input?

7. Problem and focus (cont.) … failure observation = ? (0/1) left context execution time line execution start s t

8. Problem and focus (cont.) If we can model the left context, we get the distribution {(lc, fo)}. Failure Learning Failure Prediction {(lc,fo)} Software input Failure observation Failure law

9. Method and models The whole left context is hard to model. –A probability model: p o (y|x) –x: partial left context, y: failure observation. Maximum Entropy Principle (MEP) is applied to model the p o (y|x).

10. Method and models (cont.) MEP is a well-known and widely used learning principle: –Great generalization ability –Dynamic and open –Good adaptive with data sparseness

11. Method and models (cont.) Failure cannot be well modeled without modeling fault. Failure can be well modeled only from input, and its relations with failures. Structure Viewer Surface Viewer Structure Model Surface Model

12. Method and models (cont.) Surface Model: learns the statistical co- occurrence of the surface information. Structure Model: learns the statistical cause-effect (fault-failure) relationship.

13. Method and models (cont.) SIU-Seg-Ftrs SIU-Num-Ftrs Failure-Ftrs Flr The features applied in the surface model

14. Method and models (cont.) Fault-Ftrs (Flt -> Flr) Ftrs Failure-Ftrs Flr The features applied in the structure model

15. Method and models (cont.) Supervised training Training data Objective: maximize the likelihood function.

16. Method and models (cont.) Models Evaluation: –For a given test case: Test engineer would run it and get the test_fo_sequence; The prediction model would return the predicted pred_fo_sequence. –Evaluate by the match degree (precision) between test_fo_sequence and pred_fo_sequence.

17. Results Two groups of experiments, totally 5 software involved in, 17 testing. Open test method –Testing data keeps separate with training data and keeps unknown for training. Surface Model: average precision: 0.876 Structure Model: average precision: 0.858

18. Results (cont.) Evaluation Score Distribution

19. Results (cont.)

21. Potential applications of the prediction model –Test case prioritization –Reliability Estimation –Reliability Growth Modeling

22. Conclusions A new failure prediction problem Apply statistical learning method to learn failure law and then predict failure Two models, surface model and structure model Promising evaluation results: –Surface Model: 0.876 –Structure Model: 0.858.

23. Conclusions (cont.) Lessons learnt: –To design and start experiments ASAP to verify model. –Complex model does not always perform well.  model simplification. –DO NOT draw much assumption on the generation of data.

24. Thank you for the attentions Ready for questions!