1. © 2013 IBM Corporation A. Craik, C Black, P Doyle 4-Nov-14 Mincer: a distributed automated problem determination tool

2. © 2013 IBM Corporation Debugging Optimizing Compilers  Nearly every optimizing transformation is optional  Adopt the scientific method – run experiments varying compilation control parameters to find failure  Problems  Time consuming & tedious to configure & run  Resource constraints limit number of experiments  Experience needed to choose the best experiments  Environment may cause problem to occur intermittently  Resources wasted repeatedly running the same experiments

3. © 2013 IBM Corporation Objectives  Tool to run tests in a distributed machine farm  Distributed experiment selection and execution logic for scalability  Configurability & extensibility to support new transformations and new debugging techniques  Reduce resource waste – gain knowledge from every experiment run  Tolerate false-positive & false-negative results and handle variable failure rates

4. © 2013 IBM Corporation Problem Conceptualization  A problem determination tactic aims to isolate a problem by varying an experimental parameter and observing the effect  Problem has two discrete components: – Deciding which problem determination tactics to use – Applying chosen problem determination tactics to failure  Each experimental parameter can be mapped back to an integer search space  Mathematically model the selection of experiments and the results they produce

5. © 2013 IBM Corporation Mincer Architecture mincer.pl Data Layer SequentialSubset LastOptIndexOptLevel Tactic Execution Engine GatherLogs Solvers Tactics Results DB LastOptSub Index LimitFile

6. © 2013 IBM Corporation Experiment Selection Logic

7. © 2013 IBM Corporation The Numerical Solvers Solvers use Bayesian Inference to update the underlying hyperparameters of the experiments Solvers use Information Theory to make optimal test parameter choices Solve to a user-defined confidence level (-c option) Currently using 2 solvers: - sequential (contiguous subranges from 1 to some n) - subset (unordered subsets) Solvers also track failure rate to guide experiment selection

8. © 2013 IBM Corporation Sequential Solver

9. © 2013 IBM Corporation Sequential Solver

10. © 2013 IBM Corporation Sequential Solver

11. © 2013 IBM Corporation Sequential Solver

12. © 2013 IBM Corporation Sequential Solver

13. © 2013 IBM Corporation Sequential Solver

14. © 2013 IBM Corporation Sequential Solver

15. © 2013 IBM Corporation Sequential Solver

16. © 2013 IBM Corporation Sequential Solver

17. © 2013 IBM Corporation Current Status  Mincer being used to debug problems in production  We have implemented tactics for several different experimental parameters on our Java JIT  Working on Infinite Sequential Solver to allow us to isolate intermittent failures to specific changesets

18. © 2013 IBM Corporation Q & A

19. © 2013 IBM Corporation Sequential Solver Theory Have a finite, ordered set, T Random Variable P, support on T Random Variable f, support on [0, 1] Random Variables {X i }, binary

20. © 2013 IBM Corporation Sequential Solver Theory Each X i is associated with an index n i in T Pr(X i passing | f, P) = (1-f) if n i ≥ P Pr(X i passing | f, P) = 1 if n i < P Choose index to associate with next X i optimally

21. © 2013 IBM Corporation Subset Solver Theory Each X i is associated with a subset N i of T Pr(X i passing | f, P) = (1-f) if P N i Pr(X i passing | f, P) = 1 otherwise Choose next subset optimally Faster convergence than possible with Sequential Solver