1. Mincer: a distributed automated problem determination tool
A. Craik, C Black, P Doyle 4-Nov-14
Mincer: a distributed automated problem determination tool

2. Debugging Optimizing Compilers
Nearly every optimizing transformation is optional
Adopt the scientific method – run experiments varying compilation control parameters to find failure
Problems
Time consuming & tedious to configure & run
Resource constraints limit number of experiments
Experience needed to choose the best experiments
Environment may cause problem to occur intermittently
Resources wasted repeatedly running the same experiments

3. Objectives Tool to run tests in a distributed machine farm
Distributed experiment selection and execution logic for scalability
Configurability & extensibility to support new transformations and new debugging techniques
Reduce resource waste – gain knowledge from every experiment run
Tolerate false-positive & false-negative results and handle variable failure rates

4. Problem Conceptualization
A problem determination tactic aims to isolate a problem by varying an experimental parameter and observing the effect
Problem has two discrete components:
Deciding which problem determination tactics to use
Applying chosen problem determination tactics to failure
Each experimental parameter can be mapped back to an integer search space
Mathematically model the selection of experiments and the results they produce

5. Tactic Execution Engine
Mincer Architecture
mincer.pl
Data
Layer
Sequential
Subset
Results DB
Solvers
GatherLogs
OptLevel
LastOptIndex
LimitFile
LastOptSub Index
Tactics
Tactic Execution Engine

6. Experiment Selection Logic

7. The Numerical Solvers
Solvers use Bayesian Inference to update the underlying hyperparameters of the experiments
Solvers use Information Theory to make optimal test parameter choices
Solve to a user-defined confidence level (-c option)
Currently using 2 solvers: sequential (contiguous subranges from 1 to some n) subset (unordered subsets)
Solvers also track failure rate to guide experiment selection

8. Sequential Solver

9. Sequential Solver

10. Sequential Solver

11. Sequential Solver

12. Sequential Solver

13. Sequential Solver

14. Sequential Solver

15. Sequential Solver

16. Sequential Solver

17. Current Status Mincer being used to debug problems in production
We have implemented tactics for several different experimental parameters on our Java JIT
Working on Infinite Sequential Solver to allow us to isolate intermittent failures to specific changesets

18. Q & A

19. Sequential Solver Theory Have a finite, ordered set, T
Random Variable P, support on T
Random Variable f, support on [0, 1]
Random Variables {Xi}, binary

20. Sequential Solver Theory Each Xi is associated with an index ni in T
Pr(Xi passing | f, P) = (1-f) if ni ≥ P
Pr(Xi passing | f, P) = 1 if ni < P
Choose index to associate with next Xi optimally

21. Subset Solver Theory Each Xi is associated with a subset Ni of T
Pr(Xi passing | f, P) = (1-f) if P ϵ Ni
Pr(Xi passing | f, P) = 1 otherwise
Choose next subset optimally
Faster convergence than possible with Sequential Solver