Mincer: a distributed automated problem determination tool A. Craik, C Black, P Doyle 4-Nov-14 Mincer: a distributed automated problem determination tool
Debugging Optimizing Compilers Nearly every optimizing transformation is optional Adopt the scientific method – run experiments varying compilation control parameters to find failure Problems Time consuming & tedious to configure & run Resource constraints limit number of experiments Experience needed to choose the best experiments Environment may cause problem to occur intermittently Resources wasted repeatedly running the same experiments
Objectives Tool to run tests in a distributed machine farm Distributed experiment selection and execution logic for scalability Configurability & extensibility to support new transformations and new debugging techniques Reduce resource waste – gain knowledge from every experiment run Tolerate false-positive & false-negative results and handle variable failure rates
Problem Conceptualization A problem determination tactic aims to isolate a problem by varying an experimental parameter and observing the effect Problem has two discrete components: Deciding which problem determination tactics to use Applying chosen problem determination tactics to failure Each experimental parameter can be mapped back to an integer search space Mathematically model the selection of experiments and the results they produce
Tactic Execution Engine Mincer Architecture mincer.pl Data Layer Sequential Subset Results DB Solvers GatherLogs OptLevel LastOptIndex LimitFile LastOptSub Index Tactics Tactic Execution Engine
Experiment Selection Logic
The Numerical Solvers Solvers use Bayesian Inference to update the underlying hyperparameters of the experiments Solvers use Information Theory to make optimal test parameter choices Solve to a user-defined confidence level (-c option) Currently using 2 solvers: - sequential (contiguous subranges from 1 to some n) - subset (unordered subsets) Solvers also track failure rate to guide experiment selection
Sequential Solver
Sequential Solver
Sequential Solver
Sequential Solver
Sequential Solver
Sequential Solver
Sequential Solver
Sequential Solver
Sequential Solver
Current Status Mincer being used to debug problems in production We have implemented tactics for several different experimental parameters on our Java JIT Working on Infinite Sequential Solver to allow us to isolate intermittent failures to specific changesets
Q & A
Sequential Solver Theory Have a finite, ordered set, T Random Variable P, support on T Random Variable f, support on [0, 1] Random Variables {Xi}, binary
Sequential Solver Theory Each Xi is associated with an index ni in T Pr(Xi passing | f, P) = (1-f) if ni ≥ P Pr(Xi passing | f, P) = 1 if ni < P Choose index to associate with next Xi optimally
Subset Solver Theory Each Xi is associated with a subset Ni of T Pr(Xi passing | f, P) = (1-f) if P ϵ Ni Pr(Xi passing | f, P) = 1 otherwise Choose next subset optimally Faster convergence than possible with Sequential Solver