Hydra-MIP: Automated Algorithm Configuration and Selection for Mixed Integer Programming Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown Department of Computer Science University of British Columbia

2 Solving MIP more effectively Portfolio-based algorithm selection (SATzilla) [Xu et al., 2007;2008;2009] Where are the solvers? Parameter settings of a single solver (e.g. CPLEX) How to find good settings? Automated algorithm configuration tool [Hutter et al., 2007;2009] How to find good candidates for algorithm selection? Algorithm configuration with dynamic performance metric [Xu et al., 2010] Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

Some particularly related work: [Gratch & Dejong, 1992]; [Balaprakash, Birattari & Stuetzle, 2007]; [Hutter, Babic, Hoos & Hu, 2007]; [Hutter, Hoos, Stuetzle & Leyton-Brown, 2009] Some particularly related work: [Rice, 1976]; [Leyton-Brown, Nudelman & Shoham, 2003; 2009]; [Guerri & Milano, 2004]; [Nudelman, Leyton-Brown, Shoham & Hoos, 2004] 3 Hydra Portfolio-based algorithm selection: Automated algorithm configuration: Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

Outline Improve algorithm selection – SATzilla – Drawback of SATzilla – New SATzilla with cost sensitive classification – Results Reduce the construction cost – Hydra – The cost – Make full use of configuration – Results Conclusion Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP 4

5 Given: – training set of instances – performance metric – candidate solvers – portfolio builder (incl. instance features) Training: – collect performance data – portfolio builder learns predictive models At Runtime: – predict performance – select solver Metric Portfolio Builder Training Set Novel Instance Portfolio-Based Algorithm Selector Candidate Solvers Selected Solver SATzilla: Portfolio-Based Algorithm Selection [Xu, Hutter, Hoos, Leyton-Brown, 2007; 2008] Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP 5

6 Drawback of SATzilla Algorithm selection in SATzilla based on regression: – Predict each solver performance independently – Select best predicted solver – Classification based on regression Goal of regression: Accurately predict each solvers performance Algorithm selection: Pick solvers on a per-instance basis in order to minimize some overall performance metric Better regression Better algorithm selection Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP Algorithm Selector

7 Cost sensitive classification for SATzilla Loss function: the performance difference – Punish misclassifications in direct proportion to their impact on portfolio performance – No need for predicting runtime Implementation: Binary cost sensitive classifier: decision forest (DF) – Build DF for each pair of candidate solvers – one vote for the better solver – Most votes -> Best solver Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

8 SATzilla DF performance DataSetModelAverage TimeSolved PercentageTime speedup RAND LR % HAND LR % INDU LR % LR: linear regression as used in previous SATzilla; DF: cost sensitive decision forest Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

SATzilla DF performance DataSetModelAverage TimeSolved PercentageTime speedup RAND LR % 1.08× DF % HAND LR % 1.16× DF % INDU LR % 1.12× DF % LR: linear regression as used in previous SATzilla; DF: cost sensitive decision forest 9 Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

10 MIPzilla DF performance DataSetModelAverage TimeSolved Percentage Time speedup LR % LR % ISAC (new) LR % MIX LR5699.6% Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

11 MIPzilla DF performance DataSetModelAverage TimeSolved Percentage Time speedup LR % 1.00× DF % LR % 1.04× DF % ISAC (new) LR % 1.18× DF % MIX LR5699.6% 1.05× DF4899.6% Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

12 Hydra Procedure: Iteration 1 Algorithm Configurator Metric Training Set Portfolio-Based Algorithm Selector Candidate Solver Set Candidate Solver Parameterized Algorithm Portfolio Builder Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

13 Hydra Procedure: Iteration 2 Algorithm Configurator Metric Training Set Portfolio-Based Algorithm Selector Candidate Solver Set Candidate Solver Parameterized Algorithm Portfolio Builder Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

14 Hydra Procedure: Iteration 3 Algorithm Configurator Metric Training Set Portfolio-Based Algorithm Selector Candidate Solver Set Candidate Solver Parameterized Algorithm Portfolio Builder Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

15 Output: Hydra Procedure: After Termination Portfolio-Based Algorithm Selector Novel Instance Selected Solver Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

16 We are wasting configuration results! Algorithm Configurator Metric Training Set Candidate Solver Parameterized Algorithm Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

17 Make full use of configurations Algorithm Configurator Metric Training Set Parameterized Algorithm Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP k Candidate Solvers

18 Make full use of configurations Advantage: – Add k solvers instead of 1 in each iteration (good for algorithm selection) – No need for validation step in configuration (SAVE time) Disadvantage: – Need to collect runtime data for more solvers (COST time) In our experiment, we found SAVE = COST (k=4) Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

19 Experimental Setup: Hydras Inputs Portfolio Builder: MIPzilla LR (SATzilla for MIP) [Xu et al., 2008] MIPzilla DF (MIPzilla using cost sensitive DF) Parameterized Solver: CPLEX12.1 Algorithm Configurator: FocusedILS [Hutter, Hoos, Leyton-Brown, 2009] Performance Metric: Penalized average runtime (PAR) Instance Sets: 4 heterogeneous sets by combining homogeneous subsets [Hutter et al., 2010];[Kadioglu et al., 2010]; [Ahmadizadeh et al., 2010] Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

20 Three versions of Hydra for MIP Hydra LR,1 : Original Hydra for MIP [Xu et al., 2010] Hydra DF,1 : Hydra for MIP with Improvement I Hydra DF,4 : Hydra for MIP with Improvement I and II Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

MIP-Hydra performance on MIX Hydra DF,* performs better than Hydra LR,1 Hydra DF,4 performs similar to Hydra DF,1, but converge faster Performance close to Oracle and MIPzilla DF 21 Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP

22 Conclusion Cost sensitive classification based SATzilla outperforms original SATzilla New Hydra-MIP outperforms CPLEX default, algorithm configuration alone, and original Hydra on four heterogeneous MIP sets Technical contributions: – Cost sensitive classification results better algorithm selection for SAT and MIP – Using multiple configurations speeds up the convergence of Hydra Xu, Hutter, Hoos, and Leyton-Brown: Hydra-MIP