1. SATzilla: Portfolio-based Algorithm Selection for SAT Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown Department of Computer Science University of British Columbia Mainly based on [Xu, Hutter, Hoos, Leyton-Brown, JAIR, 2008]

2. SATzilla Excellent performance in SAT competitions [http://www.satcompetition.org/] – 2 second places, one third place in 2003 SAT competition [Leyton-Brown, Nudelman, Andrew, McFadden & Shoham, 2003] – 3 Gold, 1 Silver, 1 Bronze in 2007 SAT competition [Xu, Hutter, Hoos, & Leyton-Brown, 2007] – 3 Gold, 2 Silver in 2009 SAT competition [Xu, Hutter, Hoos & Leyton-Brown, 2009] One gold in every category: Industrial/Handmade/Random Domain independent – MIPzilla, CSP, Planning … Xu, Hutter, Hoos, and Leyton-Brown: SATzilla2 Solvers Instance Feature extractor Algorithm selector

3. Impact on the SAT community Encouraged research on portfolio-based solvers for SAT Final 2011 SAT competition results [http://www.satcompetition.org/] Xu, Hutter, Hoos, and Leyton-Brown: SATzilla CategoryRandomHandmadeIndustrial Medalists 3S ppfolio (2) ppfolio seq sparrow2011 sattime2011 eagleup march_rw mphasesat_m 3S (3) Ppfolio (2) ppfolio seq (2) clasp glucose glucose (2) glueminisat (2) lingeling contrasat hack cirminisat hack mphasesat64 qutersat Total medals4/97/90/9 3

4. Outline Introduction Related Work: – Algorithm selection SATzilla overview: – Building SATzilla – Running SATzilla SATzilla07 and beyond Results SATzilla since 2008 Conclusion Xu, Hutter, Hoos, and Leyton-Brown: SATzilla4

5. Related work: algorithm selection Select the best solver for each instance [Rice, 76] – Regression-based Linear regression (SATzilla) [Leyton-Brown, Nudelman, Andrew, McFadden & Shoham, 2003; Xu, Hutter, Hoos, Leyton-Brown, 2007+] Matchbox [Stern et al., 2010] – Classification-based Logistic regression [Samulowitz & Memisevic, 2007] K-NN [OMahony & Hebrard, 2008] Decision tree [Guerri & Milano, 2004] Xu, Hutter, Hoos, and Leyton-Brown: SATzilla5

6. What is SATzilla? Portfolio-based algorithm selection based on empirical hardness models [Leyton-Brown et al., 2002;2009] – Use regression models to predict each algorithms performance – Select the predicted best Many new techniques for improving robustness – Pre-solvers – Backup solver – Solver subset selection Xu, Hutter, Hoos, and Leyton-Brown: SATzilla6

7. SATzilla: prerequisites Identify a target distribution of instances Select a set of candidate solvers, and measure runtime on a training set Identify a set of features based on domain knowledge and collect feature values – Cheap to compute – Informative for prediction Xu, Hutter, Hoos, and Leyton-Brown: SATzilla7

8. SATzilla core: predictive models Goal: accurately predict solver performance based on cheaply computable features Supervised machine learning: learn function f: features performance – Input: pairs – Output: f Xu, Hutter, Hoos, and Leyton-Brown: SATzilla f w () = w T Performance(y) Features () 23.34 7.21 … Algorithm selector 8

9. SATzilla: selection based on models Predict each solvers performance Pick solver based on predictions Feature extractor Algorithm selector Instance Compute features Make prediction Run on selected solver Xu, Hutter, Hoos, and Leyton-Brown: SATzilla9

10. Improving robustness Solver subset selection – More candidate solvers can hurt – select subset of solvers that achieves best overall performance Pre-solvers – Solve some instances quickly without need for feature computation – Increase model robustness Backup solver – In case something goes wrong Feature extractor Algorithm selector Dealing with imperfect predictions: Xu, Hutter, Hoos, and Leyton-Brown: SATzilla Algorithm selector 10

11. Time to run SATzilla! Xu, Hutter, Hoos, and Leyton-Brown: SATzilla11

12. Step 1: pre-solving Run pre-solvers for a fixed short amount of time – If solved, done Instance Feature extractor Algorithm selector Xu, Hutter, Hoos, and Leyton-Brown: SATzilla12

13. Step 2: compute features Run feature extractor to get a set of feature values – If solved, done – If anything fails, simply run backup solver Instance Feature extractor Algorithm selector Xu, Hutter, Hoos, and Leyton-Brown: SATzilla13

14. Step 3: performance prediction Predict performance of solvers and order them according to runtime predictions Feature extractor Algorithm selector Xu, Hutter, Hoos, and Leyton-Brown: SATzilla14

15. Step 4: run top-ranked solver Run the top-ranked solver. If a run fails, run the next-ranked solver. Xu, Hutter, Hoos, and Leyton-Brown: SATzilla15 Feature extractor Algorithm selector Instance

16. SATzilla07: construction Pre-solvers: manually selected Backup solver: overall best Model building: – Predict runtime – Hierarchical hardness models [Xu, Hoos, Leyton-Brown, 2007] – Models handle censored data [Schmee & Hahn, 1979] Solver subset selection: exhaustive search Xu, Hutter, Hoos, and Leyton-Brown: SATzilla16

17. Improving SATzilla07 (1.) Problem: manual pre-solver selection is not ideal Solution: auto pre-solver selection based on validation Xu, Hutter, Hoos, and Leyton-Brown: SATzilla17 Feature extractor Algorithm selector

18. Improving SATzilla07 (2.-5.) Problem: overall best solver best backup solver Solution: choose the best solver for instances not solved by pre-solvers and with problematic features Problem: objective is not always runtime Solution: predict and optimize true objective directly also enables cleaner use of incomplete solvers Problem: exhaustive search for solver subset selection is infeasible for large portfolios Solution: local search for solver subset selection Xu, Hutter, Hoos, and Leyton-Brown: SATzilla18

19. SATzilla vs candidate solvers HANDMADE Xu, Hutter, Hoos, and Leyton-Brown: SATzilla19

20. SATzilla vs candidate solvers HANDMADE Xu, Hutter, Hoos, and Leyton-Brown: SATzilla Oracle 20

21. SATzilla vs candidate solvers HANDMADE Pre-solving Feat. Comp. Xu, Hutter, Hoos, and Leyton-Brown: SATzilla Oracle SATzilla07 21

22. SATzilla vs candidate solvers HANDMADE Xu, Hutter, Hoos, and Leyton-Brown: SATzilla Oracle SATzilla07 (improved) 22

23. SATzilla09 [Xu et al., 2009] New techniques – New features: clause learning, … – Prediction of feature computation cost Feature time predictor Instance Minimal cost feature extractor Results: 3 Gold, 2 Silver in 2009 SAT competition - Gold for Industrial (SAT) category - Gold for Handmade (UNSAT) category - Gold for Random (SAT+UNSAT) category Xu, Hutter, Hoos, and Leyton-Brown: SATzilla23 Feature extractor Algorithm selector

24. New Results: Cost sensitive classification Goal: directly optimize the objective [Xu et al., 2011] – Cost sensitive classification for every pair of candidates – Vote to decide the order of candidates Improved performance over regression models, particularly with heterogeneous data sets New SATzilla on SAT competition 2011 Xu, Hutter, Hoos, and Leyton-Brown: SATzilla Random Gap closed: 80% 55% 33% Oracle Handmade Industrial 24

25. Recent work: Combining algorithm selection and algorithm configuration Problem: what if you only have one strong solver but it is highly parameterized? Hydra: combine algorithm selection and algorithm configuration [Xu et al., 2010; 2011] – Automatically find strong, uncorrelated parameter settings – Construct portfolio from these Results: clear improvement for both SAT and MIP – 2 × to 31 × faster for MIP – 1.2 × to 73 × faster for SAT Xu, Hutter, Hoos, and Leyton-Brown: SATzilla25

26. Conclusion Portfolio-based algorithm selection can improve performance and robustness SATzilla-like portfolio approaches are the state of the art for many problems New directions since our JAIR paper: – better predictive models – automatic portfolio construction (Hydra) Xu, Hutter, Hoos, and Leyton-Brown: SATzilla26