Presentation is loading. Please wait.

Presentation is loading. Please wait.

Lin Xu, Holger H. Hoos, Kevin Leyton-Brown

Published byBenedict Harmon Modified over 6 years ago

Similar presentations

Presentation on theme: "Lin Xu, Holger H. Hoos, Kevin Leyton-Brown"— Presentation transcript:

1 Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection
Lin Xu, Holger H. Hoos, Kevin Leyton-Brown Department of Computer Science University of British Columbia

2 Two automated algorithm design ideas
SATzilla [Xu, Hutter, Hoos, Leyton-Brown, 2007; 2008] portfolio-based algorithm selection SATenstein [KhudaBukhsh, Xu, Hoos, Leyton-Brown, 2009] algorithm design via automatic configuration Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

3 Two automated algorithm design ideas
SATzilla [Xu, Hutter, Hoos, Leyton-Brown, 2007; 2008] portfolio-based algorithm selection Exploit per-instance variation between solvers using learned runtime models practical: e.g., won 10 medals in 2007, 2009 SAT competitions fully automated: requires only cluster time rather than human design effort Key drawback: requires a set of strong, relatively uncorrelated candidate solvers can’t be applied in domains for which such solvers do not exist Some particularly related work: [Rice, 1976]; [Leyton-Brown, Nudelman & Shoham, 2003; 2009]; [Guerri & Milano, 2004]; [Nudelman, Leyton-Brown, Shoham & Hoos, 2004] Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

4 Two automated algorithm design ideas
SATenstein [KhudaBukhsh, Xu, Hoos, Leyton-Brown, 2009] algorithm design via automatic configuration Instead of manually exploring a design space, build a highly-parameterized algorithm and then configure it automatically Can find powerful, novel designs matched or outperformed existing SLS algorithms on six SAT domains But: only produces single algorithms designed to perform well on the entire training set Some particularly related work: [Gratch & Dejong, 1992]; [Fukunaga, 2002]; [Balaprakash, Birattari & Stutzle, 2007]; [Hutter, Babic, Hoos & Hu, 2007]; [Ansotegui, Sellmann & Tierney, 2009]; [Hutter, Hoos, Stutzle & Leyton-Brown, 2009] Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

5 Two automated algorithm design ideas
Hydra automatic portfolio synthesis Starting from a single parameterized algorithm, automatically find a set of uncorrelated configurations that can be used to build a strong portfolio. Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

6 Plan of This Talk Background Portfolio Synthesis Experimental Results
SATzilla: Portfolio-Based Algorithm Selection SATenstein: Algorithm Configuration as Design Portfolio Synthesis Related Work Hydra Experimental Results Conclusions and Future Work Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

7 SATzilla: Portfolio-Based Algorithm Selection
[Xu, Hutter, Hoos, Leyton-Brown, 2007; 2008] Metric Training Set Given: training set of instances performance metric candidate solvers portfolio builder (incl. instance features) Training: collect performance data portfolio builder learns predictive models At Runtime: predict performance select solver Candidate Solvers Portfolio Builder Novel Instance Portfolio-Based Algorithm Selector Selected Solver Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

8 Existing Algorithm Components Parameterized Algorithm
SATenstein: Automated Algorithm Design [KhudaBukhsh, Xu, Hoos, Leyton-Brown, 2009] Existing Algorithm Components Designer creates highly-parameterized algorithm from existing components Given: training set of instances performance metric parameterized algorithm algorithm configurator Configure algorithm: run configurator on training instances output is a configuration that optimizes metric Domain Expert Parameterized Algorithm Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

9 SATenstein: Automated Algorithm Design
Designer creates highly-parameterized algorithm from existing components Given: training set of instances performance metric parameterized algorithm algorithm configurator Configure algorithm: run configurator on training instances output is a configuration that optimizes metric Metric Instance set Algorithm Configurator Parameterized Algorithm New Configuration Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

10 Plan of This Talk Background Portfolio Synthesis Experimental Results
SATzilla: Portfolio-Based Algorithm Selection SATenstein: Algorithm Configuration as Design Portfolio Synthesis Related Work Hydra Experimental Results Conclusions and Future Work Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

11 Related Work Algorithm synthesis; portfolios and online algorithm selection [Minton 1993]; [Huberman, Lukose & Hogg 1997]; [Howe et al, 1999]; [Gomes & Selman 2001]; [Carchrae & Beck 2005]; [Gagliolo & Schmidhuber 2006]; [Streeter, Golovin & Smith 2007]; [Roberts & Howe, 2007]; [Gaspero & Schaerf 2007]; [Monette, Deville & van Hentenryck 2009] Two proposals for synthesis of selection-based portfolios: “Boosting as a Metaphor for Algorithm Design” [L-B et al., 2003; 2009] Stochastic Offline Programming [Malitsky & Sellmann, 2009] partition instances into k clusters based on features find best-performing algorithm for each cluster assumes that all algorithms repeatedly (1) sample from a distribution over heuristics; (2) use the sampled heuristic for one search step best-performing algorithms identified using a custom optimization method our goal is to construct an entirely general method for portfolio synthesis CP-Hydra [O’Mahony, Hebrard, Holland, Nugent, & O’Sullivan, 2008] selection-based portfolio for constraint programming Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

12 Boosting as a Metaphor for Algorithm Design [Leyton-Brown, Nudelman, Andrew, McFadden, Shoham, 2003]; [Leyton-Brown, Nudelman, Shoham, 2009] Core idea re-weight instance distribution to emphasize problems on which an existing portfolio P performs poorly Interpretation as an automatic procedure: generate a new distribution D that is hard for P find a new solver maximizing average performance on D We intended to implement this procedure. But: discovered examples in which the algorithm with best average performance does not improve the portfolio thus, the portfolio synthesis procedure can stagnate, even when other, helpful algorithms exist Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

13 Hydra: Dynamic Performance Metric
Avoid stagnation via a dynamic performance metric: return performance of s when s outperforms P return performance of P otherwise Intuitively: s is scored for its marginal contribution to P This metric is given to an off-the-shelf configurator, which optimizes it to find a new configuration s* Thus, we retain the same core idea as “boosting” build a new algorithm that explicitly aims to improve upon an existing portfolio Contrast with Stochastic Offline Programming: algorithms target sets of instances having very different features these feature differences can be irrelevant to algorithm performance Xu, Hoos, Leyton-Brown Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

14 Hydra Procedure: Iteration 1
Parameterized Algorithm Training Set Metric Candidate Solver Set Algorithm Configurator Portfolio Builder Candidate Solver Portfolio-Based Algorithm Selector Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

15 Hydra Procedure: Iteration 2
Parameterized Algorithm Training Set Metric Candidate Solver Set Algorithm Configurator Portfolio Builder Candidate Solver Portfolio-Based Algorithm Selector Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

16 Hydra Procedure: Iteration 3
Parameterized Algorithm Training Set Metric Candidate Solver Set Algorithm Configurator Portfolio Builder Candidate Solver Portfolio-Based Algorithm Selector Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

17 Hydra Procedure: After Termination
Output: Novel Instance Selected Solver Portfolio-Based Algorithm Selector Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

18 Plan of This Talk Background Portfolio Synthesis Experimental Results
SATzilla: Portfolio-Based Algorithm Selection SATenstein: Algorithm Configuration as Design Portfolio Synthesis Related Work Hydra Experimental Results Conclusions and Future Work Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

19 Problem Domain Even though Hydra is most useful in other domains, we evaluated it on SAT. High bar for comparison strong state-of-the-art solvers portfolio-based solvers already successful to be able to argue that Hydra does well, we want to compare to a strong portfolio Pragmatic benefits a wide variety of interesting datasets existing instance features SATenstein is a suitable configuration target Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

20 Experimental Setup: Hydra’s Inputs
Portfolio Builder: SATzilla framework [Xu, Hutter, Hoos, Leyton-Brown, 2008] Parameterized Solver: SATenstein-LS [KhudaBukhsh, Xu, Hoos, Leyton-Brown, 2009] Algorithm Configurator: FocusedILS 2.3 [Hutter, Hoos, Leyton-Brown, 2009] Performance Metric: Penalized average runtime (PAR) Instance Sets: 2 from SATenstein paper [KhudaBukhsh, Xu, Hoos, Leyton-Brown, 2009] 2 from previous SAT competitions Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

21 Experimental Setup: Challengers
Individual state-of-the-art solvers 11 manually-crafted SLS solvers all 7 SLS winners of any SAT competition 2002 – 2007 4 other prominent solvers 6 SATenstein solvers Also considered portfolios of challengers used same portfolio builder (SATenstein) Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

22 Performance Summary Solver RAND HAND BM INDU Best Challenger (of 17)
224.53 11.89 Portfolio of 11 Challengers 897.37 54.04 135.84 Portfolio of 17 Challengers 813.72 3.06* 7.74* Hydra (7 iterations) 631.35 3.06 7.77 * Statistically insignificant performance difference (sign rank test). Hydra’s performance was significantly better in all other pairings. Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

23 Performance Progress, RAND
Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

24 Selection Percentages After 7 Iterations, RAND
Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

25 Improvement After 7 Iterations, RAND
Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

26 Conclusions Thank You! Hydra: an automatic design approach combining
portfolio-based algorithm selection (here: “SATzilla”) automated algorithm configuration (here: “SATenstein”) Completely automated Algorithm/configurator/portfolio-builder agnostic Most useful in domains where few strong solvers exist Nevertheless met or exceeded state-of-the-art performance on SLS for SAT in 4 domains Thank You! Xu, Hoos, Leyton-Brown. Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

Download ppt "Lin Xu, Holger H. Hoos, Kevin Leyton-Brown"

Similar presentations

Automated Parameter Setting Based on Runtime Prediction: Towards an Instance-Aware Problem Solver Frank Hutter, Univ. of British Columbia, Vancouver, Canada.

Automated Parameter Setting Based on Runtime Prediction: Towards an Instance-Aware Problem Solver Frank Hutter, Univ. of British Columbia, Vancouver, Canada.
SATzilla: Portfolio-based Algorithm Selection for SAT Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown Department of Computer Science University.

SATzilla: Portfolio-based Algorithm Selection for SAT Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown Department of Computer Science University.
Hydra-MIP: Automated Algorithm Configuration and Selection for Mixed Integer Programming Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown Department.

Hydra-MIP: Automated Algorithm Configuration and Selection for Mixed Integer Programming Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown Department.
On-line learning and Boosting

On-line learning and Boosting
Automatic Tuning1/33 Boosting Verification by Automatic Tuning of Decision Procedures Domagoj Babić joint work with Frank Hutter, Holger H. Hoos, Alan.

Automatic Tuning1/33 Boosting Verification by Automatic Tuning of Decision Procedures Domagoj Babić joint work with Frank Hutter, Holger H. Hoos, Alan.
On the Potential of Automated Algorithm Configuration Frank Hutter, University of British Columbia, Vancouver, Canada. Motivation for automated tuning.

On the Potential of Automated Algorithm Configuration Frank Hutter, University of British Columbia, Vancouver, Canada. Motivation for automated tuning.
Patch to the Future: Unsupervised Visual Prediction

Patch to the Future: Unsupervised Visual Prediction
Automatic Algorithm Configuration based on Local Search EARG presentation December 13, 2006 Frank Hutter.

Automatic Algorithm Configuration based on Local Search EARG presentation December 13, 2006 Frank Hutter.
SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British.

SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British.
Hierarchical Hardness Models for SAT Lin Xu, Holger H. Hoos and Kevin Leyton-Brown University of British Columbia {xulin730, hoos,

Hierarchical Hardness Models for SAT Lin Xu, Holger H. Hoos and Kevin Leyton-Brown University of British Columbia {xulin730, hoos,
Integrating Bayesian Networks and Simpson’s Paradox in Data Mining Alex Freitas University of Kent Ken McGarry University of Sunderland.

Integrating Bayesian Networks and Simpson’s Paradox in Data Mining Alex Freitas University of Kent Ken McGarry University of Sunderland.
Frank Hutter, Holger Hoos, Kevin Leyton-Brown

Frank Hutter, Holger Hoos, Kevin Leyton-Brown
Hierarchical Hardness Models for SAT Hierarchical Hardness Models for SAT Building the hierarchical hardness models 1.The classifier is trained on a set.

Hierarchical Hardness Models for SAT Hierarchical Hardness Models for SAT Building the hierarchical hardness models 1.The classifier is trained on a set.
CBLOCK: An Automatic Blocking Mechanism for Large-Scale Deduplication Tasks Ashwin Machanavajjhala Duke University with Anish Das Sarma, Ankur Jain, Philip.

CBLOCK: An Automatic Blocking Mechanism for Large-Scale Deduplication Tasks Ashwin Machanavajjhala Duke University with Anish Das Sarma, Ankur Jain, Philip.
©2008 Srikanth Kallurkar, Quantum Leap Innovations, Inc. All rights reserved. Apollo – Automated Content Management System Srikanth Kallurkar Quantum Leap.

©2008 Srikanth Kallurkar, Quantum Leap Innovations, Inc. All rights reserved. Apollo – Automated Content Management System Srikanth Kallurkar Quantum Leap.
1 Adaptive Problem-Solving for Large-Scale Scheduling Problems: A Case Study by Jonathan Gratch and Steve Chien Published in JAIR, 1996 EARG presentation.

1 Adaptive Problem-Solving for Large-Scale Scheduling Problems: A Case Study by Jonathan Gratch and Steve Chien Published in JAIR, 1996 EARG presentation.
Treatment Learning: Implementation and Application Ying Hu Electrical & Computer Engineering University of British Columbia.

Treatment Learning: Implementation and Application Ying Hu Electrical & Computer Engineering University of British Columbia.
A Framework for Elastic Execution of Existing MPI Programs Aarthi Raveendran Graduate Student Department Of CSE 1.

A Framework for Elastic Execution of Existing MPI Programs Aarthi Raveendran Graduate Student Department Of CSE 1.
Parallel Algorithm Configuration Frank Hutter, Holger Hoos, Kevin Leyton-Brown University of British Columbia, Vancouver, Canada.

Parallel Algorithm Configuration Frank Hutter, Holger Hoos, Kevin Leyton-Brown University of British Columbia, Vancouver, Canada.
RecBench: Benchmarks for Evaluating Performance of Recommender System Architectures Justin Levandoski Michael D. Ekstrand Michael J. Ludwig Ahmed Eldawy.

RecBench: Benchmarks for Evaluating Performance of Recommender System Architectures Justin Levandoski Michael D. Ekstrand Michael J. Ludwig Ahmed Eldawy.

Similar presentations

Ads by Google