Hierarchical Hardness Models for SAT Lin Xu, Holger H. Hoos and Kevin Leyton-Brown University of British Columbia {xulin730, hoos,

Slides:

Advertisements

Similar presentations

SATzilla: Portfolio-based Algorithm Selection for SAT Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown Department of Computer Science University.

Advertisements

Hydra-MIP: Automated Algorithm Configuration and Selection for Mixed Integer Programming Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown Department.

ECG Signal processing (2)

Integrated Instance- and Class- based Generative Modeling for Text Classification Antti PuurulaUniversity of Waikato Sung-Hyon MyaengKAIST 5/12/2013 Australasian.

SVM - Support Vector Machines A new classification method for both linear and nonlinear data It uses a nonlinear mapping to transform the original training.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.

SVM—Support Vector Machines

IBM Labs in Haifa © 2005 IBM Corporation Adaptive Application of SAT Solving Techniques Ohad Shacham and Karen Yorav Presented by Sharon Barner.

Dynamic Restarts Optimal Randomized Restart Policies with Observation Henry Kautz, Eric Horvitz, Yongshao Ruan, Carla Gomes and Bart Selman.

Automatic Tuning1/33 Boosting Verification by Automatic Tuning of Decision Procedures Domagoj Babić joint work with Frank Hutter, Holger H. Hoos, Alan.

On the Potential of Automated Algorithm Configuration Frank Hutter, University of British Columbia, Vancouver, Canada. Motivation for automated tuning.

Partitioned Logistic Regression for Spam Filtering Ming-wei Chang University of Illinois at Urbana-Champaign Wen-tau Yih and Christopher Meek Microsoft.

The Disputed Federalist Papers : SVM Feature Selection via Concave Minimization Glenn Fung and Olvi L. Mangasarian CSNA 2002 June 13-16, 2002 Madison,

Paper presentation for CSI5388 PENGCHENG XI Mar. 23, 2005

SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

1 PEGASOS Primal Efficient sub-GrAdient SOlver for SVM Shai Shalev-Shwartz Yoram Singer Nati Srebro The Hebrew University Jerusalem, Israel YASSO = Yet.

Predictive Automatic Relevance Determination by Expectation Propagation Yuan (Alan) Qi Thomas P. Minka Rosalind W. Picard Zoubin Ghahramani.

Frank Hutter, Holger Hoos, Kevin Leyton-Brown

Combining Labeled and Unlabeled Data for Multiclass Text Categorization Rayid Ghani Accenture Technology Labs.

AAAI00 Austin, Texas Generating Satisfiable Problem Instances Dimitris Achlioptas Microsoft Carla P. Gomes Cornell University Henry Kautz University of.

ML ALGORITHMS. Algorithm Types Classification (supervised) Given -> A set of classified examples “instances” Produce -> A way of classifying new examples.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Hierarchical Hardness Models for SAT Hierarchical Hardness Models for SAT Building the hierarchical hardness models 1.The classifier is trained on a set.

For Better Accuracy Eick: Ensemble Learning

Selective Sampling on Probabilistic Labels Peng Peng, Raymond Chi-Wing Wong CSE, HKUST 1.

What is the Best Multi-Stage Architecture for Object Recognition Kevin Jarrett, Koray Kavukcuoglu, Marc’ Aurelio Ranzato and Yann LeCun Presented by Lingbo.

1 Efficiently Learning the Accuracy of Labeling Sources for Selective Sampling by Pinar Donmez, Jaime Carbonell, Jeff Schneider School of Computer Science,

Machine Learning CS 165B Spring 2012

Language Identification of Search Engine Queries Hakan Ceylan Yookyung Kim Department of Computer Science Yahoo! Inc. University of North Texas 2821 Mission.

Efficient Model Selection for Support Vector Machines

8/25/05 Cognitive Computations Software Tutorial Page 1 SNoW: Sparse Network of Winnows Presented by Nick Rizzolo.

Intelligent Database Systems Lab 國立雲林科技大學 National Yunlin University of Science and Technology A data mining approach to the prediction of corporate failure.

Topic Modelling: Beyond Bag of Words By Hanna M. Wallach ICML 2006 Presented by Eric Wang, April 25 th 2008.

Transfer Learning Task. Problem Identification Dataset : A Year: 2000 Features: 48 Training Model ‘M’ Testing 98.6% Training Model ‘M’ Testing 97% Dataset.

Parallel Algorithm Configuration Frank Hutter, Holger Hoos, Kevin Leyton-Brown University of British Columbia, Vancouver, Canada.

Stochastic Subgradient Approach for Solving Linear Support Vector Machines Jan Rupnik Jozef Stefan Institute.

Combining multiple learners Usman Roshan. Bagging Randomly sample training data Determine classifier C i on sampled data Goto step 1 and repeat m times.

Ensemble Methods: Bagging and Boosting

A Novel Local Patch Framework for Fixing Supervised Learning Models Yilei Wang 1, Bingzheng Wei 2, Jun Yan 2, Yang Hu 2, Zhi-Hong Deng 1, Zheng Chen 2.

Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms Frank Hutter 1, Youssef Hamadi 2, Holger Hoos 1, and Kevin Leyton-Brown.

T. Messelis, S. Haspeslagh, P. De Causmaecker B. Bilgin, G. Vanden Berghe.

Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms: An Initial Investigation Frank Hutter 1, Youssef Hamadi 2, Holger.

1/18 New Feature Presentation of Transition Probability Matrix for Image Tampering Detection Luyi Chen 1 Shilin Wang 2 Shenghong Li 1 Jianhua Li 1 1 Department.

Quality of LP-based Approximations for Highly Combinatorial Problems Lucian Leahu and Carla Gomes Computer Science Department Cornell University.

Guest lecture: Feature Selection Alan Qi Dec 2, 2004.

CpSc 881: Machine Learning

Patrick De Causmaecker Stefaan Haspeslagh Tommy Messelis.

Classification Ensemble Methods 1

Balance and Filtering in Structured Satisfiability Problems Henry Kautz University of Washington joint work with Yongshao Ruan (UW), Dimitris Achlioptas.

Brian Lukoff Stanford University October 13, 2006.

… Algo 1 Algo 2 Algo 3 Algo N Meta-Learning Algo.

A Kernel Approach for Learning From Almost Orthogonal Pattern * CIS 525 Class Presentation Professor: Slobodan Vucetic Presenter: Yilian Qin * B. Scholkopf.

Tommy Messelis * Stefaan Haspeslagh Burak Bilgin Patrick De Causmaecker Greet Vanden Berghe *

BSP: An iterated local search heuristic for the hyperplane with minimum number of misclassifications Usman Roshan.

Predictive Automatic Relevance Determination by Expectation Propagation Y. Qi T.P. Minka R.W. Picard Z. Ghahramani.

Combining Models Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya.

Data Driven Resource Allocation for Distributed Learning

Reflectance Function Approximation

Erich Smith Coleman Platt

Table 1. Advantages and Disadvantages of Traditional DM/ML Methods

Lin Xu, Holger H. Hoos, Kevin Leyton-Brown

Schizophrenia Classification Using

Students: Meiling He Advisor: Prof. Brain Armstrong

Combining Base Learners

A New Boosting Algorithm Using Input-Dependent Regularizer

Machine Learning Ensemble Learning: Voting, Boosting(Adaboost)

Analysis for Predicting the Selling Price of Apartments Pratik Nikte

INTRODUCTION TO Machine Learning 3rd Edition

Advisor: Dr.vahidipour Zahra salimian Shaghayegh jalali Dec 2017

Presentation transcript:

Hierarchical Hardness Models for SAT Lin Xu, Holger H. Hoos and Kevin Leyton-Brown University of British Columbia {xulin730, hoos,

22 Hierarchical Hardness Models --- SATzilla-07 One Application of Hierarchical Hardness Models --- SATzilla-07  Old SATzilla [ Nudelman, Devkar, et. al, 2003 ]  2nd Random  2nd Handmade (SAT)  3rd Handmade  SATzilla-07 [Xu, et.al, 2007]  1 st Handmade  1 st Handmade (UNSAT)  1 st Random  2 nd Handmade (SAT)  3 rd Random (UNSAT)

3 Outline  Introduction  Motivation  Predicting the Satisfiability of SAT Instances  Hierarchical Hardness Models  Conclusions and Futures Work

Introduction

5  NP-hardness and solving big problems  Using empirical hardness model to predict algorithm’s runtime Identification of factors underlying algorithm’s performance Induce hard problem distribution Automated algorithm tuning [Hutter, et al. 2006] Automatic algorithm selection [Xu, et al. 2007] …

6 Empirical Hardness Model (EHM)  Predicted algorithm’s runtime based on poly-time computable features Features: anything can characterize the problem instance and can be presented by a real number  9 category features [Nudelman, et al, 2004] Prediction: any machine learning technique can return prediction of a continuous value  Linear basis function regression [Leyton-Brown et al, 2002; Nudelman, et al, 2004]

7 Linear Basis Function Regression Features () Runtime (y) f w () = w T  …

Motivation

9  If we train EHM for SAT/UNSAT separately (M sat, M unsat ), Models are more accurate and much simpler [Nudelman, et al, 2004] M oracular : always use good modelM uncond. : Trained on mixture of instance Solver: satelite; Dataset: Quasi-group completion problem

10 Motivation  If we train EHM for SAT/UNSAT separately (M sat, M unsat ), Models are more accurate and much simpler Using wrong model can cause huge error M unsat : only use UNSAT modelM sat : only use SAT model Solver: satelite; Dataset: Quasi-group completion problem

11 Motivation Question: How to select the right model for a given problem? Attempt to guess Satisfiability of given instance!

Predicting Satisfiability of SAT Instances

13 Predict Satisfiability of SAT Instances  NP-complete No poly-time method has 100% accuracy of classification  BUT For many types of SAT instances, classification with poly-time features is possible Classification accuracies are very high

14 Performance of Classification  Classifier: SMLR  Features: same as regression  Datasets rand3sat-var rand3sat-fix QCP SW-GCP DatasetClassification Accuracy sat.unsat.overall rand3sat-var rand3sat-fix QCP SW-QCP [Krishnapuram, et al. 2005]

15 Performance of Classification (QCP)

16 Performance of Classification For all datasets:  Much higher accuracies than random guessing  Classifier output correlates with classification accuracies  High accuracies with few features

Hierarchical Hardness Models

18 Back to Question Question: How to select the right model for a given problem?  Can we just use the output of classifier? NO, need to consider the error distribution  Note: best performance would be achieved by model selection Oracle! Question: How to approximate model selection oracle based on features

19 Hierarchical Hardness Models , s Features & Classifier output s: P(sat) Z Model selection Oracle y Runtime Mixture of experts problem with FIXED experts, use EM to find the parameters for z [Murphy, 2001]

20 B A Importance of Classifier’s Output Two ways: A: Using classifier’s output as a feature B: Using classifier’s output for EM initialization A+B

21 Big Picture of HHM Performance OracularUncond.Hier.OracularUncond.Hier. SolverRMSE (rand3-var)RMSE (rand3-fix) satz march_dl kcnfs Oksolver SolverRMSE (QCP)RMSE (SW-GCP) Zchaff Minisat Satzoo Satelite Sato oksolver

22 Example for rand3-var (satz) Left: unconditional model Right: hierarchical model

23 Example for rand3-fix (satz) Left: unconditional model Right: hierarchical model

24 Example for QCP (satelite) Left: unconditional model Right: hierarchical model

25 Example for SW-GCP (zchaff) Left: unconditional model Right: hierarchical model

26 Correlation Between Prediction Error and Classifier’s Confidence Left: Classifier’s output vs runtime prediction error Right: Classifier’s output vs RMSE Solver: satelite; Dataset: QCP

Conclusions and Futures Work

28 Conclusions  Models conditioned on SAT/UNSAT have much better runtime prediction accuracies.  A classifier can be used to distinguish SAT/UNSAT with high accuracy.  Conditional models can be combined into hierarchical model with better performance.  Classifier’s confidence correlates with prediction error.

29 Future Work  Better features for SW-GCP  Test on more real world problems  Extend underlying experts beyond satisfiability