Hierarchical Hardness Models for SAT Hierarchical Hardness Models for SAT Building the hierarchical hardness models 1.The classifier is trained on a set of training instances, then tested on validation and test data sets. 2.Three empirical hardness models M sat, M unsat, M uncond are trained using training data with satisfiable, unsatisfiable and mixture instances. Each model is used to predict the runtime of every instance in the test set, and the responses are r sat, r unsat, r uncond. 3.For each test instance, the predicted runtime is selected from one of three response sets based on its classification score and score threshold of sat and unsat. Those thresholds are selected by cross-validation. Classifying SAT Instances Lin Xu, Holger H. Hoos, Kevin Leyton-Brown, University of British Columbia Unconditional model: the model trained on a mixture of satisfiable and unsatisfiable instances. Magic model: if we had an oracle that could determine the satisfiability of an unsolved test instance, we could use models trained on satisfiable instances (M sat ) or unsatisfiable instances (M unsat ) to predict the runtime. Background and MotivationHierarchical Hardness Models One goal of using empirical hardness models is to predict the runtime of an algorithm based on some polytime-computable features. We use ridge linear regression in this research. The response is a linear combination of basis functions: The free parameters  are computed by: Encouraging Results rand3-var, rand3-fix QCP, SW-GCP Four types of SAT-distributions: Random 3-SAT Structured SAT 84 features can be classified into nine categories: problem size, variable-clause graph, variable graph, clause graph, balance features, proximity-to-horn formulae, LP- based, DPLL search space and local search space Solvers for different problem distributions: Random 3-SAT Structured SAT kcnfs, oksolver oksolver, zchaff, sato march_dl, satz satelite, minisat, satzoo Motivation Much simpler and more accurate empirical hardness models can be learned when all instances are either satisfiable or unsatisfiable. Approximating the performance of the magic model offers the possibility of performance gains for empirical hardness models. Using the wrong model could result in a significant reduction in performance. Key point: An accurate, computationally efficient way for distinguishing between satisfiable and unsatisfiable instances. SMLR: Sparse Multinomial Logistic Regression Input vector: the same set of raw features as that in regression Output: the probabilities of an instance belonging to each class 1. High overall classification accuracy 2. Big fraction of instances with high classification scores Distribution: QCP; Solver:satelite Left: rand3-var; Right: QCP Left: rand3-fix; Right: SW-GCP 3. High classification accuracy for a small number of features Experimental Setup Using only five features can achieve over 97% of the accuracy obtained using all features. Local search-based features turned out to be very important for all four data sets. The important features for regression and classification are similar. Using a classifier to distinguish satisfiable and unsatisfiable instances is feasible and reliable. Since all the features are computed for ridge linear regression anyway, the classification is free. NP-hardness of SAT problems indicates that no existing classifier can achieve 100% classification accuracy. Experimental Results Greater accuracy and less bias runtime prediction Distribution: rand3-var; Solver: satz References 1.L. Xu, H. H. Hoos, K. Leyton-Brown. Hierarchical hardness model. Submitted April, E. Nudelman, K. Leyton-Brown, H. H. Hoos, A. Devkar, and Y. Shoham. Understanding random SAT: Beyond the clauses-to-variables ratio. In CP 04, , Acknowledgement: This research was supported by a Precarn scholarship 16 th Annual Canadian Conference on Intelligent Systems, Victoria, BC, May, 2006