1. IBM Labs in Haifa © 2005 IBM Corporation Adaptive Application of SAT Solving Techniques Ohad Shacham and Karen Yorav Presented by Sharon Barner

2. IBM Labs in Haifa © 2005 IBM Corporation 2 Agenda  DPLL SAT Solvers – Terms  Motivation  Performance Metrics  Adaptive Solving  Experimentation  Conclusion

3. IBM Labs in Haifa © 2005 IBM Corporation 3 DPLL SAT Solvers  Input: Boolean Formula, in CNF  Output: Satisfying assignment / Unsatisfiable  Iterative exhaustive search  Decisions  Assign a value to a single variable – Decision Heuristic  increment decision level  BCP  propagate effect of assignment  Learning  when a conflict occurs, add a “conflict clause” to the database so that the same combination of assignments will not happen again  backtrack appropriately, decrement decision level

4. IBM Labs in Haifa © 2005 IBM Corporation 4 Motivation  SAT solving is based on heuristics and strategies  There is no winning strategy  Best choice cannot be determined beforehand Previous Solutions  Learning from a training set  does not work well for BMC – no representative set  Choosing on-the-fly according to biased random function  applicable only to decision heuristic  does not stabilize on best option Herbstritt & Becker 2003 Lagoudakis & Littman 2001, Nudelman et al 2004

5. IBM Labs in Haifa © 2005 IBM Corporation 5 Solution – Adaptive Solving  Switching options when not progressing well  Track the progress of the search on-the-fly  Decision Heuristic  Clause Deletion  Conflict Clause Generation Algorithm  Clause Replication ...

6. IBM Labs in Haifa © 2005 IBM Corporation 6 Performance Metrics  Produces a numerical score  Calculated On-The-Fly  Every fixed number of decisions evaluate the metric  Cheap to evaluate  Calculating the space to be searched is not an option... [SATometer]  Must rely on readily available information  Corresponds (roughly) to the effectiveness of the search

7. IBM Labs in Haifa © 2005 IBM Corporation 7 Metrics  Average Decision Level  Too high - implies the solver may be “stuck” on a small space with no solution  Acceptable level varies between solvers  Average Conflict Clause Size  Smaller clauses potentially reduce the space more significantly  Percentage of Binary Conflict Clauses  Binary clauses are beneficial because they cause implications with little overhead

8. IBM Labs in Haifa © 2005 IBM Corporation 8 More Metrics  BCP Ratio  Average number of steps per clause v 1  0 v1v1  A high BCP Ratio means the solver makes less decisions per second

9. IBM Labs in Haifa © 2005 IBM Corporation 9 More Metrics  Unary Clause Learning  Permanent values for variables  Each one reduces the search space by half And others...

10. IBM Labs in Haifa © 2005 IBM Corporation 10 Adaptive Solving  Evaluate the performance metric every fixed number of decisions  Given the metric score, decide whether to make a switch  Can have different switching conditions for different options  Adaptive Solving requires tuning !  Choose the parameters wisely  Tune the metric to the chosen parameter  Tune the metric to the solver structure  Tune according to the chosen domain

11. IBM Labs in Haifa © 2005 IBM Corporation 11 Insights  The Parameter to control  capable of high impact on performance – both ways  easy to switch  The Sample Size  Large enough to allow a change to take effect  Switching  Better to disable switches for a while after a switch is performed  Switching condition becomes stronger after each switch  Total limit on the number of switches

12. IBM Labs in Haifa © 2005 IBM Corporation 12 Experimentation  IBM benchmarks  Time out set to 10000 seconds  No time outs to prevent the time out constant from influencing speedup results  Parameter controlled – the value given to a decision variable  first the decision variable is chosen, and only then the value  by default - value is according to the literal with the higher score  attempting to satisfy more clauses  -sign option switches the choice to the literal with the lower score  attempting to generate more conflicts  In general, the default is much better  in some cases “-sign” improves run times significantly

13. IBM Labs in Haifa © 2005 IBM Corporation 13 Experimentation NativeSignDLCCSBINBCPUNARY SAT Time86621457986097726670270577933 Speedup-0.591.001.121.291.231.09 Min-0.100.770.870.83 0.91 Max-4.351.643.884.042.954.01 UNSAT Time14955252561306792288269126377313 Speedup-0.591.141.621.801.162.04 Min-0.070.170.510.750.410.52 Max-4.443.905.2911.751.335.90 ALL Time23618398352167616954149711969515247 Speedup-0.591.091.391.581.181.55 Min-0.070.170.510.750.410.52 Max-4.443.905.2911.752.955.90 Works better on UNSAT instances Maximum Speedup is on larger example Detrimental effect more pronounced on smaller examples Global Runtime Native Global Runtime Adaptive

14. IBM Labs in Haifa © 2005 IBM Corporation 14 Conclusion  Adaptive solving enables making use of ideas that don’t always work  Enabling an option on parts of the search space can give better results than enabling it or disabling it all of the time!  Even when the option is inherently bad for this example  Need more and better metrics  Combine metrics  Relate metrics to the parameter they control  Can also be applied in-between BMC instances  be careful – the “best configuration” for short instances is not the best for long ones (experimentation found no correlation)