Practical Tractability of CSPs by Higher Level Consistency and Tree Decomposition Shant Karakashian Dissertation Defense Collaborations: Bessiere, Geschwender, Hartke, Reeson, Scott, Woodward. Support: NSF CAREER Award # & NSF Grant No. RI Experiments were conducted on the equipment of the Holland Computing Center at UNL.

Context Constraint Satisfaction – General paradigm for modeling & solving combinatorial decision problems – Applications in Engineering, Management, Computer Science Constraint Satisfaction Problems (CSPs) – NP-complete in general Islands of tractability – Classes of CSPs solvable in polynomial time 2 Karakashian: Ph.D. Defense4/11/2013

Our Focus One tractability condition links [Freuder 82] – Consistency level to – Width of the constraint network, a structural parameter (treewidth) Catch 22 – Computing width is in NP-hard – Enforcing higher-levels of consistency may require adding constraints, which increases the treewidth Goal – Exploit above condition to achieve practical tractability 3 Karakashian: Ph.D. Defense4/11/2013

Our Approach Exploit a tree decomposition – Localize application of the consistency algorithm – Steer constraint propagation along the branches of the tree – Add redundant constraints at separators to enhance propagation Karakashian: Ph.D. Defense 4 4/11/2013 Propose consistency properties that – Enforce a (parameterized) consistency level – While preserving the width of a problem instance

Outline Background Contributions – R( ∗,m)C: Consistency property & algorithms [SAC 10, AAAI 10] – Localized consistency & structure-guided propagation – Bolstering propagation at separators [CP 12, AAAI 13] – Counting solutions – (Appendices include other incidental contributions) Conclusions & Future Research Karakashian: Ph.D. Defense 5 4/11/2013 BackgroundR(*,m)CLocallizationBolsteringSol CountingConclusions

Constraint Satisfaction Problem (CSP) Given – A set of variables, here X ={A,B,C,D,E,F,G} – Their domains, here D ={D A,D B,D C,D D,D E,D F,D G }, D i ={0,1} – A set of constraints, here C ={C 1,C 2,C 3,C 4,C 5 } where C i = R i,scope(R i ) Question: Find one solution (NPC), find all solutions (#P) R1R1 AEF R3R3 ABC R4R4 ADG R2R2 BE R5R5 BD Karakashian: Ph.D. Defense4/11/2013 R4R4 R5R5 R2R2 R1R1 R3R3 GFAEBCD A0A0 B0B0 C1C1 D1D1 E1E1 F0F0 G0G0 BackgroundR(*,m)CLocallizationBolsteringSol CountingConclusions

Graphical Representations of a CSP HypergraphPrimal graphDual graph A B C D EF G A,B,C A,E,FE,B B,D A,D,G A A A B D B B E 7 Karakashian: Ph.D. Defense ABC D E F G R4R4 R5R5 R2R2 R1R1 R3R3 4/11/2013 R4R4 R5R5 R2R2 R1R1 R3R3 BackgroundR(*,m)CLocallizationBolsteringSol CountingConclusions

Redundant Edges in a Dual Graph R 1 R 4 forces the value of A in R 1 and R 4 The value of A is enforced through R 1 R 3 and R 3 R 4 R 1 R 4 is redundant 4/11/2013Karakashian: Ph.D. Defense 8 A,B,C A,E,FE,B B,D A,D,G A A A B D B B E R4R4 R5R5 R2R2 R1R1 R3R3 BackgroundR(*,m)CLocallizationBolsteringSol CountingConclusions

Solving CSPs A CSP can be solved by – Search (conditioning) Backtrack search Iterative repair (local search) – Synthesizing & propagating constraints (inference) Backtrack search – Constructive, exhaustive exploration of search space – Variable ordering improves the performance – Constraint propagation prunes the search tree 9 Karakashian: Ph.D. Defense4/11/2013 BackgroundR(*,m)CLocallizationBolsteringSol CountingConclusions

Consistency Property: Definition Example: k-consistency requires that – For all combinations of k-1 variables … all combinations of consistent values … can always be extended to every k th variable A consistency property – Guarantees that the values of all combinations of variables of a given size verify some set of constraints – Is a necessary but not sufficient condition for partial solution to appear in a complete solution 10 Karakashian: Ph.D. Defense any consistent assignment of length k-1 k th variable 4/11/2013 BackgroundR(*,m)CLocallizationBolsteringSol CountingConclusions

Algorithms for Enforcing a Consistency Property May require adding new implicit or redundant constraints – For k-consistency: add constraints to eliminate inconsistent (k-1)-tuples – Which may increase the width of the problem We propose a consistency property that – Never increases the width of the problem – Allows us to increase & control the level of consistency – Operates by deleting tuples from relations 11 Karakashian: Ph.D. Defense any consistent assignment of length k-1 k th variable 4/11/2013 BackgroundR(*,m)CLocallizationBolsteringSol CountingConclusions

Tree Decomposition A tree decomposition: 〈 T,, 〉 – T : a tree of clusters – : maps variables to clusters – : maps constraints to clusters {A,B,C,E}, {R 2,R 3 } {A,B,D},{R 3,R 5 } {A,E,F},{R 1 } {A,D,G},{R 4 } C1C1 C2C2 C3C3 C4C4 12 Karakashian: Ph.D. Defense HypergraphTree decomposition Conditions – Each constraint appears in at least one cluster with all the variables in the constraint’s scope – For every variable, the clusters where the variable appears induce a connected subtree 4/11/2013 ABC D E F G R4R4 R5R5 R2R2 R1R1 R3R3 (C1) BackgroundR(*,m)CLocallizationBolsteringSol CountingConclusions

A separator of two adjacent clusters is the set of variables associated to both clusters Width of a decomposition/network – Treewidth = maximum number of variables in clusters Tree Decomposition: Separators 13 Karakashian: Ph.D. Defense A B C D E F G C1C1 C2C2 C3C3 C4C4 4/11/2013 {A,B,C,E},{R 2,R 3 } {A,B,D},{R 3,R 5 } {A,E,F},{R 1 } {A,D,G},{R 4 } C1C1 C2C2 C3C3 C4C4 BackgroundR(*,m)CLocallizationBolsteringSol CountingConclusions

Outline Background Contributions – R( ∗,m)C: Consistency property & algorithms [SAC 10, AAAI 10] – Localized consistency & structure-guided propagation – Bolstering propagation at separators [CP 12, AAAI 13] – Counting solutions – (Appendices include other incidental contributions) Conclusions & Future Research Karakashian: Ph.D. Defense 14 4/11/2013 BackgroundR(*,m)CLocallizationBolsteringSol CountingConclusions

Relational Consistency R( ∗,m)C 4/11/2013Karakashian: Ph.D. Defense 15 [SAC 2010, AAAI 2010] A parametrized relational consistency property Definition – For every set of m constraints – every tuple in a relation can be extended to an assignment – of variables in the scopes of the other m-1 relations R(*,m)C ≡ every m relations form a minimal CSP R(*,m)CLocallizationBolsteringSol CountingConclusions Background

Index-Tree Data Structure Given: two relations, R 1 & R 2 – Scope(R 1 )={X,A,B,C} Scope(R 2 )={A,B,C,D} For a given tuple in R 1, find matching tuples in R 2 4/11/2013Karakashian: Ph.D. Defense 16 R2R2 ABCDdeleted t1t t2t t3t t4t t1t1 t2t2 t3t3 t4t4 A B C Root R(*,m)CLocallizationBolsteringSol CountingConclusions Background

R 1 R 2 R 3 R 1 R 2 R 4 R 1 R 2 R 5 R 1 R 3 R 4 R 2 R 3 R 4 R 2 R 4 R 5 R 3 R 4 R 5 Weaken R( ∗,m)C by removing redundant edges [Jégou 89] Weakening R( ∗,m)C 17 Karakashian: Ph.D. Defense R 1 R 2 R 3 R 1 R 2 R 5 R 1 R 3 R 4 R 2 R 4 R 5 R 3 R 4 R 5 CF CG AD AE ABD ACEG BCF ADE CFG R1R1 R3R3 R2R2 R4R4 R5R5 A B C CF CG AD AE ABD ACEG BCF ADE CFG R1R1 R3R3 R2R2 R4R4 R5R5 B 4/11/2013 R( ∗,3)CwR( ∗,3)C R(*,m)CLocallizationBolsteringSol CountingConclusions Background

Characterizing R( ∗,m)C 4/11/2013Karakashian: Ph.D. Defense 18 GACmaxRPWC R3C R( ∗,2)C wR( ∗,2)C R2C R( ∗,3)CR( ∗,4)C R4C wR( ∗,3)CwR( ∗,4)C R( ∗,m)C RmCRmC wR( ∗,m)C GAC [Waltz 75] maxRPWC [Bessiere+ 08] RmC: Relational m Consistency [Dechter+ 97] [Jégou 89] R(*,m)CLocallizationBolsteringSol CountingConclusionsBackground

Empirical Evaluations (1) 4/11/2013Karakashian: Ph.D. Defense 19 AlgorithmAvg. #NodesAvg. Time sec#Completed#Fastest#BF SAT aim-100 (instances: 16, vars: 100, dom: 2, rels: 307, arity: 3) GAC9,459, wR( ∗,2)C 234, wR( ∗,3)C 3, wR( ∗,4)C SAT modifiedRenault (instances: 19, vars: 110, dom: 42, rels: 128, arity: 10) GAC1,171, wR( ∗,2)C wR( ∗,3)C wR( ∗,4)C R(*,m)CLocallizationBolsteringSol CountingConclusionsBackground

Empirical Evaluations (2) 4/11/2013Karakashian: Ph.D. Defense 20 AlgorithmAvg. #NodesAvg. Time sec#Completed#Fastest#BF UNSAT aim-100 (instances: 8, vars: 100, dom: 2, rels: 173, arity: 3) GAC--000 wR( ∗,2)C 4,619,373.02, wR( ∗,3)C 18, wR( ∗,4)C 18, UNSAT modifiedRenault (instances: 31, vars: 111, dom: 42, rels: 130, arity: 10) GAC1,171, wR( ∗,2)C wR( ∗,3)C wR( ∗,4)C R(*,m)CLocallizationBolsteringSol CountingConclusionsBackground

Algorithms for Enforcing R( ∗,m)C P ER T UPLE – For each tuple find a solution for the variables in the m-1 relations – Many satisfiability searches Effective when there are many solutions Each search is quick & easy A LL S OL – Find all solutions of problem induced by m relations, & keep their tuples – A single exhaustive search Effective when there are few or no solutions Hybrid Solvers (portfolio based) [+Scott] 4/11/2013Karakashian: Ph.D. Defense 21 t1t1 titi t2t2 t3t3 R(*,m)CLocallizationBolsteringSol CountingConclusionsBackground

Hybrid Solver Choose between P ER T UPLE & A LL S OL Parameters to characterize the problem – κ predicts if instance is at the phase transition – relLinkage approximates the likelihood of a tuple at the overlap two relations to appear in a solution Classifier built using Machine Learning – C4.5 – Random Forests 4/11/2013Karakashian: Ph.D. Defense 22 [+Scott] [Gent+ 96] R(*,m)CLocallizationBolsteringSol CountingConclusionsBackground

Decision Tree 4/11/2013Karakashian: Ph.D. Defense 23 #1≤ 0.22 No #3≤-2.79 No #7≤0.03 No #10≤10.05 Yes No #2≤ Yes No P ER T UPLE A LL S OL Yes P ER T UPLE Yes A LL S OL #7≤0.23 Yes No A LL S OL P ER T UPLE #1 κ #2 log 2 (avg( relLinkage )) #3 log 2 (stDev( relLinkage )) #7 stDev( tupPerVvpNorm ) #10 avg( relPerVar ) R(*,m)CLocallizationBolsteringSol CountingConclusionsBackground

Empirical Evaluations (3) 4/11/2013Karakashian: Ph.D. Defense 24 #Instances solved by…Average CPU sec Partition A LL S OL P ER T UPLE S OLVER C4.5 S OLVER RF #Instances A LL S OL P ER T UPLE S OLVER C4.5 S OLVER RF A 5,7775,7765,777 5, P 10,09515,45715,43914,01210, A ∪ P 15,87221,33321,21619,78915, Task: compute the minimal CSP R(*,m)CLocallizationBolsteringSol CountingConclusionsBackground

Outline Background Contributions – R( ∗,m)C: Consistency property & algorithms [SAC 10, AAAI 10] – Localized consistency & structure-guided propagation – Bolstering propagation at separators [CP 12, AAAI 13] – Counting solutions – (Appendices include other incidental contributions) Conclusions & Future Research Karakashian: Ph.D. Defense 25 4/11/2013 LocallizationBolsteringSol CountingConclusionsBackgroundR(*,m)C

Localized Consistency Consistency property cl-R( ∗,m)C – Restrict R( ∗,m)C to the clusters Constraint propagation – Guide along a tree structure 4/11/2013Karakashian: Ph.D. Defense 26 LocallizationBolsteringSol CountingConclusionsBackgroundR(*,m)C

C1C1 C2C2 C7C7 C3C3 C4C4 C5C5 C6C6 C8C8 C9C9 C 10 Triangulate the primal graph using min-fill Identify the maximal cliques using M AX C LIQUES Connect the clusters using J OIN T REE Add constraints to clusters where their scopes appear Generating a Tree Decomposition 4/11/2013Karakashian: Ph.D. Defense 27 ABC E D FGH IJK ML N R7R7 R2R2 R3R3 R4R4 R5R5 R6R6 R1R1 A B C E D F G H I J K M L N C8C8 C2C2 A,B,C,N A,I,N B,C,D,H I,M,N B,D,F,H C1C1 C3C3 C7C7 A,I,K C4C4 I,J,K C5C5 A,K,L C6C6 B,D,E,F C9C9 F,G,H C 10 Elimination order M AX C LIQUES {A,B,C,N},{R 1 } C2C2 C7C7 C3C3 C8C8 C1C1 C4C4 C5C5 C6C6 C9C9 C 10 J OIN T REE min-fill LocallizationBolsteringSol CountingConclusionsBackgroundR(*,m)C

Information Transfer Between Clusters Two clusters communicate via their separator – Constraints common to the two clusters – Domains of variables common to the two clusters 4/11/2013Karakashian: Ph.D. Defense 28 E R6R6 R5R5 R7R7 R4R4 R2R2 R1R1 R3R3 BADC F LocallizationBolsteringSol CountingConclusionsBackgroundR(*,m)C R4R4 AD BADC

Characterizing cl-R( ∗,m)C 4/11/2013Karakashian: Ph.D. Defense 29 GAC maxRPWC R3C R( ∗,2)C ≡ wR( ∗,2)C R2C R( ∗,3)CR( ∗,4)C R4C wR( ∗,3)CwR( ∗,4)C R( ∗,m)C RmCRmC wR( ∗,m)C cl-R( ∗,3)Ccl-R( ∗,2)Ccl-R( ∗,4)Ccl-R( ∗,m)C cl-w( ∗,3)Ccl-w( ∗,2)Ccl-w( ∗,4)Ccl-w( ∗,m)C GAC [Waltz 75] maxRPWC [Bessiere+ 08] RmC: Relational m Consistency [Dechter+ 97] LocallizationBolsteringSol CountingConclusionsBackgroundR(*,m)C

Empirical Evaluations: Localization 4/11/2013Karakashian: Ph.D. Defense 30 LocallizationBolsteringSol CountingConclusionsBackgroundR(*,m)C

Structure-Guided Propagation Orderings – R ANDOM : FIFO/arbitrary – S TATIC, P RIORITY, D YNAMIC : Leavesroot Structure-based propagation – S TATIC Order of M AX C LIQUES – P RIORITY : Process a cluster once in each direction Select most significantly filtered cluster – D YNAMIC Similar to P RIORITY But may process ‘active’ clusters more than once 4/11/2013Karakashian: Ph.D. Defense 31 {A,B,C,N},{R 1 } C2C2 C7C7 C3C3 C8C8 C1C1 C4C4 C5C5 C6C6 C9C9 C 10 C8C8 C2C2 A,B,C,N A,I,N B,C,D,H I,M,N B,D,F,H C1C1 C3C3 C7C7 A,I,K C4C4 I,J,K C5C5 A,K,L C6C6 B,D,E,F C9C9 F,G,H C 10 Leaves Root M AX C LIQUES LocallizationBolsteringSol CountingConclusionsBackgroundR(*,m)C

Empirical Evaluations: Propagation 4/11/2013Karakashian: Ph.D. Defense 32 + maxRPWC, m=2,4 cl-wR( ∗,3)C cl-R( ∗,| ψ (cl)|)C # Instances R ANDOM STATIC PRIORITY DYNAMIC RA NDOM STATIC PRIORITY DYNAMIC Completed UNSAT %48.4%48.6%48.9%54.5%59.5%58.9% SAT % 82.5%81.5%77.0%76.0%75.5% Fastest UNSAT %32.8%24.4%23.8%31.5%45.9%33.6%32.4% SAT %44.0%55.5%54.0%25.0%44.0%37.0%42.0% Avg. Time (sec) UNSATNbr instances 232Nbr instances SATNbr instances 162Nbr instances LocallizationBolsteringSol CountingConclusionsBackgroundR(*,m)C

Outline Background Contributions – R( ∗,m)C: Consistency property & algorithms [SAC 10, AAAI 10] – Localized consistency & structure-guided propagation – Bolstering propagation at separators [CP 12, AAAI 13] – Counting solutions – (Appendices include other incidental contributions) Conclusions & Future Research Karakashian: Ph.D. Defense 33 4/11/2013 BolsteringSol CountingConclusionsBackgroundR(*,m)CLocallization

Bolstering Propagation at Separators Localization – Improves performance – Reduces the enforced consistency level Ideally: add unique constraint – Space overhead, major bottleneck Enhance propagation by bolstering – Projection of existing constraints – Adding binary constraints – Adding clique constraints 4/11/2013Karakashian: Ph.D. Defense 34 [CP 2012, AAAI 2013] E F R6R6 R5R5 R7R7 R2R2 R1R1 R3R3 BADC R sep BolsteringSol CountingConclusionsBackgroundR(*,m)CLocallization

Bolstering Schemas: Approximate Unique Separator Constraint 4/11/ Karakashian: Ph.D. Defense E R6R6 R5R5 R7R7 R4R4 R2R2 R1R1 R3R3 BADC F R3’R3’ E F R’ 3 BADC RaRa E BADC F RyRy RxRx B A D CB A D C By projectionBinary constraintsClique constraints BolsteringSol CountingConclusionsBackgroundR(*,m)CLocallization B A D C RyRy RxRx RaRa E R3R3 DC E R6R6 R2R2 BA F E R5R5 R1R1 BC F

Resulting Consistency Properties GAC maxRPWC cl+clq-R( ∗,3)Ccl+clq-R( ∗,4)Ccl+clq-R( ∗,2)C R( ∗,4)C cl-R( ∗,| ψ(cl i ) |)C cl+proj-R( ∗,| ψ(cl i ) |)C cl+proj-R( ∗,3)C R( ∗,3)C cl-R( ∗,2)C cl+bin-R( ∗,4)Ccl+bin-R( ∗,3)C cl+bin-R( ∗,| ψ(cl i ) |)C cl+clq-R( ∗,| ψ(cl i ) |)C cl+bin-R( ∗,2)C cl+proj-R( ∗,2)C R( ∗,2)C cl-R( ∗,3)Ccl-R( ∗,4)C cl+proj-R( ∗,4)C 4/11/ Karakashian: Ph.D. Defense BolsteringSol CountingConclusionsBackgroundR(*,m)CLocallization

Empirical Evaluations 4/11/2013Karakashian: Ph.D. Defense 37 + maxRPWC, m=3,4 wR( ∗,2)CR( ∗,|(cl i )|)C #inst. GAC global local Proj. binary clique local Proj. binary clique Completed UNSAT % % % % % % % % % % SAT % % % % % % % % % % BT-Free UNSAT % % % % % % % % % % SAT % % % % % % % % % % Min(#NV) UNSAT % % % % % % % % % % SAT % % % % % % % % % % Fastest UNSAT % % % 5 1.0% 1 0.2% 1 0.2% % % % % SAT % % % % % % % % % % BolsteringSol CountingConclusionsBackgroundR(*,m)CLocallization

Outline Background Contributions – R( ∗,m)C: Consistency property & algorithms [SAC 10, AAAI 10] – Localized consistency & structure-guided propagation – Bolstering propagation at separators [CP 12, AAAI 13] – Counting solutions – (Appendices include other incidental contributions) Conclusions & Future Research Karakashian: Ph.D. Defense 38 4/11/2013 Sol CountingConclusionsBackgroundR(*,m)CLocallizationBolstering

Solution Counting BTD is used to count the number of solutions [Favier+ 09] Using C OUNT algorithm on trees [Dechter+ 87] Witness-based solution counting Find a witness solution before counting Karakashian: Ph.D. Defense 39 4/11/2013 CpCp apap wastedno solution Sol CountingConclusionsBackgroundR(*,m)CLocallizationBolstering

Empirical Evaluations Task: Count solutions – BTD versus W ITNESS BTD – GAC: Pre-processing & full look-ahead 4/11/2013Karakashian: Ph.D. Defense 40 Sol CountingConclusions BTDW ITNESS BTD Avg. #NV UNSAT (89)1,437, , SAT (101)4,785, ,735, Avg. Time (sec) UNSAT (89) SAT (101)1, , BackgroundR(*,m)CLocallizationBolstering

Empirical Evaluations 4/11/2013Karakashian: Ph.D. Defense 41 + maxRPWC, m=2,4 wR( ∗,3)CR( ∗,|(cl i )|)C #inst. GAC global local Proj. binary clique local Proj. binary clique Completed UNSAT % % % % % % % % % % SAT % % % % % % % % % % BT-Free UNSAT % % % % % % % % % % SAT % % % % % % % % % % Min(#NV) UNSAT % % % % % % % % % % SAT % % % % % % % % % % Fastest UNSAT % % % % % % % % % % SAT % % % % 9 4.5% 9 4.5% % % 9 4.5% 9 4.5% Sol CountingConclusionsBackgroundR(*,m)CLocallizationBolstering

Conclusions Question – Practical tractability of CSPs exploiting the condition linking the level of consistency to the width of the constraint graph Solution – Introduced a parameterized consistency property R( ∗,m)C – Designed algorithms for implementing it P ER T UPLE and A LL S OL Hybrid algorithms – Adapted R( ∗,m)C to a tree decomposition of the CSP Localizing R( ∗,m)C to the clusters Strategies for guiding propagation along the structure Bolstering separators to strengthen the enforced consistency – Improved the BTD algorithm for solution counting, W ITNESS BTD +Two incidental results in appendices 4/11/2013Karakashian: Ph.D. Defense 42 ConclusionsBackgroundR(*,m)CLocallizationBolsteringSol Counting

Future Research Extension to non-table constraints Automating the selection of – a consistency property – consistency algorithms [Geschwender+ 13] Characterizing performance on randomly generated problems + much more in dissertation 4/11/2013Karakashian: Ph.D. Defense 43 ConclusionsBackgroundR(*,m)CLocallizationBolsteringSol Counting

4/11/2013Karakashian: Ph.D. Defense 44 Thank You Collaborations: Bessiere, Geschwender, Hartke, Reeson, Scott, Woodward. Support: NSF CAREER Award # & NSF Grant No. RI Experiments were conducted on the equipment of the Holland Computing Center at UNL.

Computing All k-Connected Subgraphs Avoids enumerating non-connected k-subgraphs Competitive on large graphs with small k 4/11/2013Karakashian: Ph.D. Defense 45 cd eab

Solution Cover Problem is in NP-C Solution Cover Problem – Given a CSP with global constraints – is there a set of k solutions – such that ever tuple in the minimal CSP is covered by at least one solution in the set? Set Cover Problem Solution Cover Problem Set Cover Problem – Given a finite set U and a collection S of subsets of U – Are there k elements of S – Whose union is U? 4/11/2013Karakashian: Ph.D. Defense 46

W ITNESS BTD May Visit More Nodes than GAC W ITNESS BTD may search for the same solution more than once in C 2 〈 A,B,C,D 〉 = 〈 0,0,0,0 〉 No witness in C 3 〈 A,B,C,D 〉 = 〈 1,1,0,0 〉 Witness found Restart search in C 2 4/11/2013Karakashian: Ph.D. Defense 47 A B C D C1C1 C2C2 C3C3

#isnt. 230GAC Cl+proj-R( ∗,|(cl i )|)C Average Nogoods87, Goods27,180.38,811.1 Max Nogoods4,880,1278,224 Goods1,870,0511,069,081 completed (547): MEMOUT163