1. Problem Solving With Constraints
Arc Consistency
Problem Solving With Constraints
CSCE421/821, Spring 2011
All questions:
Berthe Y. Choueiry (Shu-we-ri)
Avery Hall, Room 360
Tel: +1(402)

2. Lecture Sources Required reading Recommended reading
Algorithms for Constraint Satisfaction Problems, Mackworth and Freuder AIJ'85 (focus on Node consistency, arc consistency)
Sections 3.1 & 3.2 Chapter 3. Constraint Processing. Dechter
Recommended reading
Bartak: Consistency Techniques (link)
Constraint Propagation with Interval Labels, Davis, AIJ'87

3. Outline Motivation (done) & background
Node consistency and its complexity
Arc consistency and its complexity
Criteria for performance comparison and CSP parameters.
Experiments
Random CSPs (Model B)
Statistical Analysis

4. Basic Consistency: Notation
Examining finite, binary CSPs
Notation:
Given variables i,j,k with values x,y,z, the predicate
Pijk(x,y,z) is true iff the 3-tuple (x,y,z)  Rijk
Node consistency: checking Pi(x)
Arc consistency: checking Pij(x,y)

5. Worst-case asymptotic complexity
time
space
Worst-case complexity as a function of the input parameters
Upper bound: f(n) is O(g(n)) means that f(n)  c.g(n)
f grows as g or slower
Lower bound: f(n) is  (h(n)) means that f(n)  c.h(n)
f grows as g or faster
Input parameters for a CSP:
n = number of variables
a = (max) size of a domain
dk = degree of Vk ( n-1)
e = number of edges (or constraints)  [(n-1), n(n-1)/2]

6. Outline Motivation (done) & background
Node consistency and its complexity
Arc consistency and its complexity
Criteria for performance comparison and CSP parameters.
Experiments
Random CSPs (Model B)
Statistical Analysis

7. If Dvi (Di=∅) Then BREAK
Node consistency (NC)
Procedure NC({V1,V2,…,Vn})
For i  1 until n do DVi  DVi  { x | Pi(x) }
For each variable, we check a values
We have n variables, we do n.a checks
NC is O(a.n)
Alert: check for domain wipe-out, domain annihilation
If Dvi (Di=∅) Then BREAK

8. Outline Motivation (done) & background
Node consistency and its complexity
Arc consistency and its complexity
Algorithms AC1, AC3, AC4, AC2001
Criteria for performance comparison and CSP parameters.
Experiments
Random CSPs (Model B)
Statistical Analysis

9. Arc Consistency Adapted from Dechter
Definition: Given a constraint graph G,
A variable Vi is arc consistent relative to Vj iff for every value a  DVi, there exists b  DVj | (a,b)  RVi,Vj
The constraint CVi,Vj is arc consistent iff
Vi is arc consistent relative to Vj and
Vj is arc consistent relative to Vi
A binary CSP is arc-consistent iff every constraint (or sub-graph of size 2) is arc consistent 1 2 3 Vi Vj

10. Main Functions Variable-value pair CHECK(⟨Vi,a⟩,⟨Vj,b⟩)
(Vi,a), ⟨Vi,a⟩: variable, value from its domain
CHECK(⟨Vi,a⟩,⟨Vj,b⟩)
Returns true if (a,b)∈RVi,Vj, false otherwise
SUPPORTED(⟨Vi,a⟩,Vj)
Verifies that ⟨Vi,a⟩ has at least one support in CVi,Vj
Is not standard ‘terminology,’ but my ‘convention’
REVISE(Vi,Vj)
Updates DVi given RVi,Vj
BREAK when domain wipe-out

11. Arc Consistency Algorithms
AC-1, AC-3, …, AC2001: arc consistency algorithms
Update all domains given all constraints
Call REVISE
Differ in how they manage updates (queue)
Complexity
Nbr of variables: n, Domain size: d, Nbr of constraints: e, degree of graph deg
#CC is a global variable
Keeps track of the number of times that the definition of a binary relation is accessed
Is a cost measure, to compare algorithms’ performance in practive
Assumption
Domains are sorted alphabetically (lexicographic)

12. CHECK(⟨Vi,a⟩,⟨Vj,b⟩) Operation Pseudo code, necessary?
Verifies whether a constraint exists between Vi,Vj
If a constraint exists
Increases #CC
Accesses the constraint between Vi,Vj
Verifies if (a,b)∈RVi,Vj
Returns true if (a,b) is consistent
Returns false if (a,b) is not consistent
If no constraints exist
Returns true (universal constraint!)
Pseudo code, necessary?
Complexity? cost in practice?

13. SUPPORTED(⟨Vi,a⟩,Vj) SUPPORTED(⟨Vi,a⟩,Vj) Complexity?
support  nil
 b  DVj
If CHECK(⟨Vi,a⟩,⟨Vi,b⟩)
Then Begin
support  true
RETURN support
End
Complexity?
Once you find a support, stop looking
Cost in practice depends on implementation

14. REVISE(Vi, Vj): Description
Function
Updates DVi given the constraint CVi,Vj
Is directional: does not update DVj
Calls SUPPORTED(⟨Vi,*⟩,Vj)
If DVi is modified
Returns true, false otherwise

15. REVISE(Vi, Vj): Pseudocode
revised  false
 x  DVi
found  SUPPORTED(⟨Vi,x⟩,Vj)
If found = false
Then Begin
revised  true
DVi  DVi \ {x}
End
RETURN revised
Complexity?

16. Revise: example R. Dechter
REVISE(Vi,Vj) Vi Vj Vi Vj 1 1 1 2 2 2 2 3 3 3

17. Arc Consistency (AC-1) AC-1 does not update Q, the queue of arcs
Procedure AC-1:
NC(Problem)
Q  {(i, j) | (i,j)  directed arcs in constraint network of Problem, i  j }
Repeat
change  false
Foreach (i, j)  Q do
Begin // for each
updated  REVISE(i, j)
If DVi = {} Then Return false Else change  (updated or change)
End // for each
Until change = false
Return Problem
AC-1 does not update Q, the queue of arcs
No algorithm can have time complexity below O(ea2)
AC1 should test for empty domains (does not in the paper)

18. Warning Most papers do not check for domain wipe-out 
In your code, have AC1
Always check for domain wipe out and
terminate/interrupt the algorithm when that occurs
Complexity versus performance
Does not affect worst-case complexity
In practice, saves lots of #CC and cycles

19. Arc Consistency (AC-1) If a domain is wiped out, AC1 returns nil
Otherwise, returns the problem given as input (alternatively, change)
Procedure AC-1:
NC(Problem)
Q  {(i, j) | (i,j)  directed arcs in constraint network of Problem, i  j }
Repeat
change  false
Foreach (i, j)  Q do
Begin // for each
updated  REVISE(i, j)
If DVi = {} Then Return false Else change  (updated or change)
End // for each
Until change = false
Return Problem

20. Arc consistency AC may discover the solution
Example borrowed from Dechter V2 V3 V1

21. Arc consistency 2. AC may discover inconsistency
Example borrowed from Dechter

22. NC & AC Example courtesy of M. Fromherz
Unary constraint x>3 is imposed
AC propagates bounds again
AC propagates bound
[0, 10]
x  y-3 x y
[0, 7]
[3, 10]
x  y-3 x y
[4, 7]
[7, 10]
x  y-3 x y

23. Complexity of AC-1 Note: Q is not modified and |Q| = 2e
Procedure AC-1:
1 begin
2 for i  1 until n do NC(i)
3 Q  {(i, j) | (i,j)  arcs(G), i  j
4 repeat
5 begin
6 CHANGE  false
for each (i, j)  Q do CHANGE  (REVISE(i, j) or CHANGE)
8 end
9 until ¬ CHANGE
10 end
Note: Q is not modified and |Q| = 2e
4  9 repeats at most n·a times
Each iteration has |Q| = 2e calls to REVISE
Revise requires at most a2 checks of Pij
 AC-1 is O(a3 · n · e)

24. AC versions AC-1 does not update Q, the queue of arcs
AC-2 iterates over arcs connected to at least one node whose domain has been modified. Nodes are ordered.
AC-3 same as AC-2, nodes are not ordered

25. AC-3
AC-3 iterates over arcs connected to at least one node whose domain has been modified
Procedure AC-3:
NC(Problem)
for i  1 until n do NC(i)
Q  { (i, j) | (i, j)  directed arcs in constraint network of Problem, i  j }
While Q is not empty do
Begin
select and delete any arc (k,m)  Q
If REVISE(k,m) Then Q  Q  { (I,k) | (I,k)  arcs(G), ik, im }
End
(Don’t forget to make AC-3 check for domain wipe out)

26. Complexity of AC-3 Procedure AC-3: 1 begin
2 for i  1 until n do NC(i)
3 Q  {(i, j) | (I,j)  arcs(G), i  j }
4 While Q is not empty do
5 begin
select and delete any arc (k,m)  Q
7 If Revise(k, m) then Q  Q  { (i,k) | (I,k)  arcs(G), ik, im }
8 end
9 end
First |Q| = 2e, then it grows and shrinks 48
Worst case:
1 element is deleted from DVk per iteration
none of the arcs added is in Q
Line 7: a·(dk - 1)
Lines 4 - 8:
Revise: a2 checks of Pij
 AC-3 is O(a2(2e + a(2e-n)))
Connected graph (e  n – 1), AC-3 is O(a3e)
If AC-p, AC-3 is O(a2e)  AC-3 is (a2e)
Complete graph: O(a3n2)

27. Example: Apply AC-3 Example: Apply AC-3 Thanks to Xu Lin
DV1 = {1, 2, 3, 4, 5}
DV2 = {1, 2, 3, 4, 5}
DV3 = {1, 2, 3, 4, 5}
DV4 = {1, 2, 3, 4, 5}
CV2,V3 = {(2, 2), (4, 5), (2, 5), (3, 5), (2, 3), (5, 1), (1, 2), (5, 3), (2, 1), (1, 1)}
CV1,V3 = {(5, 5), (2, 4), (3, 5), (3, 3), (5, 3), (4, 4), (5, 4), (3, 4), (1, 1), (3, 1)}
CV2,V4 = {(1, 2), (3, 2), (3, 1), (4, 5), (2, 3), (4, 1), (1, 1), (4, 3), (2, 2), (1, 5)}
{ 1, 2, 3, 4, 5 } V1 V3 V2 V4
{ 1, 3, 5 }
{ 1, 2, 3, 5 }
{ 1, 2, 3, 4 }

28. Applying AC-3 Thanks to Xu Lin
Queue = {CV2, V4, CV4,V2, CV1,V3, CV2,V3, CV3, V1, CV3,V2}
Revise(V2,V4): DV2 DV2 \ {5} = {1, 2, 3, 4}
Queue = {CV4,V2, CV1,V3, CV2,V3, CV3, V1, CV3, V2}
Revise(V4, V2): DV4  DV4 \ {4} = {1, 2, 3, 5}
Queue = {CV1,V3, CV2,V3, CV3, V1, CV3, V2}
Revise(V1, V3): DV1  {1, 2, 3, 4, 5}
Queue = {CV2,V3, CV3, V1, CV3, V2}
Revise(V2, V3): DV2  {1, 2, 3, 4}

29. Applying AC-3 (cont.) Thanks to Xu Lin
Queue = {CV3, V1, CV3, V2}
Revise(V3, V1): DV3  DV3 \ {2} = {1, 3, 4, 5}
Queue = {CV2, V3, CV3, V2}
Revise(V2, V3): DV2  DV2
Queue = {CV3, V2}
Revise(V3, V2): DV3  {1, 3, 5}
Queue = {CV1, V3}
Revise(V1, V3): DV1  {1, 3, 5} END

30. AC-4 Mohr & Henderson (AIJ 86): AC-3 O(a3e)  AC-4 O(a2e)
AC-4 is optimal
Trade repetition of consistency-check operations with heavy bookkeeping on which and how many values support a given value for a given variable
Data structures it uses:
m: values that are active, not filtered
s: lists all vvp's that support a given vvp
counter: given a vvp, provides the number of support provided by a given variable
How it proceeds:
Generates data structures
Prepares data structures
Iterates over constraints while updating support in data structures

31. Step 1: Generate 3 data structures
m and s have as many rows as there are vvp’s in the problem
counter has as many rows as there are tuples in the constraints 4 ), , ( 1 2 3 V
counter M

32. Step 2: Prepare data structures
Data structures: s and counter.
Checks for every constraint CVi,Vj all tuples Vi=ai, Vj=bj_)
When the tuple is allowed, then update:
s(Vj,bj)  s(Vj,bj)  {(Vi, ai)} and
counter(Vj,bj)  (Vj,bj) + 1
Update counter ((V2, V3), 2) to value 1
Update counter ((V3, V2), 2) to value 1
Update s-htable (V2, 2) to value ((V3, 2))
Update s-htable (V3, 2) to value ((V2, 2))
Update counter ((V2, V3), 4) to value 1
Update counter ((V3, V2), 5) to value 1
Update s-htable (V2, 4) to value ((V3, 5))
Update s-htable (V3, 5) to value ((V2, 4))

33. Constraints CV2,V3 and CV3,V2
Nil
(V3, 2)
((V2, 1), (V2, 2))
(V3, 3)
((V2, 5), (V2, 2))
(V2, 2)
((V3,1), (V3,3), (V3, 5), (V3,2))
(V3, 1)
((V2, 1), (V2, 2), (V2, 5))
(V2, 3)
((V3, 5))
(V1, 2)
(V2, 1)
((V3, 1), (V3, 2))
(V1, 3)
(V3, 4)
(V4, 2)
(V3, 5)
((V2, 3), (V2, 2), (V2, 4))
(v4, 3)
(V1, 1)
(V2, 4)
(V2, 5)
((V3, 3), (V3, 1))
(V4, 1)
(V1, 4)
(V1, 5)
(V4, 4)
Counter
(V4, V2), 3
(V4, V2), 2
(V4, V2), 1
(V2, V3), 1 2
(V2, V3), 3 4
(V4, V2), 5 1
(V2, V4), 1
(V2, V3), 4
(V4, V2), 4
(V2, V4), 2
(V2, V3), 5
(V2, V4), 3
(V2, V4), 4
(V2, V4), 5
(V3, V1), 1
(V3, V1), 2
(V3, V1), 3 …
(V3, V2), 4
Etc…
Etc.
Updating m
Note that (V3, V2),4  0 thus
we remove 4 from the domain of V3
and update (V3, 4)  nil in m
Updating counter
Since 4 is removed from DV3 then
for every (Vk, l) | (V3, 4)  s[(Vk, l)],
we decrement counter[(Vk, V3), l] by 1

34. AC2001 When checking (Vi,a) against Vj
Keep track of the Vj value that supports (Vi,a)
Last((Vi,a),Vi)
Next time when checking again (Vi,a) against Vj, we start from Last((Vi,a),Vi) and don’t have to traverse again the domain of Vj again or list of tuples in CViVj
Big savings..

35. Summary of arc-consistency algorithms
AC-4 is optimal (worst-case) [Mohr & Henderson, AIJ 86]
Warning: worst-case complexity is pessimistic.
Better worst-case complexity: AC-4
Better average behavior: AC [Wallace, IJCAI 93]
AC-5: special constraints [Van Hentenryck, Deville, Teng 92]
functional, anti-functional, and monotonic constraints
AC-6, AC-7: general but rely heavily on data structures for bookkeeping [Bessière & Régin]
Now, back to AC-3: AC-2000, AC-2001≡AC-3.1, AC3.3, etc.
Non-binary constraints:
GAC (general) [Mohr & Masini 1988]
all-different (dedicated) [Régin, 94]

36. Outline Motivation (done) & background
Node consistency and its complexity
Arc consistency and its complexity
Performance comparison
Criteria and CSP parameters
Experiments
Random CSPs (Model B)
Statistical Analysis

37. CSP parameters ⟨n,a, t, d⟩ n is number of variables
a is maximum domain size
t is constraint tightness:
d is constraint density
where e is the #constraints, emin=(n-1), and emax = n(n-1)/2
Lately, we use constraint ratio p = e/emax
→ Constraints in random problems often generated uniform
→ Use only connected graphs (throw the unconnected ones away)

38. Criteria for performance comparison
Bounding time and space complexity (theoretical)
worst-case: always
average-case: never
Best-case: sometimes
Counting #CC and #NV (theoretical, empirical)
Measuring CPU time (empirical)

39. Performance comparison Courtesy of Lin XU
AC-3, AC-7, AC-4 on n=10,a=10,t,d=0.6 displaying #CC and CPU time
AC-3
AC-4
AC-7

40. Wisdom (?) Free adaptation from Bessière
When a single constraint check is very expensive to make, use AC-7.
When there is a lot of propagation, use AC-4
When little propagation and constraint checks are cheap, use AC-3x

41. Performance comparison Courtesy of Shant
AC3, AC3.1, AC4 on on n=40,a=16,t,d=0.15 displaying #CC and CPU time

42. AC-what? Instructor’s personal opinion Used to recommend using AC-3
Now, recommend using AC2001
Do the project on AC-* if you are curious.. [Régin 03]

43. AC is not enough Example borrowed from Dechter
Arc-consistent?
Satisfiable?
 seek higher levels of consistency V V 1 1
b a
a b = = V V V V 2 2 3 3
a b
a b
b a
a b =

44. WARNING
→ Completeness: (i.e., for solving the CSP) Running the Waltz Algorithm does not solve the problem.
A=2  B=3 is still not a solution!
→ Quiescence: The Waltz algorithm may go into infinite loops even if problem is solvable
x  [0, 100] x = y
y  [0, 100] x = 2y
→ Davis characterizes the completeness and quiescence of the Waltz algorithm (see Table 3) in terms of
constraint types
domain types

45. Outline Motivation (done) & background
Node consistency and its complexity
Arc consistency and its complexity
Criteria for performance comparison and CSP parameters.
Experiments
Random CSPs (Model B)
Statistical Analysis

46. Empirical evaluations: random problems
Various models exist (use Model B)
Models A, B, C, E, F, etc.
Vary parameters: <n, a, t, p>
Number of variables: n
Domain size: a, d
Constraint tightness: t = |forbidden tuples| / | all tuples |
Proportion of constraints (a.k.a., constraint density, constraint probability): p1 = e / emax
Issues:
Uniformity
Difficulty (phase transition)
Solvability of instances (for incomplete search techniques)

47. Model B Input: n, a, t, p1 Generate n nodes
Generate a list of n.(n-1)/2 tuples of all combinations of 2 nodes
Choose e elements from above list as constraints to between the n nodes
If the graph is not connected, throw away, go back to step 4, else proceed
Generate a list of a2 tuples of all combinations of 2 values
For each constraint, choose randomly a number of tuples from the list to guarantee tightness t for the constraint

48. Phase transition [Cheeseman et al. ‘91]
Mostly solvable problems
Mostly un-solvable problems
Cost of solving
Critical value of order parameter
Order parameter
Significant increase of cost around critical value
In CSPs, order parameter is constraint tightness & ratio
Algorithms compared around phase transition

49. Tests Fix n, a, p1 and Fix n, a, t and
Vary t in {0.1, 0.2, …,0.9}
Fix n, a, t and
Vary p1 in {0.1, 0.2, …,0.9}
For each data point (for each value of t/p1)
Generate (at least) 50 instances
Store all instances
Make measurements
#CC, CPU time (in other contexts, #NV, #messages, etc.)

50. Comparing two algorithms A1 and A2
Store all measurements in Excel
Use Excel, R, SAS, etc. for statistical measurements
Use the t-test, paired test
Comparing measurements
A1, A2 a significantly different
Comparing ln measurements
A1is significantly better than A2
For Excel: Microsoft button, Excel Options, Adds in, Analysis ToolPak, Go, check the box for Analysis ToolPak, Go. Intall…
#CC
ln(#CC) A1 A2 i1
100
200 … i2 i3
i50

51. t-test in Excel Using ln values p  ttest(array1,array2,tails,type)
tails=1 or 2
type1 (paired)
t  tinv(p,df)
degree of freedom = #instances – 2

52. t-test with 95% confidence
One-tailed test
Interested in direction of change
When t > 1.645, A1 is larger than A2
When t  , A2 is larger than A1
When  t  1.645, A1 and A2 do not differ significantly
|t|=1.645 corresponds to p=0.05 for a one-tailed test
Two-tailed test
Although it tells direction, not as accurate as the one-tailed test
When t > 1.96, A1 is larger than A2
When t  -1.96, A2 is larger than A1
When  t  1.96, A1 and A2 do not differ significantly
|t|=1.96 corresponds to p=0.05 for a two-tailed test
p=0.05 is a US Supreme Court ruling: any statistical analysis needs to be significant at the 0.05 level to be admitted in court

53. Computing the 95% confidence interval
The t test can be used to test the equality of the means of two normal populations with unknown, but equal, variance.
We usually use the t-test
Assumptions
Normal distribution of data
Sampling distributions of the mean approaches a uniform distribution (holds when #instances  30)
Equality of variances
Sampling distribution: distribution calculated from all possible samples of a given size drawn from a given population

54. Alternatives to the t test
To relax the normality assumption, a non-parametric alternative to the t test can be used, and the usual choices are:
for independent samples, the Mann-Whitney U test
for related samples, either the binomial test or the Wilcoxon signed-rank test
To test the equality of the means of more than two normal populations, an Analysis of Variance can be performed
To test the equality of the means of two normal populations with known variance, a Z-test can be performed

55. Alerts
For choosing the value of t in general, check
For a sound statistical analysis
consult the Help Desk of the Department of Statistics at UNL
held at least twice a week at Avery Hall.
Acknowledgments: Dr. Makram Geha, Department of UNL. All errors are mine..

56. Summary Alert Local consistency methods
Do not confuse a consistency property with the algorithms for reinforcing it
For each property, many algorithms may exist
Local consistency methods
Remove inconsistent values (node, arc consistency)
Remove Inconsistent tuples (path consistency)
Get us closer to the solution
Reduce the ‘size’ of the problem & thrashing during search
Are ‘cheap’ (i.e., polynomial time)