1. An Empirical Study of the Performance
July 30, 2003
An Empirical Study of the Performance
of Preprocessing and Look-ahead Techniques
for Solving Finite Constraint Satisfaction Problems
Zheying Jane Yang
Constraint Systems Laboratory
Department of Computer Science and Engineering
University of Nebraska-Lincoln
Under the supervision of Prof. B.Y. Choueiry

2. Outline Motivation & results Background
Experimental design and empirical study
Results and analysis
Conclusions & relation to previous work
Summary of contributions
Future work

3. 1. Motivation CSP used to model NP-complete decision problems
Search (exponential) is necessary, improved with
Preprocessing algorithms:
remove inconsistent combinations prior to search
Look-ahead algorithms:
remove inconsistencies during search
Preprocessing
Search w/
look-ahead
CSP
Smaller CSP
Solution(s)

4. Algorithms studied & goal
Several competing algorithms
Preprocessing: removes inconsistencies prior to search
Arc-consistency: AC3, AC2001
Neighborhood inverse consistency (NIC), requires search
Look-ahead: filters search space during search
Forward checking (FC)
Maintaining arc-consistency (MAC)
Controversies about their relative performance exist
Our goal is
to characterize empirically the relative performance of
combinations of preprocessing and look-ahead schemes
as a function of the CSP’s constraint probability & tightness

5. Current beliefs & results
Our results: We specify when the above claims hold or not
Bessière & Régin 2001
AC3 vs AC2001
AC2001 is better
Claims
Issue
Freuder & Elfe 1996
NIC is better
AC vs NIC
MAC vs FC
Haralick & Elliott 1980
Nadel 1989
Bessière & Régin 1996
FC is better
MAC is much better
MAC is the winner in large spase CSPs
FC is the winner in dense CSPs
Sabin & Freuder 1994
Grant 1997
Gent & Prosser 2000
Preprocessing
Look-ahead
No winner performs extremely
well on all types of CSPs

6. 2. Background

7. What is a CSP? A CSP problem is defined as a triple P=(V,D,C)Z Y X
{(b,c), (b,d)}
{(b,e), (a,f)}
{(c,e), (d,f)}
{c,d}
{e,f}
{a,b}
A set of variables
For each variable, a finite set of possible values.
A set of constraints restriction the values that the
variables can take simultaneously.
The goal is to find
One solution
All solutions

8. Solving CSPs with systematic BT search
Backtrack search is sound and complete, but exponential
expands a partial solution
explores every combination systematically
Improve performance of search
Preprocessing (constraint filtering/propagation)
Look-ahead (constraint filtering during search)
Backtracking: chronological/intelligent
Variable & value ordering: static/dynamic, heuristic
etc.
Preprocessing (constraint filtering/propagation)
Removes inconsistent combinations, which reduces the size of the problem and the corresponding space
2. Intelligent backtraking
3. Variable ordering and value ordering (etc=symmetry, decomposition)

9. Solving a CSP We study the performance of Preprocessing
Look-ahead
Preprocessing
Following an assignment, removes values from the domains of the future variables that are inconsistent with the current assignment.
Enforces different levels of local consistency by deleting inconsistent values from variable domains
Algorithms: AC3, AC2001, NIC
Algorithms: FC & MAC
Say a few words about AC3/2001: guarantee the same property, but are different implementations. AC2001 uses special structures
Say that NIC guarantees a stronger consistency than AC in general, but it requires some
Introduce: past, current,future variables
Say how FC and MAC work.
Forward Checking (FC) algorithm
Checks forward each time to make a new instantiation
Each time a variable is assigned a value, it will restrict
the domains of future variables.
Remove inconsistent values from the current domain.
If current domain is wiped out backtrack occurs.
We study the performance of
Preprocessing
Hybrid search = preprocessing x look-ahead algorithms

10. Preprocessing a graph coloring problem
{R, G}
{G}
{R, G, B} V1 V3 V2 V3
{R, G}
{G}
{R, G, B} V1 V2
{R, G}
{G}
{R, B} V1 V3 V2  
{R}
{G}
{R, B} V1 V3 V2
{R}
{G}
{B} V1 V3 V2 

11. Solving CSP using forward checking (FC)
{R, G}
{G}
{R} V1 V3 V2
Start

12. Solving CSP using forward checking (FC)
Start
Domain wiped-out
backtrack

13. Solving CSP using forward checking (FC)
{R, G}
{G} V1 V3 V2
Start
Domain wiped-out
backtrack

14. Solving CSP using forward checking (FC)
{R, G}
{G}
{B} V1 V3 V2
Start

15. Solving CSP using forward checking (FC)
Start V1 V2 V3 V1
{B} V2 V3
{R}
{G}

16. Solving CSP using forward checking (FC)
Start
Solution!

17. 3. Experimental design
& empirical study

18. Algorithms tested Preprocessing  Look-ahead AC3 AC2001 FC MAC-AC3
NIC 
Preprocessing: 5 algorithms
Look-ahead: 3 algorithms
Hybrid search: 7 combinations

19. Working assumptions Considered only binary constraints
Restricted to finding first solution
Restricted to chronological backtracking
Use least domain (LD) as variable ordering heuristic
Variable ordering done dynamically
No value ordering heuristic, too costly in general

20. Problems tested
Random CSPs
We need many instances with similar characteristics in order to analyze the performance of search
Real-world problems cannot be controlled by explicit parameters to enable statistical analysis of the performance
Generated instances
connected graphs
instances guaranteed solvable

21. Control parameters < n, d, C, t >
Number of variables n
We choose n = 50
Domain size d (uniform)
We choose d = 10. Thus, problem size is 1050, relatively large
Constraint tightness t = (uniform)
We vary t = [0.1, 0.2, …, 0.9] and t = [0.05, 0.1, …, 0.95]
Number of constraints C
determines constraint probability p= , Cmax= n(n-1)/2
We vary C = [20, 490] corresponding to p = [0.024, 0.4]
We report C = 30, 50, 80, 120, 130, 245, 340, 490
Disallowed tuples
All possible tuples C
Cmax

22. Comparisons Evaluation criteria
Filtering effectiveness measures reduction of CSP size
Number of constraint checks #CC measures filtering effort
Number of nodes visited #NV measures backtracking effort
CPU time [ms] measures overall performance
Since constraints are defined in extension, CPU time reflects #CC
Preprocessing: Filtering effectiveness, #CC, CPU time
Hybrid search: #CC, #NV, CPU time

23. Sampling and code characteristics
Evaluated performance by running each algorithm
on 30 CSP instances of equal characteristics,
calculating average & standard deviation values for each evaluation criterion
Generated approximately 6,000 CSP instances
Implemented in JAVA:
21 JAVA classes
4,000 lines of code
Experiments carried out on PrairieFire.unl.edu

24. 4. Results and analysis
Part I: Preprocessing
Part II: Hybrids

25. Preprocessing: filtering effectiveness
NIC reduces search space at a large amount

26. Preprocessing results
» Better than,  comparable
p < 0.05
0.05  p < 0.1
0.1  p < 0.2
p  0.2
Filtering effectiveness
Filtering effectiveness increases with p
NIC-MAC-x » NIC-FC » AC3  AC2001

27. CC (NICx) >> CC (AC3, AC2001)
Preprocessing: #CC, sparse CSPs
CC (AC3) > CC (AC2001)
CC (NICx) >> CC (AC3, AC2001)

28. Preprocessing: #CC, denser CSPs
NIC becomes prohibitive
When p=0.2, NICx are all costly
NIC should never be combined with MAC

29. Preprocessing results
» Better than,  comparable
p < 0.05
0.05  p < 0.1
0.1  p < 0.2
p  0.2
Filtering effectiveness
Filtering effectiveness increases with p
NIC-MAC-x » NIC-FC » AC3  AC2001
#CC
AC2001 » AC3 » NICx
Never use MAC with NIC

30. Preprocessing: CPU time
When p > 0.2, NIC-x is too costly

31. Preprocessing results
» Better than,  comparable
p < 0.05
0.05  p < 0.1
0.1  p < 0.2
p  0.2
Filtering effectiveness
Filtering effectiveness increases with p
NIC-MAC-x » NIC-FC » AC3  AC2001
#CC
AC2001 » AC3 » NICx
Never use MAC with NIC
CPU time
AC3 » AC2001 » NICx

32. Preprocessing: summary
When to use what
p < 0.05
0.05  p < 0.1
0.1  p < 0.2
p  0.2
ACx
Preprocessing not effective
Not effective
Effective
NIC-x
Quite effective
Effective, but MAC too costly
Too costly, avoid
AC2001 is not significantly better than AC3, and is not worth the extra data structures
In general, we disagree with Bessière and Régin
NIC-x: powerful filtering, but too costly.
Use it on large problems when checking constraints is cheap

33. Hybrids: #CC, p = 5%, 11%
When p < 0.05: ACx is not effective,ACx-FC are costly,
NIC-FC-FC is OK, NIC-MAC-x is best
When p > 0.10: avoid MAC, stick with FC

34. Hybrids results » Better than,  comparable When to use what
NIC-MAC-Acx
» NIC-FC-FC
» AC-MAC-x
» ACx-FC
NIC-FC-FC
» NIC-MAC-ACx
 AC-MAC-x
CPU time
#CC &
When to use what
p < 0.05
0.05  p < 0.10
0.10  p < 0.15
0.15  p < 0.2
p  0.2 FC
Avoid FC
FC dominates
MAC
MAC dominates
Avoid MAC
NIC
NIC helps
NIC still helps
ACx
ACx useless

35. Hybrids: #CC, p= 15%, 28%
As p increases: MAC deteriorates, NIC becomes expensive, use ACx-FC

36. Hybrids results » Better than,  comparable When to use what
NIC-MAC-Acx
» NIC-FC-FC
» AC-MAC-x
» ACx-FC
NIC-FC-FC
» NIC-MAC-ACx
 AC-MAC-x
ACx-FC
» NIC-MAC-Acx
CPU time
#CC &
When to use what
p < 0.05
0.05  p < 0.10
0.10  p < 0.15
0.15  p < 0.2
p  0.2 FC
Avoid FC
FC dominates
MAC
MAC dominates
Avoid MAC
NIC
NIC helps
NIC still helps
NIC deteriorates
Avoid NIC
ACx
ACx useless
AC-x starts to work
ACx helps

37. Hybrids: #NV At phase transition, NIC and MAC do powerful filtering
but influence of MAC is stronger

38. NIC-MAC-Acx » AC-MAC-x AC-MAC-x » NIC-MAC-ACx
Hybrids: summary
» Better than,  comparable
p < 0.05
0.05  p < 0.10
0.10  p < 0.15
0.15  p < 0.2
p  0.2
NIC-MAC-Acx
» NIC-FC-FC
» AC-MAC-x
» ACx-FC
NIC-FC-FC
» NIC-MAC-ACx
 AC-MAC-x
ACx-FC
» NIC-MAC-Acx
#NV
NIC-MAC-Acx » AC-MAC-x
» NIC-FC-FC » ACx-FC
AC-MAC-x » NIC-MAC-ACx
» ACx-FC » NIC-FC-FC
CPU time
#CC &
When to use what
p < 0.05
0.05  p < 0.10
0.10  p < 0.15
0.15  p < 0.2
p  0.2 FC
Avoid FC
FC dominates
MAC
MAC dominates
Avoid MAC
NIC
NIC helps
NIC still helps
NIC deteriorates
Avoid NIC
ACx
ACx useless
AC-x starts to work
ACx helps

39. 5. Conclusions Preprocessing Look-ahead
AC3 vs. AC2001: AC2001 is not significantly better than AC3, and is not worth the extra data structures
In general, we disagree with Bessière and Régin
NIC-x: powerful filtering, but too costly
Use it on large problems when checking constraints is cheap
Look-ahead
MAC vs. FC: performance depends on constraint probability and tightness. MAC only wins in low p and high t
In general we disagree with results
of Sabin, Freuder, Bessière & Régin

40. Relation to previous work (I)
NIC is better than AC Freuder & Elfe 1996
The instances tested were all with low probability (p < 0.05). In this region, AC is ineffective.
MAC is better than FC Sabin & Freuder 1994
They tested CSPs with low probability (p = ), and relatively high constraint-tightness (t = 0.15 – 0.675). In our study, MAC is effective in this region, but not outside it.
MAC is better than FC Bessière & Régin 1996
The instances tested were also in the region of low probability (p = 0.017, 0.024, 0.074, 0.08, 0.1, 0.12, 0.15), except instance#1 and instance#2, with relative high probability (p = 0.3, and 0.84). But here they test only 2 instances

41. Relation to previous work (II)
Gent & Prosser 2000 questioned validity of previous results on MAC. They concluded that:
in large, sparse CSPs with tight constraints, MAC is winner
in dense CSPs with loose constraints, FC is winner.
Grant 1997 showed that FC is winner on small CSPs with all range of probabilities
All concluded that: “A champion algorithm which perform extremely well on all types of problems does not exist.”
Our characterizations are more thorough and precise.

42. 6. Summary of contributions
Random generator that guarantees solvability
Empirical evaluation of the performance of 7
combinations of preprocessing and look-ahead
Uncovered (restricted) validity conditions of previously
reported results
Summarized best working conditions for preprocessing
and look-ahead algorithms
Developed a Java library with 7 hybrid algorithms

43. 7. Directions for future work
Compare to other, less common filtering algorithms, e.g. SRPC, PC, SAC, Max-RPC Debryune & Bessière 2001
Combining these preprocessing algorithms with intelligent backtrack search algorithms
Validate results on larger CSPs, real-world applications, non-binary
Test and analyze the effect of the topology of constraint networks on the performance of search

44. Acknowledgments Dr. Berthe Y. Choueiry (advisor) Dr. Sebastian Elbaum
Dr. Peter Revesz
Ms. Catherine L. Anderson
Mr. Daniel Buettner
Ms. Deborah Derrick (proof reading)
Mr. Eric Moss
Mr. Lin Xu
Ms. Yaling Zheng
Mr. Hui Zou

45. THANK YOU!