1. Foundations of Constraint Processing Evaluation to BT Search 1 Foundations of Constraint Processing CSCE421/821, Spring 2011 www.cse.unl.edu/~choueiry/S11-421-821/ All questions to cse421@cse.unl.edu Berthe Y. Choueiry (Shu-we-ri) Avery Hall, Room 360 choueiry@cse.unl.edu Tel: +1(402)472-5444 Evaluation of (Deterministic) BT Search Algorithms

2. Foundations of Constraint Processing Evaluation to BT Search 2 Outline Evaluation of (deterministic) BT search algorithms [Dechter, 6.6.2] –CSP parameters –Comparison criteria –Theoretical evaluations –Empirical evaluations

3. Foundations of Constraint Processing Evaluation to BT Search 3 CSP parameters Binary:  n,a,p 1,t  ; Non-binary:  n,a,p 1,k,t  Number of variables: n Domain size: a, d Degree of a variable: deg Arity of the constraints: k Constraint tightness: Proportion of constraints (a.k.a., constraint density, constraint probability) p 1 = e / e max, e is number of constraints

4. Foundations of Constraint Processing Evaluation to BT Search 4 Comparison criteria 1.Number of nodes visited (#NV) Every time you call label 2.Number of constraint check (#CC) Every time you call check(i,j) 3.CPU time Be as honest and consistent as possible 4.Number of Backtracks (#BT) Every un-assignment of a variable in unlabel 5.Some specific criterion for assessing the quality of the improvement proposed Presentation of values: Descriptive statistics of criterion: average, median, mode, max, min (qualified) run-time distribution Solution-quality distribution

5. Foundations of Constraint Processing Evaluation to BT Search 5 Theoretical evaluations Comparing NV and/or CC Common assumptions: – for finding all solutions – static/same orderings

6. Foundations of Constraint Processing Evaluation to BT Search 6 Empirical evaluation: data sets Use real-world data (anecdotal evidence) Use benchmarks –csplib.org –Solver competition benchmarks Use randomly generated problems –Various models of random generators –Guaranteed with a solution –Uniform or structured

7. Foundations of Constraint Processing Evaluation to BT Search 7 Empirical evaluations: random problems Various models exist (use Model B) –Models A, B, C, E, F, etc. Vary parameters: –Number of variables: n –Domain size: a, d –Constraint tightness: t = |forbidden tuples| / | all tuples | –Proportion of constraints (a.k.a., constraint density, constraint probability): p 1 = e / e max Issues: –Uniformity –Difficulty (phase transition) –Solvability of instances (for incomplete search techniques)

8. Foundations of Constraint Processing Evaluation to BT Search 8 Model B 1.Input: n, a, t, p1 2.Generate n nodes 3.Generate a list of n.(n-1)/2 tuples of all combinations of 2 nodes 4.Choose e elements from above list as constraints to between the n nodes 5.If the graph is not connected, throw away, go back to step 4, else proceed 6.Generate a list of a 2 tuples of all combinations of 2 values 7.For each constraint, choose randomly a number of tuples from the list to guarantee tightness t for the constraint

9. Foundations of Constraint Processing Evaluation to BT Search 9 Phase transition [Cheeseman et al. ‘91] Cost of solving Mostly solvable problems Mostly un-solvable problems Order parameter Critical value of order parameter Significant increase of cost around critical value In CSPs, order parameter is constraint tightness & ratio Algorithms compared around phase transition

10. Foundations of Constraint Processing Evaluation to BT Search Tests Fix n, a, p 1 and –Vary t in {0.1, 0.2, …,0.9} Fix n, a, t and –Vary p 1 in {0.1, 0.2, …,0.9} For each data point (for each value of t/p 1 ) –Generate (at least) 50 instances –Store all instances Make measurements –#NV, #CC, CPU time, #messages, etc.

11. Foundations of Constraint Processing Evaluation to BT Search Comparing two algorithms A 1 and A 2 Store all measurements in Excel Use Excel, R, SAS, etc. for statistical measurements Use the t-test, paired test Comparing measurements –A 1, A 2 a significantly different Comparing ln measurements –A 1 is significantly better than A 2 For Excel: Microsoft button, Excel Options, Adds in, Analysis ToolPak, Go, check the box for Analysis ToolPak, Go. Intall… #CCln(#CC) A1A1 A2A2 A1A1 A2A2 i1i1 100200…… i2i2 … i3i3 … i 50

12. Foundations of Constraint Processing Evaluation to BT Search t-test in Excel Using ln values –p  ttest(array1,array2,tails,type) tails=1 or 2 type  1 (paired) –t  tinv(p,df) degree of freedom = #instances – 2

13. Foundations of Constraint Processing Evaluation to BT Search t-test with 95% confidence One-tailed test –Interested in direction of change –When t > 1.645, A 1 is larger than A 2 –When t  -1.645, A 2 is larger than A 1 –When -1.645  t  1.645, A 1 and A 2 do not differ significantly –|t|=1.645 corresponds to p=0.05 for a one-tailed test Two-tailed test –Although it tells direction, not as accurate as the one-tailed test –When t > 1.96, A 1 is larger than A 2 –When t  -1.96, A 2 is larger than A 1 –When -1.96  t  1.96, A 1 and A 2 do not differ significantly –|t|=1.96 corresponds to p=0.05 for a two-tailed test p=0.05 is a US Supreme Court ruling: any statistical analysis needs to be significant at the 0.05 level to be admitted in court

14. Foundations of Constraint Processing Evaluation to BT Search Computing the 95% confidence interval The t test can be used to test the equality of the means of two normal populations with unknown, but equal, variance. We usually use the t-test Assumptions Normal distribution of data Sampling distributions of the mean approaches a uniform distribution (holds when #instances  30) Equality of variances Sampling distribution: distribution calculated from all possible samples of a given size drawn from a given population

15. Foundations of Constraint Processing Evaluation to BT Search Alternatives to the t test To relax the normality assumption, a non-parametric alternative to the t test can be used, and the usual choices are:non-parametric –for independent samples, the Mann-Whitney U testMann-Whitney U test –for related samples, either the binomial test or the Wilcoxon signed-rank testbinomial testWilcoxon signed-rank test To test the equality of the means of more than two normal populations, an Analysis of Variance can be performedAnalysis of Variance To test the equality of the means of two normal populations with known variance, a Z-test can be performedZ-test

16. Foundations of Constraint Processing Evaluation to BT Search Alerts For choosing the value of t in general, check http://www.socr.ucla.edu/Applets.dir/T-table.html http://www.socr.ucla.edu/Applets.dir/T-table.html For a sound statistical analysis –consult the Help Desk of the Department of Statistics at UNL –held at least twice a week at Avery Hall. Acknowledgments: Dr. Makram Geha, Department of Statistics @ UNL. All errors are mine..