Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, Foundations of Constraint Processing CSCE421/821, Spring 2008: Berthe Y. Choueiry (Shu-we-ri) Avery Hall, Room 123B Tel: +1(402) Evaluation of (Deterministic) BT Search Algorithms

Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, Outline Evaluation of (deterministic) BT search algorithms [Dechter, 6.6.2] –CSP parameters –Comparison criteria –Theoretical evaluations –Empirical evaluations

Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, CSP parameters Number of variables: n Domain size: a, d Constraint tightness: t = |forbidden tuples| / | all tuples | Proportion of constraints (a.k.a., constraint density, constraint probability): p 1 = e / e max, e is nbr of constraints

Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, Comparison criteria 1.Number of nodes visited (#NV) Every time you call label 2.Number of constraint check (#CC) Every time you call check(i,j) 3.CPU time Be as honest and consistent as possible 4.Number of Backtracks (#BT) Every un-assignment of a variable in unlabel 5.Some specific criterion for assessing the quality of the improvement proposed Presentation of values: Descriptive statistics of criterion: average, median, mode, max, min (qualified) run-time distribution Solution-quality distribution

Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, Theoretical evaluations Comparing NV and/or CC Common assumptions: –for finding all solutions –static orderings

Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, Empirical evaluation: data sets Use real-world data (anecdotal evidence) Use benchmarks –csplib.org –Solver competition benchmarks Use randomly generated problems –Various models of random generators –Guaranteed with a solution –Uniform or structured

Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, Empirical evaluations: random problems Various models exist (use Model B) –Models A, B, C, E, F, etc. Vary parameters: –Number of variables: n –Domain size: a, d –Constraint tightness: t = |forbidden tuples| / | all tuples | –Proportion of constraints (a.k.a., constraint density, constraint probability): p 1 = e / e max Issues: –Uniformity –Difficulty (phase transition) –Solvability of instances (for incomplete search techniques)

Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, Model B 1.Input: n, a, t, p1 2.Generate n nodes 3.Generate a list of n.(n-1)/2 tuples of all combinations of 2 nodes 4.Choose e elements from above list as constraints to between the n nodes 5.If the graph is not connected, throw away, go back to step 4, else proceed 6.Generate a list of a 2 tuples of all combinations of 2 values 7.For each constraint, choose randomly a number of tuples from the list to guarantee tightness t for the constraint

Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, Phase transition [Cheeseman et al. ‘91] Cost of solving Mostly solvable problems Mostly un-solvable problems Order parameter Critical value of order parameter Significant increase of cost around critical value In CSPs, order parameter is constraint tightness & ratio Algorithms compared around phase transition

Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, 2008 Tests Fix n, a, p 1 and –Vary t in {0.1, 0.2, …,0.9} Fix n, a, t and –Vary p 1 in {0.1, 0.2, …,0.9} For each data point (for each value of t/p 1 ) –Generate (at least) 50 instances –Store all instances Make measurements –#NV, #CC, CPU time, #messages, etc.

Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, 2008 Comparing two algorithms A 1 and A 2 Store all measurements in Excel Use Excel, R, SAS, etc. for statistical measurements Use the t-test, paired test Comparing measurements –A 1, A 2 a significantly different Comparing ln measurements –A 1 is significantly better than A 2 #CCln(#CC) A1A1 A2A2 A1A1 A2A2 i1i …… i2i2 … i3i3 … i 50

Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, 2008 t-test in Excel Using ln values –p  ttest(array1,array2,tails,type) tails=1 or 2 type  1 (paired) –t  tinv(p,df) degree of freedom = #instances – 2

Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, 2008 t-test with 95% confidence One-tailed test –Interested in direction of change –When t > 1.645, A 1 is larger than A 2 –When t  , A 2 is larger than A 1 –When  t  1.645, A 1 and A 2 do not differ significantly –|t|=1.645 corresponds to p=0.05 for a one-tailed test Two-tailed test –Although it tells direction, not as accurate as the one-tailed test –When t > 1.96, A 1 is larger than A 2 –When t  -1.96, A 2 is larger than A 1 –When  t  1.96, A 1 and A 2 do not differ significantly –|t|=1.96 corresponds to p=0.05 for a two-tailed test p=0.05 is a US Supreme Court ruling: any statistical analysis needs to be significant at the 0.05 level to be admitted in court

Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, 2008 Computing the 95% confidence interval The t test can be used to test the equality of the means of two normal populations with unknown, but equal, variance. We usually use the t-test Assumptions Normal distribution of data Sampling distributions of the mean approaches a uniform distribution (holds when #instances  30) Equality of variances Sampling distribution: distribution calculated from all possible samples of a given size drawn from a given population

Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, 2008 Alternatives to the t test To relax the normality assumption, a non-parametric alternative to the t test can be used, and the usual choices are:non-parametric –for independent samples, the Mann-Whitney U testMann-Whitney U test –for related samples, either the binomial test or the Wilcoxon signed-rank testbinomial testWilcoxon signed-rank test To test the equality of the means of more than two normal populations, an Analysis of Variance can be performedAnalysis of Variance To test the equality of the means of two normal populations with known variance, a Z-test can be performedZ-test

Foundations of Constraint Processing, Spring 2008 Evaluation to BT SearchApril 16, 2008 Alerts For choosing the value of t in general, check For a sound statistical analysis, consult the Help Desk of the Department of Statistics at UNL, held at least twice a week at Avery Hall. Acknowledgments: Makram Geha, PhD candidate, Department of Statistics. All errors are mine..