1. DIMACS EAA:ICS 2006/04/041 Franc Brglez Raleigh, NC, USA Software Analysis via Data Analysis Xiao Yu Li Seattle, WA, USA Matthias F. Stallmann, 2006/04/04, DIMACS EAA:ISC planning meeting Based on joint work with:

2. DIMACS EAA:ICS 2006/04/042 Software versus Algorithms o Problems are NP-hard; require heuristics or (worst case) exponential algorithms. o Simple algorithms must be compared with cplex and other ILP solvers metaheuristics (SA, GA, “particle swarm”, “ants…”, etc.) with lots of adjustable parameters o Want “black box” comparison with (in many cases) no prior understanding of (some of) the algorithms o Problems are NP-hard; require heuristics or (worst case) exponential algorithms. o Simple algorithms must be compared with cplex and other ILP solvers metaheuristics (SA, GA, “particle swarm”, “ants…”, etc.) with lots of adjustable parameters o Want “black box” comparison with (in many cases) no prior understanding of (some of) the algorithms

3. DIMACS EAA:ICS 2006/04/043 Data Presentations based on CPLEX runs for (permutations of) a single instance o Descriptive Statistics mean / median / stdev - 600.4 / 25.3 / 1767.3 o Histogram o Percent solved o Descriptive Statistics mean / median / stdev - 600.4 / 25.3 / 1767.3 o Histogram o Percent solved

4. DIMACS EAA:ICS 2006/04/044 Stretching the Truth or making it clearer? more information here, but we need to look carefully

5. DIMACS EAA:ICS 2006/04/045 A more “normal” distribution CPLEX with different settings under the same conditions

6. DIMACS EAA:ICS 2006/04/046 uf250..87: QT2/QT1 vs UW2/UW1 (1) runtime (seconds) solvability What about random 3-SAT instances? exp. d. 16.7/17.2

7. DIMACS EAA:ICS 2006/04/047 uf250..87: QT2/QT1 vs UW2/UW1 (2) runtime (seconds) solvability exp. d. 12.3/12.5 exp. d. 16.7/17.2 UW1 performs the same as QT1 (t-test: t = 1.88 > 1.97)

8. DIMACS EAA:ICS 2006/04/048 uf250..87: QT2/QT1 vs UW2/UW1 (3) runtime (seconds) solvability exp. d. 12.3/12.5 exp. d. 16.7/17.2 exp. d. 0.39/0.29 UW2 outperforms UW1 by a factor of 31... UW1 performs the same as QT1 (t-test: t = 1.88 > 1.97)

9. DIMACS EAA:ICS 2006/04/049 uf250..87: QT2/QT1 vs UW2/UW1 (4) exp. d. 0.31/0.28 exp. d. QT2 outperforms UW2 slightly (t-test: t = 2.24 > 1.97) UW2 outperforms UW1 by a factor of 31... UW1 performs the same as QT1 (t-test: t = 1.88 > 1.97) runtime (seconds) solvability exp. d. 16.7/17.2 exp. d. 12.3/12.5 exp. d. 0.39/0.29 exp. d. 0.21/0.19

10. DIMACS EAA:ICS 2006/04/0410 Sources of Data Distributions first, 100 random instances (SAT) median4.8 mean21.2 stdev42.9 Heavy tail

11. DIMACS EAA:ICS 2006/04/0411 Not all instances are equal: here’s an easy one (128 permutations + original) median4.4 mean6.7 stdev6.5 Exponential

12. DIMACS EAA:ICS 2006/04/0412 …and here’s a harder one median84.8 mean126.7 stdev117.6 Exponential

13. DIMACS EAA:ICS 2006/04/0413 Another wrinkle: stochastic search first, same seed and 32 permuted instances + original median42484 mean62457 stdev86551 slightly worse than Exponential number of flips

14. DIMACS EAA:ICS 2006/04/0414 …versus 33 different seeds; same distribution? median36637 mean50185 stdev83226 slightly worse than Exponential number of flips

15. DIMACS EAA:ICS 2006/04/0415 Things get strange when the solver is not completely stochastic (e.g. B&B with stochastic search) Bi-modal: Stochastic search either finds optimum at root, or at first branch. Lower bound “finds” optimum at root. No randomness in LB method.

16. DIMACS EAA:ICS 2006/04/0416 Lower bound method is extremely sensitive to input ordering Heavy tail (and one instance times out): Lower bound method finds optimum at root or in an early branch only if input order is “friendly”.

17. DIMACS EAA:ICS 2006/04/0417 Another Data Analysis Application: Instance Profiling o sao2.b is very easy to solve (variables easy to distinguish) o e64.b is very difficult (lots of variables occur equally often)