1. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 1 High-Contrast Algorithm Behavior Observation, Hypothesis, and Experimental Design Matthias F. Stallmann and Franc Brglez

2. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 2 Summary Two competing algorithms: int-dual - all-integer dual simplex old idea, only limited success cplex - industrial strength optimizer Restricted domain (logic minimization). Extreme differences across instances: time out for one algorithm vs. seconds for the other. Want to “profile” the instances, or at least explain the differences.

3. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 3 Outline  Observation in a limited domain  Initial experiments: design  Initial experiments: results  A hypothesis  More experimental evidence  Theoretical explanation  Summary and future work algorithms treated as black boxes int-dualcplex

4. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 4 a typical problem instance x1 + x4 >= 1 x2 + x3 + x4 >= 1 x1 + x2 + x4 >= 1 x1 – x4 >= 0 x2 – x4 >= 0 x3 – x4 >= 0 minimize x1 + x2 + x3 + x4 subject to x1 x2 x3 x4 constraint matrixpictorial representation

5. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 5 A pattern? algN/33med.meanstdev Int- dual 2915.9N/a cplex0N/a Benchmark: e64.b (logic synthesis) -- a “canonical” picture # succ. completions, one hour original + 32 random row/col permutations dedicated fast processor

6. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 6 A pattern: Less pronounced algN/33med.meanstdev Int-dual3314.621.115.1 cplex333.96.06.1 Benchmark: rot.b

7. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 7 A pattern: Not much of one algN/33med.meanstdev Int-dual0N/a cplex333.63.50.9 Benchmark: max1024

8. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 8 The pattern int-dualcplex Benchmark: e64.b Benchmark: max1024

9. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 9 Process for generating pictures crossing minimization (moves nonzeros close to the diagonal) may still look “random”

10. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 10 this is as good as it gets... This is max1024 after crossing minimization

11. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 11 Conversion to block structure does not improve performance crossing minimization int-dual performance: not necessarily improved underlying block structure... random permutations } better aggregate performance for int-dual e64 as posted: 27.5after minimization: 55.0 recall that median = 15.9

12. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 12 Basic idea  More underlying structure => better for int-dual  Less underlying structure => better for cplex Why is this interesting? e.g., optimum for e64.b was unknown prior to int-dual  What do we mean by structure?  More precise (and testable) hypothesis?

13. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 13 Outline  Observation in a limited domain  Initial experiments: design  Initial experiments: results  A hypothesis  More experimental evidence  Theoretical explanation  Summary and future work

14. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 14 From observation to controlled experiment Behavior “in the wild”: algorithms on industrial instances Behavior “in captivity”: algorithms on carefully designed instances

15. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 15 Instances with pure blocks Create K copies of a small instance, arrange on diagonal, permute rows and columns randomly 32 times to obtain a class of equivalent instances.

16. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 16 Construction of blocks with added rows xxxx two pure blocks add new row(s) with random nonzeros below both blocks xxxx repeat xxx x apply same process to “new” blocks Create 32 instances with different random choices of the nonzeros; randomly permute each one row factor = # of rows added each step

17. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 17 Blocks used in results reported here  st15 - instance based on Steiner triples (15 vars, 35 const)  maincont - small logic synthesis instances (61 vars, 50 const)  f51m - slightly larger logic synthesis instance (175 vars, 187 const)

18. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 18 Outline  Observation in a limited domain  Initial experiments: design  Initial experiments: results  A hypothesis  More experimental evidence  Theoretical explanation  Summary and future work

19. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 19 Details and reporting  Class of 32 random related instances  Intel(R) Xeon(TM) CPU 3.20GHz, 2 GB memory, 2048 KB cache  Report runtime only (seconds, time out = 600)  Here we report median only, but note number of successful completions, if not 32

20. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 20 results for st15, pure blocks blocks124816 int-dual 0.10.31.04.129.8 cplex 0.020.9> 10 hours

21. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 21 results for maincont, pure blocks blocks12481632 int-dual < 0.1 0.10.31.6 cplex < 0.1 0.20.5> 600 29/32 completions > 600 between 24 and 28 blocks

22. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 22 results for f51m, pure blocks blocks124816 int-dual 0.10.42.112.465.7 cplex 0.359.0> 600 int-dual is roughly between O(n^2) and O(n^3) cplex is wildly exponential

23. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 23 added rows? blank = neither dominates, X = neither finishes in 1 hour I = int-dual dominates, C = cplex dominates next page...

24. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 24 results for maincont, added rows row fact. 12345678 N/32 271451 id 0.1 0.20.72.7>600 cplex 0.20.30.41.49.620.971.190.0 N/32 312331 3027 8 blocks with increasing row factor Wild distributions! mean = 59.9 stdev = 140.5

25. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 25 this is not going well not much hope of a statistically meaningful comparison between int-dual and cplex. too many variables: will anything hold up not clear what to measure domain already very limited...

26. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 26 all is not lost: let’s stick with what we know  int-dual performs well on pure blocks  cplex performs badly on pure blocks  int-dual gets worse as more random nonzeros are added (has been observed for columns, too) id 0.1 0.20.72.7>600

27. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 27 Outline  Observation in a limited domain  Initial experiments: design  Initial experiments: results  A hypothesis  More experimental evidence  Theoretical explanation  Summary and future work

28. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 28 A viable hypothesis  int-dual runtime is predictable on pure-blocks instances and grows polynomially with increasing number of blocks  cplex runtime on pure-blocks instances becomes erratic as soon as an instance-dependent threshold is reached  int-dual runtime increases and/or becomes erratic as the number of nonzeros added to a pure-blocks instance is increased erratic = “heavy tail” distribution (stdev >> mean >> median) or time outs and overflows

29. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 29 why this is good  not dependent on comparison between the algorithms  not dependent on statistics  extreme contrasts are what we observed initially -- so we need to deal with them  appears to be true in a variety of circumstances  even though instances are artificial, may explain why int-dual has limited use

30. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 30 Outline  Observation in a limited domain  Initial experiments: design  Initial experiments: results  A hypothesis  More experimental evidence  Theoretical explanation  Summary and future work

31. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 31 f51m, added rows row fact. 12345678 N/32 2322191215107 id 3.872.652.6210.2>600 cplex >600 260.270.217.411.09.36.4 192331N/32 distributions that are not erratic 3 blocks, increasing row factor

32. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 32 Outline  Observation in a limited domain  Initial experiments: design  Initial experiments: results  A hypothesis  More experimental evidence  Theoretical explanation  Summary and future work

33. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 33 All-Integer Dual Simplex vertex cover of a triangle x2x3 x1 0111 –1 0 0 0 x1x3x2 e1 e2 e3 x1 + x2 >= 1 min x1 + x2 + x3 –e1= –x1 – x2 tableau

34. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 34 a pivot step 0111 –1 0 0 0 x1x3x2 e1 e2 e3 choose constraint to satisfy choose variable to do the trick –1101 1 10 0 1 0 x3x2 e2 e3 swap them now x1=1, e1=0 e1 x1

35. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 35 example (continued) –1101 1 10 0 1 0 101 0 1 1–2 10–11 and another, but this one involves division by –2... another pivot...

36. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 36 oops: solution is not integral –1101 0 1 1–2 10–11 –3/21/2–1/2 1/2–1/23/21/2 –1/21 1/2–1/2–3/21/2 and we get fractions. The LP solution is x1 = 1/2, x2 = 1/2, x3 = 1/2

37. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 37 Maintaining an integer tableau –1101 0 1 1–2 10–11 0 x2 x1 x3e1 ct e2 e3 –2001 101–1 111–2 0–1 1 110 x2 x1 cte1 x3 e2 e3 add a new constraint that must be met by all integer solutions x1 = x3 = 1, cost = 2

38. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 38 int-dual: another look 0111 –1 0 0 0 101 1 10 0 1 0 0 in pivot row means no change in pivot column 0 in pivot column means no change in pivot row

39. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 39 int-dual on “pure blocks” 111 000 111 000 111 000 000 222 000 222 000 222 2 00 2 0 2 0 11 0 1 0 2 00 2 0 2 0 11 0 1 0 0 11 0 1 0 2 00 2 0 2 this phenomenon is not affected by random permutation element of second block not changed by pivot in first block

40. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 40 why does int-dual work well on pure blocks?  because each block is solved independently  progress is made in whatever block contains the pivot One out of three parts (of the hypothesis) explained

41. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 41 part 2: cplex does badly on pure blocks branching tree for one block tree for two blocks Note: this is a gross oversimplification cplex is a “branch and bound” algorithm x y 0 0 1 1

42. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 42 Summary  Some observations about algorithm behavior on industrial benchmarks  Hypothesis specializes observations to a limited, verifiable domain  Extensive experiments consistent with hypothesis  Partial theoretical explanation for hypothesis  Aside: int-dual dates to 1960 ’ s; experiments in early 1970 ’ s deemed it good for small instances of set cover (but impractical otherwise)

43. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 43 Future work  better understanding of both algorithms to explain what’s going on  “pure” branch and bound instead of cplex  experiments to explore variations on variables (how to add rows, columns too, add –1 entries)  observations in other domains (particular where there has been success with int-dual)  other approaches to claiming one algorithm better than another where distributions are crazy  can we learn from outliers, i.e., easy and hard instances of a class

44. Stallmann/Brglez, High contrast… ExpCS, 2007/06/14, San Diego 44 Thanks...  Xiao Yu Li, now at Amazon, started us down this path  Eric Sills, NCSU High Performance Computing Center, helped with hardware and software availability Source code for software and (eventually) instance classes and results at http://people.engr.ncsu.edu/mfms/Software