1 Smoothed Analysis of Algorithms: Why The Simplex Method Usually Takes Polynomial Time Shang-Hua Teng Boston University/Akamai Joint work with Daniel Spielman (MIT) Gaussian Perturbation with variance  2

2 Remarkable Algorithms and Heuristics Work well in practice, but Worst case: bad, exponential, contrived. Average case: good, polynomial, meaningful?

3 Random is not typical

4 Smoothed Analysis of Algorithms: worst case max x T(x) average case avg r T(r) smoothed complexity max x avg r T(x+  r)

5 Smoothed Analysis of Algorithms Interpolate between Worst case and Average Case. Consider neighborhood of every input instance If low, have to be unlucky to find bad input instance

6 Complexity Landscape

7 Classical Example: Simplex Method for Linear Programming max z T x s.t. A x  y Worst-Case: exponential Average-Case: polynomial Widely used in practice

8 CarbsProteinFatIronCost 1 slice bread ¢ 1 cup yogurt ¢ 2tsp Peanut Butter ¢ US RDA Minimum Minimize 30 x x x 3 s.t. 30x x x 3  300 5x 1 + 9x 2 + 8x 3  x x x 3  70 10x x 3  100 x 1, x 2, x 3  0 The Diet Problem

9 max z T x s.t. A x  y Max x 1 +x 2 s.t x 1  1 x 2  1 -x 1 - 2x 2  1 Linear Programming

10 Smoothed Analysis of Simplex Method max z T x s.t. A x  y G is Gaussian max z T x s.t. (A +  G) x  y

11 Worst-Case: exponential Average-Case: polynomial Smoothed Complexity: polynomial max z T x s.t. a i T x  ±1, ||a i ||   1 max z T x s.t. (a i +  g i ) T x  ±1 Smoothed Analysis of Simplex Method

12 Perturbation yields Approximation For polytope of good aspect ratio

13 But, combinatorially

14 The Simplex Method

15 Simplex Method (Dantzig, ‘47) Exponential Worst-Case (Klee-Minty ‘72) Avg-Case Analysis (Borgwardt ‘77, Smale ‘82, Haimovich, Adler, Megiddo, Shamir, Karp, Todd) Ellipsoid Method (Khaciyan, ‘79) Interior-Point Method (Karmarkar, ‘84) Randomized Simplex Method (m O(  d) ) (Kalai ‘92, Matousek-Sharir-Welzl ‘92) History of Linear Programming

16 Shadow Vertices

17 Another shadow

18 Shadow vertex pivot rule objective start z

19 Theorem: For every plane, the expected size of the shadow of the perturbed tope is poly(m,d,1/  )

20 Theorem: For every z, two-Phase Algorithm runs in expected time poly(m,d,1/  ) z

21 Vertex on a 1,…,a d maximizes z iff z  cone(a 1,…,a d ) 0 a1a1 a2a2 z z A Local condition for optimality

22 Polar ConvexHull(a 1, a 2, … a m ) Primal a 1 T x  1 a 2 T x  1 … a m T x  1

23

24 max   z  ConvexHull(a 1, a 2,..., a m ) z Polar Linear Program

25 Initial Simplex Opt Simplex

26 Shadow vertex pivot rule

27

28 Count facets by discretizing to N directions, N  

29 Count pairs in different facets Pr Different Facets [] < c/N So, expect c Facets

30 Expect cone of large angle

31 Angle Distance

32 Isolate on one Simplex

33 Integral Formulation

34 Example: For a and b Gaussian distributed points, given that ab intersects x-axis Prob[  <  ] = O(  2 )  a b

35  a b

36 a b

37 a b

38 a b

39 a b

40 a b

41 Claim: For  <   P  <  2  a b

42 Change of variables  a b z u v da db = |(u+v)sin(  du dv dz d 

43 Analysis: For  <   P  <  2 Slight change in  has little effect on i for all but very rare u,v,z   

44 Distance: Gaussian distributed corners a1a1 a2a2 a3a3 p

45 Idea: fix by perturbation

46 Trickier in 3d

47 Future Research – Simplex Method Smoothed analysis of other pivot rules Analysis under relative perturbations. Trace solutions as un-perturb. Strongly polynomial algorithm for linear programming?

48 A Theory Closer to Practice Optimization algorithms and heuristics, such as Newton’s Method, Conjugate Gradient, Simulated Annealing, Differential Evolution, etc. Computational Geometry, Scientific Computing and Numerical Analysis Heuristics solving instances of NP-Hard problems. Discrete problems? Shrink intuition gap between theory and practice.