1. Discrete Optimization Lecture #2 2008/2/271Shi-Chung Chang, NTUEE, GIIE, GICE Last Time 1.Course Overview » A taxonomy of optimization problems » Course introduction, requirements and schedule 2.Some Basics of Optimization » Local and global optima » Feasibility » Convexity » Convex programming 3.Algorithms and Complexity »Problems, algorithms, and complexity

2. Discrete Optimization Lecture #2 Last Time: Reading Assignments 1.Chapter 1 and the Appendix of [PaS82] 2.Chapter 1 of [GaJ79] [PaS82] C. H. Papadimitriou, K. Steiglitz, Combinatorial Optimization:Algorithms and Complexity, Prentice Hall, 1982. [GaJ79] M. R. Garey and D. S. Johnson, Computers and Intractability: A Guide to the Theory of NP-Completeness, Series of Books in the Mathematical Sciences, 1979. 2008/2/272Shi-Chung Chang, NTUEE, GIIE, GICE

3. Today 1.Algorithms and Complexity » Problems, algorithms, and complexity » Polynomial time algorithms » Intractability » NP-complete problems 2.Basic Properties of Linear Programming » Forms of LP » Basic feasible solutions » Geometry of LP 3.The Revised Simplex Method Reading Assignments 1. Sections 2.1-2.4 of [PaS82] 2. Chapter 1 of [GaJ79] 3. Section 3.8 of [Lue84] [Lue84] David G. Luenberger, Linear & Nonlinear Programming 2e, Addison Wesley, 1984. 2008/2/273Shi-Chung Chang, NTUEE, GIIE, GICE

4. §1.4 Algorithms and complexity For a given (optimization) problem, Questions: 1)how hard is the problem. 2)does there exist an efficient solution algorithm?  Computational complexity analysis (a worst case analysis) 2008/2/274Shi-Chung Chang, NTUEE, GIIE, GICE

5. § 1.4.1 Problems, Algorithm and Complexity Problem: a general question to be answered, usually possessing several unspecified parameters. Example: A Linear programming problem subject to A problem is described by giving (i)a general description of its parameters (ii)A statement of what properties the answer should satisfy 2008/2/275Shi-Chung Chang, NTUEE, GIIE, GICE

6. Example Traveling salesman problem Parameters: set of cities distance solution: an ordered sequence that minimizes 2008/2/276Shi-Chung Chang, NTUEE, GIIE, GICE

7. Measuring Algorithmic Efficiency Computation Time as the measure  Time Complexity Function for an algorithm: express its time requirements by giving the largest amount of time needed by the algorithm to solve a problem instance.  a function of the “size” of the instance Example size indices of a TSP # of cities # of interconnections 2008/2/277Shi-Chung Chang, NTUEE, GIIE, GICE

8. §1. 4. 2 Polynomial Time Algorithms and Intractable Problems – A time complexity function is if for all n ≥ 0.  Algorithm An instance of a problem: a problem with all parameters specified E xample Polynomial time algorithm: Time complexity fcn. for some polynomial function p( ･ ). Exponential time algorithm That cannot be bounded by (may not be exponential, e.g. 5 6 8 3 910 2008/2/278Shi-Chung Chang, NTUEE, GIIE, GICE

9. Algorithms: General, step-by-step procedures for solving problems Solve: An algorithm solves a problem  if that algorithm can be applied to any instance I of  and produce a solution to that instance I Example: A TSP solution algorithm step 1: Generate all the feasible paths step 2: Compare the distances among paths and save the shortest one  m! evaluations and comparisons for an m city instance 20! > Assume evaluations/sec.  >31 years  interested in finding the most “efficient” algorithm How to measure algorithmic efficiency? 2008/2/279Shi-Chung Chang, NTUEE, GIIE, GICE

10. Comparisons of the Two Classes of Algorithms Time Complexity function Size 102030405060.00001 Second.00002 second.00003 second.00004 second.00005 second.00006 second.0001 second.0004 second.0009 second.0016 second.0025 second.0036 second.001 second.008 second.027 second.064 second.125 second.216 second.1 second 3.2 second 24.3 second 1.7 minutes 5.2 minutes 13.0 minutes.001 second 1.0 second 17.9 minutes 12.7 days 35.7 years 366 centuries.059 second 58 minutes 6.5 years 3855 centuries 2x centuries 1.3x centuries 2008/2/2710Shi-Chung Chang, NTUEE, GIIE, GICE

11. Size of Largest Problem Instance Solvable in 1 Hour Time Complexity function With present Computer With computer 100 times faster With computer 1000 times faster 1001000 1031.6 4.6410 2.53.98 +6.64 +9.97 +4.19 +6.29 2008/2/2711Shi-Chung Chang, NTUEE, GIIE, GICE

12. 1.An algorithm is not a good algorithm if its complexity is exponential 2. A problem is not “well-solved” until a polynomial time algorithm is known for it 3. A problem is intractable if it is so hard that no polynomial time algorithm can possibly solve it Note: Time complexity is a wont case measure Exponential algorithms may be efficient for specific applications Algorithm and Problem Intractibility 2008/2/2712Shi-Chung Chang, NTUEE, GIIE, GICE

13. Examples of Intractable problems 1) Hilbert’s tenth problem:  An integer solution ? 2) “Intractable problems in Control Theory” C.H. Papadimitriou and J. Tsitsiklis, SIAM J. control and Optimization, July, 1986 PP.639-654 2008/2/2713Shi-Chung Chang, NTUEE, GIIE, GICE

14. §1.4.4 P and NP Problem, NP- Completeness deterministic algorithms the result of every operation is uniquely defined e.g. computer programs non-deterministic algorithms Results of operations can be any of a set of alternatives Example: Shopping list A shopping list can be viewed as a very simple nondeterministic algorithm. Every item on the list is a directive to find the indicated product, but the order in which to find them is not indicated.  P-class of problems: the set of all problems which can be solved by deterministic algorithms in polynomial time NP-Class of problems the set of all problems which can be solved by non-deterministic algorithms in polynomial time 2008/2/2714Shi-Chung Chang, NTUEE, GIIE, GICE

15. NP-Completeness A problem  is NP-complete if  NP and every problem in NP can be reduced to  through polynomial time transformations (may not be deterministic) e.g. The satisfiability problem INSTANCE:Set U of variables, collection C of clauses over U. Question:Is there a satisfying truth assignment for C?  All NP-complete problems are equally difficult 2008/2/2715Shi-Chung Chang, NTUEE, GIIE, GICE

16. Facts: (1) can be solved in polynomial time by a deterministic algorithm => NP=P (2) Q: NP problems intractable?  P=NP? no answer so far! Usually, if a problem is proven to be NP-complete =>give up finding optimal solution algorithm for it =>heuristics, sub-optimality or approximations. 2008/2/2716Shi-Chung Chang, NTUEE, GIIE, GICE

17. Chapter 2 The Simplex algorithm §2.1 Basic Properties of LP §2.1.1 Forms of LP General LP(GLP) subject to (equalities) ( inequalities) 2008/2/2717Shi-Chung Chang, NTUEE, GIIE, GICE

18. Proof: G=>S 1) ( for ) 2) Canonical form( C L P )Standard Form( S L P ) Three forms are equivalent ! ! = > need to stud y onl y one form 2008/2/2718Shi-Chung Chang, NTUEE, GIIE, GICE

19. §2.1.2 Basic Feasible Solutions Goal: mathematically defines the concept of looking for the optimum among vertices Consider SLP = subject to A:m x n (m<n) Q: m ≥ n? Assume:  (A)=m A={ } B={ }  (B)=m I.e. Linearly indep. => is a basic solution if If a basic solution is also feasible, i.e. then it is a basic feasible solution (bfs) 2008/2/2719Shi-Chung Chang, NTUEE, GIIE, GICE

20. Example min[0 2 0 1 0 0 5] subject to and  B = a bfs ‚ B= = > basic not feasible 2008/2/2720Shi-Chung Chang, NTUEE, GIIE, GICE

21. The fundamental theorem of LP Let and (i) If (ii) If optimal feasible solution => optimal bfs. 2008/2/2721Shi-Chung Chang, NTUEE, GIIE, GICE

22. Proof: (i) Let => Case1 are linearly indep. => p ≤ m if p=m => is a bfs if p can find from (linearly indep) => is a basic (degenerate) solution to case2 are linearly dependent => such that 2008/2/2722Shi-Chung Chang, NTUEE, GIIE, GICE

23. Set Let otherwise (ii)Almost the same as ( i ) Read by yourself [Lue 8 4,p.2 0] 2008/2/2723Shi-Chung Chang, NTUEE, GIIE, GICE

24. Implications of the Theorem (1) need only consider bfses for optimum (2) A:m x n => # of bfses ≤ # of bs ≤ Q: How to search efficiently? Next, geometric interpretation of properties of LP.d let The fundamental theorem of LP (i)IF (ii) If optimal feasible solution => optimal bfs. 2008/2/2724Shi-Chung Chang, NTUEE, GIIE, GICE

25. Polytopes and LP 1. A polytope as the convex hull of a finite set of points Q: relation to D.O.? => can use a convex region to include a discrete region => can find a lower bound of the minimum 2. A polytope as the bounded intersection of half spaces => use linear inequalities to represent a polytope 3. 1) = m<n characterizes a polytope in 2) Any convex polytope can be alternatively viewed as the feasible region of of an LP via a simple transformation. (i.e. can be represented by a set of linear ineqs.) 2008/2/2725Shi-Chung Chang, NTUEE, GIIE, GICE

26. we now relate the vertex of a polytope to the bfs of the conesponding LP. vertex (or extreme point) => Let F be a convex set and IF it must hold that,then is called a vertex or an extreme point. 2008/2/2726Shi-Chung Chang, NTUEE, GIIE, GICE

27. §2.1.3 Geometry of LP Hyperplane: ? Closed half space: ? Ploytope: – A bounded and nonempty intersection of a finite number of half spaces – 3 kinds of faces Example: Facet dim. n-1 Vertex dim. 0 Edgedim. 1 Z (001) (010) y x (100) 2008/2/2727Shi-Chung Chang, NTUEE, GIIE, GICE

28. Convex Hull = Example v1 v2 v3 v4 v5 2008/2/2728Shi-Chung Chang, NTUEE, GIIE, GICE

29. Equivalence of Vertices and bfses Theorem Let A: m x n, =m<n, and F= X is a vertex of F  x is a bfs of Proof: “=>” Let x be a vertex of F and assume that and for i > k => Show that are linearly indep. : 2008/2/2729Shi-Chung Chang, NTUEE, GIIE, GICE

30. Assume liner dependence => such that set, => such that => => a contradiction to the fact that x is a vertex “<=“ try it by yourself 2008/2/2730Shi-Chung Chang, NTUEE, GIIE, GICE

31. §2.2 The Revised Simplex Method The simplex method for LP : G.B. Dantzig 1951 Key idea: Phase1: find a bfs of Note that an LP may not have a solution Example min s.t. 2008/2/2731Shi-Chung Chang, NTUEE, GIIE, GICE

32. phase 2 Allow one of the zero components of the bfs to become positive and force one of the original positive components to become zero. => How to pick “entering” and “leaving” component Cost Traditional form of the simplex method: Tableau => read by yourself Here we consider matrix form for conciseness of presentation and later developments. 2008/2/2732Shi-Chung Chang, NTUEE, GIIE, GICE

33. SLP: s.t. Assume P(A)=m. Partition A=[B:D] B: m linearly indep. Columns of A(Assume the first m cols.) (SLP) subject to 2008/2/2733Shi-Chung Chang, NTUEE, GIIE, GICE

34. If and a basic solution When from(2-4) (note that may not ≥0 Substituting(2-5) into the cost function Define as the relative cost vector => To minimize Z, we need only adjust How? 2008/2/2734Shi-Chung Chang, NTUEE, GIIE, GICE