1. Linear programming Thomas S. Ferguson University of California at Los Angeles Compressive Sensing Tutorial PART 3 Svetlana Avramov-Zamurovic January 29, 2009.

2. Definitions  A linear programming: problem of maximizing or minimizing a linear function subject to linear constraints.  x i ≥ 0: nonnegativity constraints.  Ax≤b: main constraints (inequalities).  A vector x or y feasible if it satisfies the corresponding constraints.  Constraint set is set of feasible vectors.  LP problem is feasible if constraint set is not empty.  The function to be maximized (or minimized) is called the objective function c T x. Objective function always takes on its maximum (minimum) value at a corner point of the constraint set if the constraint set is bounded.  A feasible problem is unbounded if the objective function can assume arbitrarily large values at feasible vectors. Otherwise it is bounded. (a)Some constraints may be equalities. Solve the equality, eliminate the variable. (b) Some variables may not be restricted to nonnegatives. x=u-v (u,v are nonnegative, u>0, v>0). This adds extra variable to the problem

3. Duality MAX c T x, due to Ax≤b, x>0 MIN y T b, due to y T A≥c T, y>0 x1x1 x2x2 …xnxn y1y1 a 11 a 12 …a 1n ≤ b 1 y2y2 a 21 a 22 …a 2n ≤ b 2 ymym a m1 a m2 …a mn ≤ b m ≥ c 1 ≥ c 2 …≥ c n Objective function Constraints MAX variable MIN variables

4. Duality: If you conjecture (guess) a solution to one problem, you may solve the dual using the complementary slackness conditions and than see if your conjecture is correct.  Thm 1:If x is feasible for the standard MAX problem and if y is feasible for its dual then c T x ≤ y T b.  C1:If a standard problem and its dual are both feasible, then both are bounded feasible.  C2:If there exists feasible x * and y * for a standard MAX problem and its dual such that c T x * = y * T b than both are optimal for their respective problems.  The Duality Thm:If a standard LP problem is bounded feasible, than so its dual, their values are equal, and there exist optimal vectors for both problems.  Equilibrium Thm:Let x * and y * be feasible vectors for a standard MAX problem and its dual respectively. Then x * and y * are optimal if and only if Complementary slackness conditions: strict inequality (slackness) in a standard problem constraint implies that complementary constraint in the dual be satisfied with equality

5. The Pivot Operation Pivoting results in matrix inverse (special case) y T A= s T => y T =A -1 s T

6. Linear programming A system of linear inequalitiessystem of linear inequalities defines a polytope aspolytope a feasible region. The simplex algorithm begins at a starting vertexvertex and moves along the edges of the polytope until it reaches the vertex of the optimum solution.

7. LP MIN: Find y to minimize y T b subject to y≥0 and y T A ≥ c T Simplex method Find y and s to minimize y T b subject to y≥0, s≥0 and s T = y T A-c T s1s1 s2s2 …snsn y1y1 a 11 a 12 …a 1n b 1 y2y2 a 21 a 22 …a 2n b 2 ymym a m1 a m2 …a mn bmbm 1-c 1 -c 2 …-c n 0 Objective function Slack variable MIN variables Simplex Tableau s T = y T A-c T

8. Simplex method is pivot madly method that tells you which points to choose as pivots to approach the solution systematically.  If -c≥0 and b≥0, there is an obvious solution to the problem: the minimum occurs at y=0; and s=-c and the minimum value y T b=0.  Assume: there is at least one nonnegative entry in the last column or last row (exclusive of the corners). If we pivot about a 11, including the last column and the last row we have r=(y 1, s 2,…, s n ) and t=(s 1,y 2, …,y m ) Find vectors y and s to minimize t T b subject to y≥0, s≥0 and r=t T A ^ -c ^  The Pivot Madly method: Pivot madly until you suddenly find -c ^ ≥0 and b ^ ≥0 Then, obtain a feasible point by setting r=-c and t=0 The value is the lower right corner. r tAb -cv LP Max problem (slack variable u=b-Ax, same tableau if –u=Ax-b)

9. Pivot rules for simplex method from the point of the MAX problem  b≥0  b≥0. Take any column with last entry negative, say -c j0 0, choose that i 0 for which the ratio b j /a i,j0 is smallest. If there are ties, choose any such i 0. Pivot around a i0,j0. After pivoting the b column stays nonnegative, so you still have a feasible point for the maximum problem. The value of the new tableau is never less than the old (generally greater.)  b ^ i=b j -a i,j0 b j0 / a i0,j0 and b ^ i=b j0 / a i0,j0  v ^ =v-(c j0 )b j0 / a i0,j0  Some b i are negative  Some b i are negative. Take the first negative b j, say b k 0 and a i,j0 >0 and choose i 0 for which this ratio is smallest (i 0 may be equal to k). You may choose any such i 0 if there are ties. Pivot on a i0,j0. George Dantzig in 1947 The algorithm moves along the edges of the polyhedron defined by the constraints, from one vertex to another, while decreasing the value of the objective function, c T x at each step.

10. Pivot rules for simplex method from the point of the MIN problem  -c≥0  -c≥0. Take any column with last entry negative, say b i0 <0. Among those j for which a i0,j <0, choose that j 0 for which the ratio -c j /a i0,j is closest to 0. If there are ties, choose any such j 0. Pivot around a i0,j0.  Some -c i are negative  Some -c i are negative. Take the first negative -c i, say - c k 0. Compare -c k /a i0,k and -c j /a i0,j for which -c j >0 and a i0,j <0 and choose j 0 for which this ratio is clossest to zero (j 0 may be equal to k). Pivot on a i0,j0.