Lecture #3; Based on slides by Yinyu Ye Geometry of LP: Feasible Regions, Feasible Directions, Basic Feasible Solutions Ashish Goel Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/~ashishg http://www.stanford.edu/class/msande211/ Lecture #3; Based on slides by Yinyu Ye

LP Feasible Region in the Inequality Form x simultaneously satisfy This is the intersection of the m Half-spaces, and it is a convex (polyhedron) set Lecture #3; Based on slides by Yinyu Ye

Lecture #3; Based on slides by Yinyu Ye

Lecture #3; Based on slides by Yinyu Ye

Corner or Extreme Points Convex Hull: Extreme Points: A point in a set that does not belong to the hull (convex combination) of the other points For LP in inequality form, an extreme point is the intersection of n hyperplanes associated with the inequality constraints. Lecture #3; Based on slides by Yinyu Ye

Lecture #3; Based on slides by Yinyu Ye Feasible Direction Direction: A direction is notated by a vector d It is always associated with a “location” or point x Together a point and a direction define a ray: x + ϵd, for all ϵ > 0 where d and kd are considered the same direction for all k > 0 Feasible Direction: A direction, d, is said to be “feasible” (relative to a given feasible point x) if x + ϵd is feasible for some ϵ> 0 For LP, all feasible directions at a feasible point form a convex (cone) set Lecture #3; Based on slides by Yinyu Ye

Extreme Feasible Direction A feasible direction d is extreme if d cannot be written as an convex combination of other feasible directions Interior Point is a point x where every direction is feasible Lecture #3; Based on slides by Yinyu Ye

LP Problem in Inequality Form Lecture #3; Based on slides by Yinyu Ye

Recall the Production Problem Objective contour c Lecture #3; Based on slides by Yinyu Ye

Basic Theorems of Linear Programming All LP problems fall into one of three cases: Problem is infeasible: Feasible region is empty. Problem is unbounded: Feasible region is unbounded towards the optimizing direction. Problem is feasible and bounded; and in this case: there exists an optimal solution or optimizer. There may be a unique optimizer or multiple optimizers. All optimizers form a convex set and they are on a face of the feasible region. There is always at least one corner (extreme) optimizer if the feasible region has a corner point. Moreover, Local optimality implies global optimality Lecture #3; Based on slides by Yinyu Ye

Sketch Proof of Local Optimality Implies Global (P) minimize f (x) subject to x ∈ Ω ⊂ Rn, is a Convex Optimization problem if f (x) is a convex function over a convex feasible region Ω. Proof by contradiction. Suppose x’ is a local minimizer but not a global minimizer x∗, that is, x’ ∈ Ω and x∗ ∈ Ω but f (x∗) < f (x’). Now the convex combination point αx’ + (1 − α)x∗ must be feasible (why?), and f (αx’+ (1 − α)x∗) ≤ αf (x’)+ (1 − α)f (x∗) < f (x’) for any 0 ≤ α < 1. This contradicts the local optimality as α can be arbitrarily close to 1 so that αx’+ (1 − α)x∗ can be arbitrarily close to x’. Lecture #3; Based on slides by Yinyu Ye

Optimality Certification of the Production Problem a4 a5 a1 a2 a3 a4 Objective contour c Lecture #3; Based on slides by Yinyu Ye

Feasible Directions at a Corner a4 a5 a1 a2 a3 a4 Objective contour c Lecture #3; Based on slides by Yinyu Ye

How to Certify a Corner being an Optimizer Every feasible direction at the corner is a worsening (descent in this case) direction, that is, cTd ≤0 in this case. Or, equivalently, c is a conic combination of the normal directions at the corner point, that is, there are multipliers α1 ≥0 and α2 ≥0, such as c= α1 ai1+ α2 ai2, in the 2-dimensional case. In the n-dimensional case: c= α1 ai1+…+ αn ain where ai1,…, ain are the normal directions associated with the corner point. Lecture #3; Based on slides by Yinyu Ye

LP in Standard (Equality) Form Lecture #3; Based on slides by Yinyu Ye

Reduction to Standard Form max cTx to min – cTx Eliminating ”free” variables: substitute with the difference of two nonnegative variables x := xl − xll, (xl, xll) ≥ 0. Eliminating inequalities: add a slack variable aT x ≤ b = ⇒ aT x + s = b, s ≥ 0 aT x ≥ b = ⇒ aT x − s = b, s ≥ 0 Lecture #3; Based on slides by Yinyu Ye

Reduction of the Production Problem min -x1 -2x2 s.t. x1 +x3 = 1 x2 +x4 +x2 +x5 = 1.5 (x1, x2, x3, x4, x5) ≥ 0 x3, x4,and x5 are called slack variables Lecture #3; Based on slides by Yinyu Ye

Basic and Basic Feasible Solution In the LP standard form, select m linearly independent columns, denoted by the variable index set B, from A. Solve AB xB = b for the dimension-m vector xB . By setting the variables, xN , of x corresponding to the remaining columns of A equal to zero, we obtain a solution x such that Ax = b. Then, x is said to be a basic solution to (LP) with respect to the basic variable set B. The variables in xB are called basic variables, those in xN are nonbasic variables, and AB is called a basis. If a basic solution xB ≥ 0, then x is called a basic feasible solution, or BFS. Note that AB and xB follow the same index order in B. Two BFS are adjacent if they differ by exactly one basic variable. Lecture #3; Based on slides by Yinyu Ye

BS of the Production Problem in Equality Form x1 x2 x1 +x3 = 1 x2 +x4 +x2 +x5 = 1.5 (x1, x2, x3, x4, x5) ≥ 0 Basis 3,4,5 1,4,5 3,4,1 3,2,5 3,4,2 1,2,3 1,2,4 1,2,5 Feasible? √ x1 , x2 0, 0 1, 0 1.5, 0 0, 1 0, 1.5 .5, 1 1, .5 1, 1 Lecture #3; Based on slides by Yinyu Ye

BFS and Corner Point Equivalence Theorem Theorem Consider the feasible region in the standard LP form. Then, a basic feasible solution and a corner (extreme) point are equivalent; the formal is algebraic and the latter is geometric. Algorithmically, we need only to look at BFSs for finding an optimizer. But still too many to look… Lecture #3; Based on slides by Yinyu Ye