1. Computer Algorithms Mathematical Programming ECE 665 Professor Maciej Ciesielski By DFG

2. 2 Department of Electrical Engineering - UMASS Overview  Classical optimization deals with unconstrained optimization problems.  For the minimization problem: Min f(x) subject to g(x) = 0 The solution is to move the constraints to the function and solve the unconstrained optimization problem. Penalty function method: replace f(x) by F(x) = f(x)+ M( g(x)) 2, M >> 0 Lagrangian multipliers: replace f(x) by L(x,λ) = f(x) + Σ λ i g i (x) At the optimum point, x*, L(x*, λ) = f(x*)

3. 3 Department of Electrical Engineering - UMASS Outline  Mathematical programming paradigm  Feasible solutions? Convexity Concave Concept example  Geometric solution  Fundamental theorem of LP  Simplex Example  LP Modifications

4. 4 Department of Electrical Engineering - UMASS Mathematical Programming Paradigm A mathematical program is an optimization problem of the form: Maximize (or Minimize) f(x) subject to: g(x) = 0 h(x) ≥ 0 where x = [x1,...xn] is a subset of R n, the functions g and h are called constraints, and f is called the objective function. Mathematical Programming paradigm

5. 5 Department of Electrical Engineering - UMASS The Nature of Mathematical Programming Of course there are some forms that deviate from this paradigm. Important examples are:  Fuzzy mathematical program Fuzzy mathematical program  Goal program Goal program  Multiple objectives Multiple objectives  Randomized program Randomized program  Stochastic program Stochastic program Therefore it is typically a modeling issue to find an standard form to solve the mathematical programming.

6. 6 Department of Electrical Engineering - UMASS When is it feasible? A point x is feasible if it is in X and satisfies the constraints: g(x) = 0 and h(x) ≥ 0. A point x* is optimal if it is feasible and if the value of the objective function is: For Maximization not less than that of any other feasible solution: f(x*) ≥ f(x) for all feasible x. For Minimization not greater than that of any other feasible solution: f(x*) ≤ f(x) for all feasible x. Min or Max f(x) Subject to: g(x) = 0 h(x) ≥ 0 Mathematical Programming Paradigm

7. 7 Department of Electrical Engineering - UMASS Convexity  To obtain global optimum, it is important that the constraint set {x | g(x) = 0, h(x) ≥ 0} and the function f(x) be convex. If the function is convex over a convex constraint set, then local minimun is also a globa1 minimum Min or Max f(x) Subject to: g(x) = 0 h(x) ≥ 0 Mathematical Programming Paradigm

8. 8 Department of Electrical Engineering - UMASS Convexity  Set convexity (If any two points are in the set, so is their line segment)line segment Convex setnon-convex set  Function convexity Convex function non-convex function ???

9. 9 Department of Electrical Engineering - UMASS Convexity A function f:X–>R is said to be convex if:  Its epigraph is convex.epigraph  X is a convex set, and for x,y  X and a[0, 1]: f(ax + (1-a)y) ≤ af(x) + (1-a)f(y) Epigraph: region on or above the graphgraph

10. 10 Department of Electrical Engineering - UMASS Convexity If the function is convex over a convex constraint set, then local minimun is also a globa1 minimum

11. 11 Department of Electrical Engineering - UMASS Concave  It is define for maximization, and it is the negative of convexity.  A function f is concave if its hypograph is convexhypograph Hypograph: region below the graphgraph

12. 12 Department of Electrical Engineering - UMASS Concave If the function is concave over a convex constraint set, then local maximun is also a globa1 maximum

13. 13 Department of Electrical Engineering - UMASS Concept Example  Can we minimize based on the local minimum? f(x) = x T Qx  Subject to: Ax ≥ b x ≥ 0  Where Q is an n x n symmetric matrix, A is a constraint matrix, and b is a constraint vector. Quadratic Programming

14. 14 Department of Electrical Engineering - UMASS Concept Example  If x T Qx > 0 for every x  0. The matrix Q is positive definite. In which case its eigenvalues are non-negative, and the quadratic form is a convex function. Thus f is convex. convex function As Q is a symmetric matrix, these eingenvalues will be reals.  Ax ≥ b defines a convex constraint set, as its solution is a R k space. (A convex polyhedron)  f and the constraint set are convex, therefore, finding the local minimum also finds the global minimum.

15. 15 Department of Electrical Engineering - UMASS Linear Programming (LP)  The goal of Linear programming is to: (Max)Minimize:C T x Subject to:Ax ≥ b x ≥ 0 Where C T is a coefficient vector for f, A is a constraint matrix, and b es a constraint vector.  The constraint set: {x | Ax ≥ b} is a convex polyhedron, and f is linear so it is convex. Therefore, LP has convexity, and the local min/max is the global min/max

16. 16 Department of Electrical Engineering - UMASS LP Example (Problem Enunciation) Optimization problem:  2 types of products are made by the factory  3 machines are needed to make each product Constraints:Machines are available for a limited time during one production session:  time on A ≤ 50 hours  time on B ≤ 35 hours  time on C ≤ 80 hours

17. 17 Department of Electrical Engineering - UMASS LP Example (Problem Enunciation) To complete the product, each lot must be processed by all three machines for a certain number of hours Required manufacturing time Machines A B C 1 Products 2 Time Constraint: 50 35 80 1055 5515

18. 18 Department of Electrical Engineering - UMASS LP Example (Problem Enunciation) Profit:  Product 1 can be sold for $100 per lot  Product 2 can be sold for $80 per lot Optimization Problem:  Design the production schedule which maximizes the profit of both products manufactured during one production session.

19. 19 Department of Electrical Engineering - UMASS LP Example (Problem Formulation)  Formulation x 1 = number of lots of product 1 x 2 = number of lots of product 2 max f (x) = 100x 1 + 80x 2 10x 1 + 5x 2 ≤ 50 5x 1 + 5x 2 ≤ 35 5x 1 + 15x 2 ≤ 80

20. 20 Department of Electrical Engineering - UMASS LP Example (Geometric Solution)  Geometric solution: Family of curves, constant for x 1, x 2 F(x) = 100 x 1 + 80 x 2 = 800 (const) (a)10x 1 + 5x 2  50 (b) 5x 1 + 5x 2  35 (c) 5x 1 + 15x 2  80 ab x1x1 x2x2 10 5 15105 c P2P2 P1P1 3 4 P0P0 F(x) = 800 Sol: X 1 = 3 X 2 = 4 Extreme point

21. 21 Department of Electrical Engineering - UMASS LP Example (Standard Linear Program Approach)  Convert the problem to a standard Linear Program (LP): Max f(x) = 100x 1 + 8Ox 2 + 0s1 + 0s2 + 0s3 10x 1 + 5x 2 + s1= 50 Subject to: 5x 1 + 5x 2 + s2= 35 5x 1 + 15x 2 + s3= 80 Basis New Matrix A Why the Basis??

22. 22 Department of Electrical Engineering - UMASS Basis and Basic solution Why the Basis? Because it finds a basic solution Given a system of equalities:Ax = b where: x = n-vector, b = m-vector, A = mn matrix select a set of m linearly independent columns such that (mm) matrix B is nonsingular, i.e. |B|  0. Then one may uniquely solve the equation:Bx b = b where: x b is an m-element subvector of x. Max f(x) = 100x 1 + 8Ox 2 + 0s1 + 0s2 + 0s3 Subject to: 10x 1 + 5x 2 + s1 = 50 5x 1 + 5x 2 + s2 = 35 5x 1 + 15x 2 + s3 = 80

23. 23 Department of Electrical Engineering - UMASS Basis and Basic solution By putting x=(x b,0) we obtain a solution to Ax = b Such a solution (with n-m components of x not associated with columns of B mm ) to the resulting set of equations is said to be a basic solution with regard to the basis B. The X b variables, associated with columns of B, are called basic variables. Basic Feasible Solution If a feasible solution (i.e. the one which satisfies all contraints) is also basic, it is called a basic feasible solution. Max f(x) = 100x 1 + 8Ox 2 + 0s1 + 0s2 + 0s3 Subject to: 10x 1 + 5x 2 + s1 = 50 5x 1 + 5x 2 + s2 = 35 5x 1 + 15x 2 + s3 = 80 B.F.S x1 = 0 s1 = 50 s3 = 80 x2 = 0 s2 = 35 basic variables

24. 24 Department of Electrical Engineering - UMASS Fundamental Theorem of Linear Programming Given an LP in standard form min C T x subject toAx = b x  0 where A is mn matrix of rank m (i.e. m < n and the m rows are linear independent), If there is a feasible solution, there is a basic feasible solution If there is an optimal feasible solution, there is an optimal basic feasible solution

25. 25 Department of Electrical Engineering - UMASS Fundamental Theorem of Linear Programming This theorem shows that it is necessary only to consider basic feasible solutions when seeking an optimal solution to a linear program, as the optimal value is always achieved as such a solution Ultimate goal: find a basic feasible solution with a base B composed of original variables only, and which is optimum.

26. 26 Department of Electrical Engineering - UMASS LP Example (Basic Feasible solution) ƒ (x) = 100 x 1 + 80 x 2 + 0 s1 + 0 s2 + 0 s3  Solution: x1 = 0, x2 = 0, s1 = 50, s2 = 35, s3 = 80, is a basic feasible solution: point P 0 (0,0) Extreme point  Function value: F(0, 0, 50, 35, 80) = 0 basis

27. 27 Department of Electrical Engineering - UMASS LP Example (Basic Feasible solution) The basic feasible solution in the Geometric approach is in the Extreme Point. ab x1x1 x2x2 10 5 15105 c P2P2 P1P1 3 4 P0P0 Extreme point

28. 28 Department of Electrical Engineering - UMASS Simplex Method  Proceed from one basic feasible solution (extreme point) to another, in such a way as to continually decrease / increase the value of the f (x) until a minimum / maximum is reached. General comments:  It is easy find initial basic feasible solutions with slack variables.  Finding initial basic solution is part of the Simplex method (Lue 84).

29. 29 Department of Electrical Engineering - UMASS Simplex Algorithm 1.Select the column, such that the new resulting basic feasible solution will yield a lower / greater value to f(x) than the previous one. 2.Select the pivot element in that column. By an elementary evaluation determine: Which vector a j should enter the basis (so that f(x) is reduced), and Which vector should leave the basis.

30. 30 Department of Electrical Engineering - UMASS Simplex Algorithm Construct a Simplex Table, [A, b ] appended by a row at the top with cost coefficients and current cost. j = 123450 rjrjrjrj 100800000 y 1j 10510050 y 2j 5501035 y 3j 51500180 y i1 y i2 y i3 y i4 y i5 y i0 Point P 0 (0,0) F(0,0,50,35,80) = 0

31. 31 Department of Electrical Engineering - UMASS Simplex Algorithm  Step 1.Select column q with r q > 0. (x q enters the basis) q : r q = max r i  Step 2.Select row p, which defines pivot element p in column q Which minimizes ratio: p : = min for positive y iq i

32. 32 Department of Electrical Engineering - UMASS Simplex Algorithm  Step 3.Pivot on element y pq and update the table (update all rows, including the top one) In row p divide y pj by y pq, j = 0, 1,…, n Subtract from each row i  p  y pj, j = 0, 1,…, n

33. 33 Department of Electrical Engineering - UMASS Simplex Algorithm Example  Pivot on y 11 j = 123450 rjrjrjrj 100800000 y 1j 10510050 y 2j 5501035 y 3j 51500180 y i1 y i2 y i3 y i4 y i5 y i0 Point P 0 (0,0) F(0,0,50,35,80) = 0 Step1: max r q Step2: min of Therfore: y pq = 10

34. 34 Department of Electrical Engineering - UMASS Simplex Algorithm Example  Update table j = 123450 rjrjrjrj 100800000 y 1j 10510050 y 2j 5501035 y 3j 51500180 y i1 y i2 y i3 y i4 y i5 y i0 Point P0 (0,0) F(0,0,50,35,80) = 0 Step 3.B: in row i ≠ p Step 3.A: in row p j = 123450 rjrjrjrj 100800000 y 1j 10.50.1005 y 2j 5501035 y 3j 51500180 y i1 y i2 y i3 y i4 y i5 y i0 j = 123450 rjrjrjrj 030-1000-500 y 1j 10.50.1005 y 2j 02.5-0.51010 y 3j 012.5-0.50155 y i1 y i2 y i3 y i4 y i5 y i0 Point P 1 (5,0) F(5,0,0,10,55) = 500

35. 35 Department of Electrical Engineering - UMASS Simplex Algorithm Example  Pivot on y 22 j = 123450 rjrjrjrj 030-1000-500 y 1j 10.50.1005 y 2j 02.5-0.51010 y 3j 012.5-0.50155 y i1 y i2 y i3 y i4 y i5 y i0 Point P1 (5,0) F(5,0,0,10,55) = 500 Step1: max r q Step2: min of Therfore: y pq = 4

36. 36 Department of Electrical Engineering - UMASS j = 123450 rjrjrjrj 100800000 y 1j 10.50.1005 y 2j 5501035 y 3j 51500180 y i1 y i2 y i3 y i4 y i5 y i0 Simplex Algorithm Example  Update table Point P1 (5,0) F(5,0,0,10,55) = 500 Step 3.B: in row i ≠ p Step 3.A: in row p Point P 2 (3,4) F(3,4,0,0,5) = 620 j = 123450 rjrjrjrj 030-1000-500 y 1j 10.50.1005 y 2j 01-0.20.404 y 3j 012.5-0.50155 y i1 y i2 y i3 y i4 y i5 y i0 j = 123450 rjrjrjrj 00-4-120-620 y 1j 101003 y 2j 01-0.20.404 y 3j 002-515 y i1 y i2 y i3 y i4 y i5 y i0

37. 37 Department of Electrical Engineering - UMASS Simplex Algorithm Example  Pivot on j = 123450 rjrjrjrj 00-4-120-620 y 1j 101003 y 2j 01-0.20.404 y 3j 002-515 y i1 y i2 y i3 y i4 y i5 y i0 Point P2 (3,4) F(3,4,0,0,5) = 620 Step1: all r q < 0 Terminate algorithm Sol: X 1 = 3 X 2 = 4

38. 38 Department of Electrical Engineering - UMASS Linear Programming Modifications 1. Maximization 2. Unconstrained x max c t x = min (-c t x) let x = x + - x - ; x +, x -  0) min (c t x + - c t x - ) [A, -A] = b x + x - x +, x -  0 s. to

39. 39 Department of Electrical Engineering - UMASS Linear Programming Modifications 3. Constraint set: Ax  b Slack variables, x s : A x + x s = b [A, I ] = b x xxxsxsxxxsxs x, x s  0 min [c T, 0] xxsxxs Columns corresponding to x s form a basic solution

40. 40 Department of Electrical Engineering - UMASS Linear Programming Modifications 4. Constraint set: Ax ≥ b Surplus variables, x p : A x - x p = b [A, -I ] = b x xxxpxpxxxpxp x, x p  0 min [c T, 0] xxpxxp

41. 41 Department of Electrical Engineering - UMASS Linear Programming Modifications 5. Artificial variables, x a : Ax - x p + x a = b (cannot star with negative basis of x p ) [A, -I, I ] = b x, x p, x a  0 min [c T, 0, k I ] xxpxaxxpxa xxpxaxxpxa S. to Columns corresponding to x a form a basic solution