Lecture 7 Constrained Optimization Lagrange Multipliers ____________________________________________ Ordinary Differential equations Math for CS Lecture 7

Constrained Optimization Constrained optimization problem can be defined as following: Minimize the function ,while searching among x, that satisfy the constraints: Math for CS Lecture 7

Example 1 Consider the problem of minimizing f(x)=x1+x2 and constraint that g(x)=x12+x22-1. The figure shows the circle defined by g(x)=0 and the lines of constant value of f(x). One can see, that at the g(x,y)= x12+x22-1=0 x* solution x*, the gradient of f(x) is orthogonal to the circle. Otherwise, there is non-zero projection of the gradient on the circle, and therefore, sliding contrary to this projection projection decreases the f(x) without violating the constraint g(x)=0. Math for CS Lecture 7

Example 2 g(x,y)=0 Minimize the path between M and C, so that the path touches the constraint g(x)=0. Each ellipse describes the points lying on paths of the same lengths. Again, in the solution the gradient of f(x) is orthogonal to the curve of the constraint. Math for CS Lecture 7

Dimensionality Reduction The straightforward method to solve constrained optimization problem is to reduce the number of free variables: If x=(x1,..xn) and there are k constraints g1(x)=0,…gk(x)=0, then, the k constraint equations can (sometimes) be solved to reduce the dimensionality of x from n to n-k: Math for CS Lecture 7

Surfaces defined by constraint Now we consider the hard case, when dimensionality reduction is impossible. If there are no constraints (k=0), the gradient of f(x) vanishes at the solution x*: In the constrained case, the gradient must be orthogonal to the subspace, defined by the constraints (otherwise a sliding along this subspace will decrease the value f(x), without violating the constraint). Math for CS Lecture 7

g1(x)=0 g2(x)=0 λ1∆g1(x) λ2∆g2(x) Explanation The constraints limit the subspace of the solution. Here the solution lies on the intersection of the planes, defined by g1(x)=0 and g2(x)=0. The gradient f(x) must be orthogonal to this subspace (otherwise there is non-zero projection of f(x) along the constraint and the function value can be further decreased). The orthogonal subspace is spanned by λ1 g1(x)+ λ2 g2(x). Thus f(x*)= λ1 g1(x*)+ λ2 g2(x*). Math for CS Lecture 7

Constrain for coordinate and relaxation for the gradient We observe, that the more additional constraints are applied, the more restricted is the coordinate of the optimum (anywhere in R3 for k=0, on the surface for k=1, on the line for k=2), but the less restricted is the gradient of the function f(x) (zero, along the normal to the surface, within the plane, orthogonal to the line). This requirement for the gradient to lie in the hyperspace, orthogonal to the intersection of the hypersurfaces, defined by the constraints can be summarized as: (1) Math for CS Lecture 7

Lagrange Multipliers The second requirement is to satisfy the constraints: The requirements (1-2) can be written together in the elegant form. Define the function: Then, we can write to satisfy (1) and to satisfy (2). (2) (2) Math for CS Lecture 7

Lagrange Multipliers In other words, we have constructed a function that depends on a variable , and we require for that function. (2) Math for CS Lecture 7

Summary The constants λi are called Lagrange Multipliers. We obtained that the solution (λ* , x*) of the constrained optimization problem Satisfies the equations ,where (2) Math for CS Lecture 7

Ordinary Differential Equations (2) Linear Equations Separable equations Math for CS Lecture 7

Differential Equations 1. A differential equation is an equation involving an unknown function and its derivatives. 2. The order of the differential equation is the order of the highest derivative of the unknown function involved in the equation. 3. A linear differential equation of order n is a differential equation written in the following form: where is not the zero function. (2) Math for CS Lecture 7

Differential Equations 4. Existence: Does a differential equation have a solution? 5. Uniqueness: Does a differential equation have more than one solution? If yes, how can we find a solution which satisfies particular conditions? 6. If the values of the unknown function y(x) and its derivatives at some point are known is called an initial value problem (in short IVP). 7. If no initial conditions are given, we call the description of all solutions to the differential equation the general solution. (2) Math for CS Lecture 7

First order differential equation A first order linear differential equation has the following form: To solve this equation, let us multiply both sides by : (2) Math for CS Lecture 7

Separable Equations The differential equation of the form is called separable, if f(x,y) = h(x)·g(y); that is, In order to solve it, perform the following steps: (1) Solve the equation g(y) = 0, which gives the constant solutions of (a); (2) Rewrite the equation (a) as (a) (2) Math for CS Lecture 7

Separable Equations (3) Now we can integrate to obtain (2) (4) Now we can write down all the solutions, obtained in (1) and (2). If this is an IVP, we must use an initial to find a particular solution. (2) Math for CS Lecture 7