1. Application of Matrix Differential Calculus for Optimization in Statistical Algorithm By Dr. Md. Nurul Haque Mollah, Professor, Dept. of Statistics, University of Rajshahi, Bangladesh 01/10/20111Dr. M. N. H. Mollah

2.  Introduction to the Optimization Problems in Statistics  Test of Convexity of Objective function for Optimization  Nonlinear Optimization for Regression Analysis  Nonlinear Optimization for Maximum Likelihood Estimation  Nonlinear Optimization for PCA  Nonlinear Objective Function for Information Theory Outline 01/10/20112Dr. M. N. H. Mollah

3.  The main problem in parametric statistics is to find the unconditional or conditional optimizer of the nonlinear objective functions.  Two classical nonlinear conditional optimization problems are the equality constrained problem (ECP) optimize f(x) subject to h(x) = 0 and its inequality constrained problem (ICP) optimize f(x) subject to g(x) ≤ 0 ( ≥0) where f : R n →R, h : R n →R m, g : R n →R r are given functions. 1. Introduction to the Optimization Problems in Statistics 01/10/20113Dr. M. N. H. Mollah

4.  If the objective function is the profit function, then optimizers are the maximizers of the objective function.  If the objective function is the loss function, then optimizers are the minimizers of the objective function.  For Example, likelihood function for the statistical model can be considered as the profit function, while any error or distance functions like sum of square errors (SSE) in regression problems, K-L or any other divergence between two distribution s can be considered as the loss function.  Matrix derivatives play the key role to detect the optimizer of the objective function 1. Introduction to the Optimization Problems in Statistics (Cont.) 01/10/20114Dr. M. N. H. Mollah

5.  An overview of analytical and computational methods for solution of the unconstraint problem (UP) optimize f(x) where f: R n →R, xє R n, is a given functions.  A vector x * is a local minimum for UP if there exists an ε>0 such that f(x*) ≤ f(x), for all x є S(x*; ε)  It is a strict local minimum if there exists an ε >0 such that f(x*) < f(x), for all x є S(x*; ε), x ≠ x* 1.1 Unconstrained Optimization 01/10/20115Dr. M. N. H. Mollah

6.  Note: S(x*; ε) is an open sphere centered at x* є R n with radius ε > 0. That is S(x*; ε)={x: |x-x*|< ε}, where x є R n  A vector x * is a local maximum for UP if there exists an ε>0 such that f(x) ≤ f(x*), for all x є S(x*; ε)  It is a strict local minimum if there exists an ε >0 such that f(x) < f(x*), for all x є S(x*; ε), x ≠ x* 1.1 Unconstrained Optimization (Cont.) 01/10/20116Dr. M. N. H. Mollah

7. We have the following well-known optimally conditions. If f: R n →R, x є R n is differentiable (i.e., f  C 1 on S(x * ; ε) for ε>0 ) and is a (local) minimum, then i.e. the Jacobian Df(x) =0.  If f is twice differentiable (i.e., f  C 2 on S(x * ; ε) for ε>0 ) and is a (local) minimum, then i.e. the Hessian D 2 f(x) is positive definite and f(x) is a convex function.  If x* is the maximizer of f(x), then i.e. the Hessian D 2 f(x) is negative definite and f(x) is a concave function. 1.1 Unconstrained Optimization (Cont.) 01/10/20117Dr. M. N. H. Mollah

8. We consider first the following equality constrained optimization problem (ECP) minimize f(x) subject to h(x)=0, where f : R n →R and h: R n →R m are given functions and m≤n. the components of h are denoted h 1, h 2,….,h m. Let x * be a vector such that h(x * )=0 and, for some ε>0, h  C 1 on S(x * ; ε). We say that x * is regular point if the gradients Dh 1 (x * ),…,Dh m (x * ) are linearly independent. Consider the Lagrangain function L:R n+m →R defined by L(x,λ)=f(x)+λ´h(x). Defination: 1.2 Constrained Optimization 01/10/20118Dr. M. N. H. Mollah

9.  Let x * be a local minimum for (ECP), and assume that, for some ε>0, f  C 1, h  C 1 on S(x * ; ε), and x * is regular point. Then there exist a unique vector λ *  R m such that  If in addition f  C 2 and h  C 2 on S(x*; ε) then 1. 2 Constrained Optimization (Cont.) 01/10/20119Dr. M. N. H. Mollah

10. We consider first the following inequality constrained optimization problem (ICP) minimize f(x) subject to h(x)=0, g(x)≤0 where f : R n →R, h: R n →R m and g: R n →R r are given functions and m ≤ n. Let x * be a vector such that h(x * )=0, g(x * ) ≤0 and, for some ε>0, h  C 1 and g  C 1 on S(x * ; ε). We say that x * is regular point if the gradients Dh 1 (x * ),…,Dh m (x * ) and Dg j (x * ), jЄA(x * ) are linearly independent. Consider the Lagrangain function L:R n+m+r →R defined by L(x,λ, µ)=f(x)+λ´h(x)+ µ ´g(x). Defination 1. 2 Constrained Optimization (Cont.) 01/10/201110Dr. M. N. H. Mollah

11.  Let x * be a local minimum for (ICP), and assume that, for some ε>0, f  C 1, h  C 1, g  C 1 on S(x * ; ε), and x * is regular point. Then there exist a unique vectors λ *  R m, µ *  R r such that,  If in addition f  C 2, h  C 2 and g  C 2 on S(x * ; ε), then 1. 2 Constrained Optimization (Cont.) 01/10/201111Dr. M. N. H. Mollah

12.  For optimization problem, convexity of the objective function f (x) is necessary.  Hessian matrix H=D 2 f (x) can be used to test the convexity of the objective function.  If all leading principal minors of H are non-negative (non- positive), then function f (x) is convex (concave).  If all leading principal minors of H are positive (negative), then f (x) is strictly convex (concave).  Note: H will always be a square symmetric matrix 2. Test of Convexity of Objective function for Optimization 01/10/201112Dr. M. N. H. Mollah

13. 2. Test of Convexity of Objective function for Optimization (Cont.) Example: Since all principal minors are not positive, so f(x) is concave. 01/10/201113Dr. M. N. H. Mollah

14. 3. Nonlinear Optimization for Regression Analysis Unconditional Optimization Let us consider a regression model Where, Y: Vector of response X: Design matrix (matrix of explanatory variable) β : Vector of regression parameter E: Vector of error terms  Objective is to find OLSE of β by minimizing error sum of squares with respect to β. 01/10/201114Dr. M. N. H. Mollah

15. Solution: Which is also known as minimizer of Necessary condition: Hessian matrix should be positive definite. That is It is an example of unconditional optimization. 01/10/201115Dr. M. N. H. Mollah 3. Nonlinear Optimization for Regression Analysis (Cont.)

16. Conditional Optimization Recall the previous linear regression model as Objective is to find OLSE of β by minimizing error sum of squares with respect to β such that 01/10/201116Dr. M. N. H. Mollah 3. Nonlinear Optimization for Regression Analysis (Cont.)

17. Solution: Since the restriction one of the equality form, we can use the method of Lagrange’s undetermined multipliers and minimize the Lagrangian function with respect to β and λ. Then Sufficient condition: should be positive definite matrix. That is 01/10/201117Dr. M. N. H. Mollah 3. Nonlinear Optimization for Regression Analysis (Cont.)

18.  The likelihood function for the parameter vector θ =( θ 1, θ 2, …, θ d ) formed from the observed data X as The problem is to estimate the vector θ by maximizing likelihood function. Ans: The normal equations produce 01/10/201118Dr. M. N. H. Mollah 4. Nonlinear Optimization for Maximum Likelihood Estimation (MLE)

19. Under regularity conditions, the expected (Fisher) information matrix is given by The standard error for the estimate of θ i is given by 01/10/201119Dr. M. N. H. Mollah 4. Nonlinear Optimization for MLE (Cont.)

20. If the normal equations becomes as implicit form, then the Newton-Raphson method or other iterative procedure is necessary for solving the normal equations. However, the Newton-Raphson iteration In this case is updated as follows for k=0,1,2,3,…, until converge. The stopping rule is as follows 01/10/201120Dr. M. N. H. Mollah 4. Nonlinear Optimization for MLE (Cont.)

21. In statistics, an expectation-maximization (EM) algorithm is a method for finding MLE or maximum a posteriori (MAP) estimates of parameters in statistical models, where the model depends on unobserved latent variables. EM is an iterative method which alternates between performing an expectation (E) step, which computes the expectation of the log-likelihood evaluated using the current estimate for the parameters, and a maximization (M) step, which computes parameters maximizing the expected log- likelihood found on the E step. These parameter-estimates are then used to determine the distribution of the latent variables in the next E step. 2101/10/2011Dr. M. N. H. Mollah 4. 1 Nonlinear Optimization for MLE using EM algorithm

22. Given a statistical model consisting of a set x of observed data, a set of unobserved latent data or missing values z,and a vector of unknown parameters θ, along with a likelihood function, the MLE of the unknown parameters θ is determined by the marginal likelihood of the observed data However, this quantity is often intractable. The EM algorithm seeks to find the MLE of the marginal likelihood by iteratively applying the following two steps: 2201/10/2011Dr. M. N. H. Mollah 4.1 Nonlinear Optimization for MLE using EM algorithm (Cont.)

23. The EM algorithm seeks to find the MLE of the marginal likelihood by iteratively applying the following two steps: Expectation step (E-step): Calculate the expected value of the log-likelihood function, with respect to the conditional distribution of Z given X under the current estimate of the parameters θ (t) : Maximization step (M-step): Find the parameter that maximizes this quantity: Note that in typical models to which EM is applied 2301/10/2011Dr. M. N. H. Mollah 4.1 Nonlinear Optimization for MLE using EM algorithm (Cont.)

24. Let x = (x 1,x 2,…,x n ) be a sample of n independent observations from a mixture of two multivariate normal distribution of dimension d, and let z=(z 1,z 2,…,z n ) be the latent variables that determine the component from which the observation originates. and i=1,2,…,n. 24 Example: Gaussian mixture 01/10/2011Dr. M. N. H. Mollah 4. 1 Nonlinear Optimization for MLE using EM algorithm (Cont.)

25. Where The aim is to estimate the unknown parameters representing the "mixing" value between the Gaussians and the means and covariances of each: where the likelihood function is: 2501/10/2011Dr. M. N. H. Mollah 4.1 Nonlinear Optimization for MLE using EM algorithm (Cont.)

26. This may be rewritten in exponential family form as E-step Given our current estimate of the parameters θ (t), the conditional distribution of the Z i is determined by Bayes theorem to be the proportional height of the normal density weighted by as follows Thus, the E-step results in the function: 2601/10/2011Dr. M. N. H. Mollah 4.1 Nonlinear Optimization for MLE using EM algorithm (Cont.)

27. 27 M-step Maximize Q( θ | θ (t) ) w.r. to θ which contains Note that may be all maximized independently of each other since they all appear in separate linear terms. Firstly, consider which has the constraint This has the same form as the MLE for the binomial distribution. So 01/10/2011Dr. M. N. H. Mollah 4.1 Nonlinear Optimization for MLE using EM algorithm (Cont.)

28. For the next estimates of ( μ 1, Σ 1 ): This has the same form as a weighted MLE for a normal distribution, so 2801/10/2011Dr. M. N. H. Mollah 4.1 Nonlinear Optimization for MLE using EM algorithm (Cont.)

29. And and, by symmetry: and 2901/10/2011Dr. M. N. H. Mollah 4.1 Nonlinear Optimization for MLE using EM algorithm (Cont.)

30. Louis(1982) showed that the observed information l obs is the difference of complete l oc and missing l om information. That is,, where and θ * denotes the MLE of θ estimated using EM. 01/10/201130Dr. M. N. H. Mollah 4.1 Nonlinear Optimization for MLE using EM algorithm (Cont.)

31. Let Cov (X p × 1 )= ∑ of order p × p. Let β be the p-component column vector such that β ' β=1. Then the variance of β ' X is (1) To determine the normalized liner combination β ' X with maximization variance, we must find a vector β satisfying β ' β=1 which maximize (1). Let (2) Where λ is a Lagrange multiplier. The vector of partial derivatives is (3) 01/10/201131Dr. M. N. H. Mollah 5. Nonlinear Optimization for PCA

32. Since β'Σβ and β'β have derivatives every where in a region containing β'β=1, a vector β maximizing β'Σβ must satisfy the expression (3) set equal to 0; that is (4) In order to gate a solution of (4) with β'β=1we must have Σ-λI singular; in other words, λ must satisfy (5) The function |Σ - λI| = 0 is a polynomial in λ of degree p. therefore (5) has p roots; let these be λ 1  λ 2  …λ p. If we multiply (4) on the left by ; we obtain (6) 01/10/201132Dr. M. N. H. Mollah 5. Nonlinear Optimization for PCA (Cont.)

33. This shows that if β satisfies (4) (and ββ=1 ), then the variance of βX [given by (1)] is λ. Thus for the maximum variance we should use in (4) the largest root λ 1. Let β (1) be a normalized solution of (Σ – λ 1 I) β = 0 Then U 1 = β (1) X is a normalized linear combination with maximum variance. [if Σ – λ 1 I is of rank p-1, then there is only one solution to (Σ – λ 1 I) β =0 and ββ=1. Now let us find a normalized combination βX that has maximum variance if all linear combinations uncorrelated with U 1. Lack of correlation means (7) Since Σβ (1) = λ 1 β (1). Thus βX is orthogonal to U in both the statistical sense (of lack of correlation) and the geometric sense(of the inner product of the vectors β and β (1) being zero). (that is λ 1 ββ (1) = 0 only if ββ (1) = 0 when λ 1 ≠ 0, and λ 1 ≠ 0 if Σ ≠ 0 ; the case of Σ = 0 is trivial and is not treated.) 01/10/201133Dr. M. N. H. Mollah 5. Nonlinear Optimization for PCA (Cont.)

34. We now want to maximized (8) Where λ and ν 1 are Lagrangain multipliers. The vector of partial derivatives is (9) And we set this equal to 0. From (9) we obtain by multip-lying on the left by β (1) (10) By (7). therefore ν 1 =0 and must satisfy (4), and therefore λ must satisfy (5). Let λ 2 be the maximum of λ 1, λ 2, … λ 2, such that there is a vector β satisfying (Σ – λ 2 I) β =0, ββ=1 and (7); call this vector β (2) and the corresponding linear combination U 2 = β (2) X. (it will be shown eventually that λ (2) = λ 2. We define λ (1) = λ 1.) 01/10/201134Dr. M. N. H. Mollah 5. Nonlinear Optimization for PCA (Cont.)

35. This procedure is continued; alt the (r+1)st step, we want to find a vector β such that βX has maximum variance of all normalized linear combinations which are uncorrelated with U 1,U 2 … U r, that is, such that i=1,2,…..r (11) We want to maximized (12) Where λ and ν 1, ν 1,…, ν r are Lagrangain multipliers. The vector of partial derivatives is (13) And we set this equal to 0. Multiplying (13) on the left by β (j), we obtain (14) If λ (j) ≠ 0, this gives -2 ν j λ (j) = 0 and ν j =0. If λ (j) = 0, then Σ β (j) = λ (j) β (j) =0 and the j-th term in the sum in (13) vanishes. Thus β must satisfy (4), and therefore λ must satisfy (5). Thus we can compute all principal and minor components sequentially by nonlinear optimization using lagrangian multiplier. 01/10/201135Dr. M. N. H. Mollah 5. Nonlinear Optimization for PCA (Cont.)

36.  Differential Entropy:  The differential entropy H of a random vector y with density p(y) is defined as:  A normalized version of entropy is given by negentropy J, which is defined as follows: where. Equality hold only for Gaussian random vectors. 01/10/201136Dr. M. N. H. Mollah 6. Nonlinear Objective Function for Information Theory

37. Mutual Information: Mutual Information between m random variables y i,i=1,2,…,m is defined as follows: Equality holds if are independent. where the constant term does not depend on B 01/10/201137Dr. M. N. H. Mollah 6. Nonlinear Objective function for Information Theory (Cont.)

38.  In signal processing, blind source signals (BSS) are generally follow non-Gaussian distribution and independent of each other.  So maximization of negentropy or equivalently minimization of mutual information both are popular techniques for recovering BSS image processing and audio signal processing. 01/10/201138Dr. M. N. H. Mollah 6. Nonlinear Objective function for Information Theory (Cont.)

39. Gradient Descent: To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient of the function at the current point. If instead one takes steps proportional to the positive of the gradient, one approaches a local maximum of that function; the procedure is then known as gradient ascent. Gradient descent is also known as steepest descent. In gradient descent, we minimize a function J(w) iteratively by starting form some initial point w(0), computing the gradient of J(w) at this point, and then moving in the direction of the negative gradient or the steepest descent by a suitable distance. Once there, we repeat the same procedure at the new point. And so on. 01/10/201139Dr. M. N. H. Mollah 6. 1 An Unconstrained Optimization for Information Theory

40. Gradient Descent: In gradient descent, we minimize a function J(w) iteratively by starting form some initial point w(0), computing the gradient of J(w) at this point, and then moving in the direction of the negative gradient or the steepest descent by a suitable distance. Once there, we repeat the same procedure at the new point. And so on. For t=1,2,… we have the update rule (4.1) We can then write the rule (3.22) either as Or even shorter as 01/10/201140Dr. M. N. H. Mollah 6. 1 An Unconstrained Optimization for Information Theory

41. The symbol is read “is proportional to” it is then understood that the vector on the left-hand side, ∆w, has the same dimension as the gradient vector on the right-hand side, but there is a positive scalar coefficient by which the length can be adjusted. In the upper version of the update rule, this coefficient is denoted by α. In many cases, this learning rate can and should in fact be time dependent. Yet a third very convenient way to write such update rules, in conformity with programming language, is Where the symbol ← means substitution, i.e., the value of the right-hand side is computed and substituted in w. 01/10/201141Dr. M. N. H. Mollah 6. 1 An Unconstrained Optimization for Information Theory (Cont.)

42. Generally, the speed of convergence can be quite low close to the minimum point, because the gradient approaches zero there. The speed can be analyzed as follows. Let us denote by w* the local or global minimum point where the algorithm will eventually converge. From (3.22) we have Let us expand the gradient vector (∂J(w) /∂w) element by element as a Taylor series around the point w*, using only the zeroth and first-order terms, we have for the ith element 01/10/201142Dr. M. N. H. Mollah 6. 1 An Unconstrained Optimization for Information Theory (Cont.)

43. Now because w* is the point of convergence, the partial derivatives of the cost function must be zero at w*.Using this result and compiling the above expansion into vector form, yields Where H(w*)is the Hessian matrix computed at the point w = w*.Substituting this in (3.24) gives This kind of convergence, which is essentially equivalent to multiplying a matrix many times with itself, is called linear. The speed of convergence depends on the learning rate and the size of Hessian matrix. 01/10/201143Dr. M. N. H. Mollah 6. 1 An Unconstrained Optimization for Information Theory (Cont.)