Presentation is loading. Please wait.

Presentation is loading. Please wait.

Maths for Signals and Systems Linear Algebra in Engineering Lectures 10-12, Tuesday 1st and Friday 4th November2016 DR TANIA STATHAKI READER (ASSOCIATE.

Similar presentations


Presentation on theme: "Maths for Signals and Systems Linear Algebra in Engineering Lectures 10-12, Tuesday 1st and Friday 4th November2016 DR TANIA STATHAKI READER (ASSOCIATE."โ€” Presentation transcript:

1 Maths for Signals and Systems Linear Algebra in Engineering Lectures 10-12, Tuesday 1st and Friday 4th November2016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE LONDON

2 In this set of lectures we will talk about:
Eigenvectors and eigenvalues Matrix diagonalization Applications of matrix diagonalization Stochastic matrices

3 Eigenvectors and eigenvalues
Consider a matrix ๐ด and a vector ๐‘ฅ. The operation ๐ด๐‘ฅ produces a vector ๐‘ฆ at some direction. I am interested in vectors ๐‘ฆ which lie in the same direction as ๐‘ฅ. In that case I have ๐ด๐‘ฅ=๐œ†๐‘ฅ with ๐œ† being a scalar. When the above relationship holds, ๐‘ฅ is called an eigenvector and ๐œ† is called an eigenvalue of matrix ๐ด. If ๐ด is singular then ๐œ†=0 is an eigenvalue. Problem: How do we find the eigenvectors and eigenvalues of a matrix?

4 Eigenvectors and eigenvalues of a projection matrix
Problem: What are the eigenvectors ๐‘ฅ โ€ฒ and eigenvalues ๐œ† โ€ฒ of a projection matrix ๐‘ƒ? In the figure, consider the matrix ๐‘ƒ which projects vector ๐‘ onto vector ๐‘. Question: Is ๐‘ an eigenvector of ๐‘ƒ? Answer: No, because ๐‘ and ๐‘ƒ๐‘ lie in different directions. Question: What vectors are eigenvectors of ๐‘ƒ? Answer: Vectors ๐‘ฅ which lie on the projection plane already. In that case ๐‘ƒ๐‘ฅ=๐‘ฅ and therefore ๐‘ฅ is an eigenvector with eigenvalue 1.

5 Eigenvectors and eigenvalues of a projection matrix (cont.)
The eigenvectors of ๐‘ƒ are vectors ๐‘ฅ which lie on the projection plane already. In that case ๐‘ƒ๐‘ฅ=๐‘ฅ and therefore ๐‘ฅ is an eigenvector with eigenvalue 1. We can find 2 independent eigenvectors of ๐‘ƒ which lie on the projection plane both associated with an eigenvalue of 1. Problem: In the 3D space we can find 3 independent vectors. Can you find a third eigenvector of ๐‘ƒ that is perpendicular to the eigenvectors of ๐‘ƒ that lie on the projection plane? Answer: YES! Any vector ๐‘’ perpendicular to the plane. In that case ๐‘ƒ๐‘’=๐ŸŽ=0๐‘’. Therefore, the eigenvalues of ๐‘ƒ are ๐œ†=0 and ๐œ†=1.

6 Eigenvectors and eigenvalues of a permutation matrix
Consider the permutation matrix ๐ด= Problem: Can you give an eigenvector of the above matrix? Or can you think of a vector that if permuted is still a multiple of itself? Answer: YES. It is the vector and the corresponding eigenvalue is ๐œ†=1. And furthermore, the vector โˆ’1 1 with eigenvalue ๐œ†=โˆ’1. ๐‘›ร—๐‘› matrices will have ๐‘› eigenvalues. It is not easy to find them. The sum of the eigenvalues, called the trace of a matrix, equals the sum of the diagonal elements of the matrix. The product of the eigenvalues equals the determinant of the matrix. Therefore, in the previous example, once I found an eigenvalue ๐œ†=1, I should know that there is another eigenvalue ๐œ†=โˆ’1.

7 Problem: Solve ๐ด๐‘ฅ=๐œ†๐‘ฅ Consider an eigenvector ๐‘ฅ of matrix ๐ด. In that case ๐ด๐‘ฅ=๐œ†๐‘ฅโ‡’๐ด๐‘ฅโˆ’๐œ†๐‘ฅ=๐ŸŽ (๐ŸŽ is the zero vector). Therefore, ๐ดโˆ’๐œ†๐ผ ๐‘ฅ=๐ŸŽ. In order for the above set of equations to have a non-zero solution, the nullspace of (๐ดโˆ’๐œ†๐ผ) must be non-zero, i.e., the matrix (๐ดโˆ’๐œ†๐ผ) must be singular. Therefore, det ๐ดโˆ’๐œ†๐ผ =0. I now have an equation for ๐œ†. It is called the characteristic equation, or the eigenvalue equation. From the roots of this equation we can find the eigenvalues. I might have repeated ๐œ†s. This might cause problems but I will deal with it later. After I find ๐œ†, I can find ๐‘ฅ from ๐ดโˆ’๐œ†๐ผ ๐‘ฅ=๐ŸŽ. Basically, I will be looking for the nullspace of ๐ดโˆ’๐œ†๐ผ . The eigenvalues of ๐ด ๐‘‡ are obtained through the equation det ๐ด ๐‘‡ โˆ’๐œ†๐ผ =0. But: det ๐ด ๐‘‡ โˆ’๐œ†๐ผ = det ๐ด ๐‘‡ โˆ’๐œ† ๐ผ ๐‘‡ = det ๐ดโˆ’๐œ†๐ผ ๐‘‡ =detโก(๐ดโˆ’๐œ†๐ผ). Therefore, the eigenvalues of ๐ด ๐‘‡ are the same as the eigenvalues of ๐ด.

8 Solve ๐ด๐‘ฅ=๐œ†๐‘ฅ. An example. Consider the matrix ๐ด= A symmetric matrix always has real eigenvalues. Furthermore, the eigenvectors of a symmetric matrix can be chosen to be orthogonal. det ๐ดโˆ’๐œ†๐ผ = 3โˆ’๐œ† 2 โˆ’1=0โ‡’3โˆ’๐œ†=ยฑ1โ‡’๐œ†=3ยฑ1โ‡’ ๐œ† 1 =4, ๐œ† 2 =2. Or det ๐ดโˆ’๐œ†๐ผ = ๐œ† 2 โˆ’6๐œ†+8=0. Note that 6= ๐œ† 1 + ๐œ† 2 and 8=det(๐ด)= ๐œ† 1 ๐œ† 2 . Find the eigenvector for ๐œ† 1 =4. ๐ดโˆ’4๐ผ= โˆ’1 1 1 โˆ’1 โ‡’ โˆ’1 1 1 โˆ’1 ๐‘ฅ ๐‘ฆ = โ‡’๐‘ฅ=๐‘ฆ Find the eigenvector for ๐œ† 2 =2. ๐ดโˆ’2๐ผ= โ‡’ ๐‘ฅ ๐‘ฆ = โ‡’๐‘ฅ=โˆ’๐‘ฆ Notice that there are families of eigenvectors, not single eigenvectors.

9 Compare the two matrices given previously
Consider the matrix ๐ด= As shown it has eigenvectors ๐‘ฅ ๐‘ฅ and โˆ’๐‘ฅ ๐‘ฅ with eigenvalues ๐œ† 1 =4 and ๐œ† 2 =2. Consider the matrix ๐ต= , also with eigenvectors ๐‘ฅ ๐‘ฅ and โˆ’๐‘ฅ ๐‘ฅ and eigenvalues ๐œ† 1 =1 and ๐œ† 2 =โˆ’1. We observe that ๐‘จ=๐‘ฉ+๐Ÿ‘๐‘ฐ. The eigenvalues of ๐‘จ are obtained from the eigenvalues of ๐‘ฉ if we increase them by 3. The eigenvectors of ๐‘จ and ๐‘ฉ are the same.

10 Generalization of the above observation
Consider the matrix ๐ด=๐ต+๐‘๐ผ. Consider an eigenvector ๐‘ฅ of ๐ต with eigenvalue ๐œ†. Then ๐ต๐‘ฅ=๐œ†๐‘ฅ and therefore, ๐ด๐‘ฅ= ๐ต+๐‘๐ผ ๐‘ฅ=๐ต๐‘ฅ+๐‘๐ผ๐‘ฅ=๐ต๐‘ฅ+๐‘๐‘ฅ=๐œ†๐‘ฅ+๐‘๐‘ฅ= ๐œ†+๐‘ ๐‘ฅ ๐ด has the same eigenvectors with ๐ต with eigenvalues ๐œ†+๐‘. There arenโ€™t any properties that enable us to find the eigenvalues of ๐‘จ+๐‘ฉ and ๐‘จ๐‘ฉ.

11 Example Take a matrix that rotates every vector by 90 o .
This is ๐‘„= cos(90) โˆ’sin(90) sin(90) cos(90) = 0 โˆ’1 1 0 ๐œ† 1 + ๐œ† 2 =0 and det ๐‘„ =๐œ† 1 ๐œ† 2 =1. What vector can be parallel to itself after rotation? det ๐‘„โˆ’๐œ†๐ผ =det โˆ’๐œ† โˆ’1 1 โˆ’๐œ† = ๐œ† 2 +1=0โ‡’๐œ†=ยฑ๐‘–. In that case we have a skew symmetric (or anti-symmetric) matrix with ๐‘„ ๐‘‡ = ๐‘„ โˆ’1 =โˆ’๐‘„. We observe that the eigenvalues are complex. Complex eigenvalues always come in complex conjugate pairs if the associated matrix is real.

12 Example Consider ๐ด= 3 1 0 3 . ๐œ† 1 + ๐œ† 2 =6 and det(๐œ† 1 ๐œ† 2 )=9.
det ๐ดโˆ’๐œ†๐ผ =det 3โˆ’๐œ† 1 0 3โˆ’๐œ† = (3โˆ’๐œ†) 2 =0โ‡’ ๐œ† 1,2 =3 The eigenvalues of a triangular matrix are the values of the diagonal. For that particular case we have ๐‘ฅ ๐‘ฆ = 3๐‘ฅ 3๐‘ฆ โ‡’๐‘ฆ=0 and ๐‘ฅ can be any number.

13 Matrix diagonalization The case of independent eigenvectors
Suppose we have ๐‘› independent eigenvectors of a matrix ๐ด. We call them ๐‘ฅ ๐‘– . The associated eigenvalues are ๐œ† ๐‘– . We put them in the columns of a matrix ๐‘†. We form the matrix: ๐ด๐‘†=๐ด ๐‘ฅ 1 ๐‘ฅ 2 โ€ฆ ๐‘ฅ ๐‘› = ๐œ† 1 ๐‘ฅ 1 ๐œ† 2 ๐‘ฅ 2 โ€ฆ ๐œ† ๐‘› ๐‘ฅ ๐‘› = ๐‘ฅ 1 ๐‘ฅ 2 โ€ฆ ๐‘ฅ ๐‘› ๐œ† โ€ฆ ๐œ† 2 โ€ฆ 0 โ‹ฎ โ‹ฎ โ‹ฑ โ€ฆ โ‹ฎ ๐œ† ๐‘› =๐‘†ฮ›โ‡’๐ด๐‘†=๐‘†ฮ› ๐‘† โˆ’1 ๐ด๐‘†=ฮ› or ๐ด= ๐‘†ฮ› ๐‘† โˆ’1 The above formulation of ๐ด is very important in Mathematics and Engineering and is called matrix diagonalization.

14 Matrix diagonalization: Eigenvalues of ๐ด ๐‘˜
If ๐ด๐‘ฅ=๐œ†๐‘ฅโ‡’ ๐ด 2 ๐‘ฅ=๐œ†๐ด๐‘ฅโ‡’ ๐ด 2 ๐‘ฅ= ๐œ† 2 ๐‘ฅ. Therefore, the eigenvalues of ๐ด 2 are ๐œ† 2 . The eigenvectors of ๐ด 2 remain the same since ๐ด 2 = ๐‘†ฮ› ๐‘† โˆ’1 ๐‘†ฮ› ๐‘† โˆ’1 =๐‘† ฮ› 2 ๐‘† โˆ’1 In general ๐ด ๐‘˜ = ๐‘† ฮ› ๐‘˜ ๐‘† โˆ’1 lim ๐ด ๐‘˜ =0 if the eigenvalues of ๐ด have the property ๐œ† ๐‘– <1. A matrix has ๐‘› independent eigenvectors and therefore is diagonalizable if all the eigenvalues are different. If there are repeated eigenvalues a matrix may, or may not have independent eigenvectors. As an example consider the identity matrix. Its eigenvectors are the row (or column) vectors. They are all independent. However, the eigenvalues are all equal to 1. Find the eigenvalues of ๐ด= 1 ๐‘Ž 0 1 ,๐‘Žโ‰ 0. This is a standard textbook matrix which cannot be diagonalized. (Answer: The eigenvectors are of the form [ ๐‘ฅ 0 ] ๐‘‡ )

15 Application of matrix diagonalization cont
Application of matrix diagonalization cont. A first order system which evolves with time Consider a system that follows an equation of the form ๐‘ข ๐‘˜+1 =๐ด ๐‘ข ๐‘˜ . ๐‘ข ๐‘˜ is the vector which consists of the system parameters which evolve with time. The eigenvalues of ๐ด characterize fully the behavior of the system. I start with a given vector ๐‘ข 0 . ๐‘ข 1 =๐ด ๐‘ข 0 , ๐‘ข 2 = ๐ด 2 ๐‘ข 0 and in general ๐‘ข ๐‘˜ = ๐ด ๐‘˜ ๐‘ข 0 In order to solve this system I assume that I can write ๐‘ข 0 = ๐‘ 1 ๐‘ฅ 1 +๐‘ 2 ๐‘ฅ 2 +โ€ฆ+ ๐‘ ๐‘› ๐‘ฅ ๐‘› where ๐‘ฅ ๐‘– are the eigenvectors of matrix ๐ด. This is a standard approach to the solution of this type of problem. ๐ด๐‘ข 0 = ๐‘ 1 ๐ด ๐‘ฅ 1 +๐‘ 2 ๐ด๐‘ฅ 2 +โ€ฆ+ ๐‘ ๐‘› ๐ด๐‘ฅ ๐‘› = ๐‘ 1 ๐œ† 1 ๐‘ฅ 1 +๐‘ 2 ๐œ† 2 ๐‘ฅ 2 +โ€ฆ+ ๐‘ ๐‘› ๐œ† ๐‘› ๐‘ฅ ๐‘› ๐ด 100 ๐‘ข 0 = ๐‘ ๐œ† ๐‘ฅ 1 +๐‘ 2 ๐œ† ๐‘ฅ 2 +โ€ฆ+ ๐‘ ๐‘› ๐œ† ๐‘› ๐‘ฅ ๐‘› = ๐‘†ฮ› 100 ๐‘ ๐‘ is a column vector that contains the coefficients ๐‘ ๐‘– . ๐‘†,ฮ› are the matrices defined previously.

16 Application of matrix diagonalization cont
Application of matrix diagonalization cont. Fibonacci example: Convert a second order scalar problem into a first order system I will take two numbers which I denote with ๐น 0 =0 and ๐น 1 =1. The Fibonacci sequence of numbers is given by the two initial numbers given above and the relationship ๐น ๐‘˜ =๐น ๐‘˜โˆ’1 + ๐น ๐‘˜โˆ’2 . The sequence is 0,1,1,2,3,5,8,13 and so on. How can I get a closed form formula for the 100th Fibonacci number or any Fibonacci number? I will present a standard approach to this problem. I define a vector ๐‘ข ๐‘˜ = ๐น ๐‘˜+1 ๐น ๐‘˜ and an extra equation ๐น ๐‘˜+1 =๐น ๐‘˜+1 . By merging the two equations I obtain the following matrix form: ๐‘ข ๐‘˜+1 = ๐น ๐‘˜+2 ๐น ๐‘˜+1 = ๐น ๐‘˜+1 ๐น ๐‘˜ =๐ด ๐‘ข ๐‘˜ I managed to convert the second order scalar problem into a first order matrix problem.

17 Application of matrix diagonalization cont. Fibonacci example cont
Application of matrix diagonalization cont. Fibonacci example cont. Convert a second order scalar problem into a first order system The eigenvalues of ๐ด are obtained from det 1โˆ’๐œ† 1 1 โˆ’๐œ† =โˆ’ 1โˆ’๐œ† ๐œ†โˆ’1=0โ‡’ ๐œ† 2 โˆ’๐œ†โˆ’1=0 Observe the analogy between ๐œ† 2 โˆ’๐œ†โˆ’1=0 and ๐น ๐‘˜ โˆ’๐น ๐‘˜โˆ’1 โˆ’ ๐น ๐‘˜โˆ’2 =0. ๐œ† 1,2 = 1ยฑ Eigenvalues add up to 1. The matrix ๐ด is diagonalizable. It can be shown that the eigenvectors are ๐‘ฅ 1 = ๐œ† and ๐‘ฅ 2 = ๐œ† How can I get a formula for the 100th Fibonacci number? ๐‘ข ๐‘˜+1 = ๐น ๐‘˜+2 ๐น ๐‘˜+1 =๐ด ๐‘ข ๐‘˜ = ๐ด 2 ๐‘ข ๐‘˜โˆ’1 etc. Therefore, ๐‘ข 100 = ๐ด 100 ๐‘ข 0 =๐‘† ฮ› 100 ๐‘† โˆ’1 ๐‘ข 0 ฮ› 100 = ๐œ† ๐œ† โ‰… ๐œ† , ๐‘†= ๐œ† 1 ๐œ† , ๐‘† โˆ’1 = 1 ๐œ† 1 โˆ’ ๐œ† โˆ’ ๐œ† 2 โˆ’1 ๐œ† 1 ๐‘ข 100 โ‰… 1 ๐œ† 1 โˆ’ ๐œ† ๐œ† 1 ๐œ† ๐œ† โˆ’ ๐œ† 2 โˆ’1 ๐œ† = 1 ๐œ† 1 โˆ’ ๐œ† ๐œ† 1 ๐œ† ๐œ† โˆ’1 = 1 ๐œ† 1 โˆ’ ๐œ† ๐œ† 1 ๐œ† ๐œ† = 1 ๐œ† 1 โˆ’ ๐œ† ๐œ† ๐œ† โ‡’ ๐น 100 = 1 ๐œ† 1 โˆ’ ๐œ† 2 ๐œ†

18 Applications of matrix diagonalization cont
Applications of matrix diagonalization cont. First order differential equations ๐‘‘๐‘ข ๐‘‘๐‘ก =๐ด๐‘ข Problem: Solve the system of differential equations: ๐‘‘ ๐‘ข 1 (๐‘ก) ๐‘‘๐‘ก =โˆ’ ๐‘ข 1 ๐‘ก +2 ๐‘ข 2 ๐‘ก , ๐‘‘ ๐‘ข 2 (๐‘ก) ๐‘‘๐‘ก = ๐‘ข 1 (๐‘ก)โˆ’2 ๐‘ข 2 (๐‘ก) Solution: We convert the system of equations into a matrix form ๐‘‘ ๐‘ข 1 (๐‘ก) ๐‘‘๐‘ก ๐‘‘ ๐‘ข 2 (๐‘ก) ๐‘‘๐‘ก =๐ด ๐‘ข 1 (๐‘ก) ๐‘ข 2 (๐‘ก) . We set ๐‘ข(๐‘ก)= ๐‘ข 1 (๐‘ก) ๐‘ข 2 (๐‘ก) . We impose the initial conditions ๐‘ข(0)= The systemโ€™s matrix is ๐ด= โˆ’1 2 1 โˆ’2 . The matrix ๐ด is singular. One of the eigenvalues is zero. Therefore, from the trace we conclude that the eigenvalues are ๐œ† 1 =0, ๐œ† 2 =โˆ’3. The solution of the above system depends exclusively on the eigenvalues of ๐ด. We can easily show that the eigenvectors are ๐‘ฅ 1 = and ๐‘ฅ 2 = 1 โˆ’1 .

19 Applications of matrix diagonalization cont
Applications of matrix diagonalization cont. First order differential equations ๐‘‘๐‘ข ๐‘‘๐‘ก =๐ด๐‘ข The solution of the above system of equations is of the form ๐‘ข ๐‘ก = ๐‘ 1 ๐‘’ ๐œ† 1 ๐‘ก ๐‘ฅ 1 + ๐‘ 2 ๐‘’ ๐œ† 2 ๐‘ก ๐‘ฅ 2 Problem: Verify the above by plugging-in ๐‘’ ๐œ† ๐‘– ๐‘ก ๐‘ฅ ๐‘– to the equation ๐‘‘๐‘ข ๐‘‘๐‘ก =๐ด๐‘ข. Letโ€™s find ๐‘ข ๐‘ก = ๐‘ 1 ๐‘’ 0๐‘ก ๐‘ 2 ๐‘’ โˆ’3๐‘ก 1 โˆ’1 = ๐‘ ๐‘ 2 ๐‘’ โˆ’3๐‘ก 1 โˆ’1 ๐‘ 1 , ๐‘ 2 comes from the initial conditions. ๐‘ ๐‘ โˆ’1 = โ‡’ โˆ’1 ๐‘ 1 ๐‘ 2 = โ‡’๐‘† ๐‘ 1 ๐‘ 2 =๐‘ข(0)โ‡’ ๐‘ 1 = 1 3 , ๐‘ 2 = 1 3 . The Steady State of the system is defined as ๐‘ข โˆž = 2/3 1/3 . Stability is achieved if the real part of the eigenvalues is negative. Note that if complex eigenvalues exists, they appear in conjugate pairs.

20 Applications of matrix diagonalization cont
Applications of matrix diagonalization cont. First order differential equations ๐‘‘๐‘ข ๐‘‘๐‘ก =๐ด๐‘ข Stability is achieved if the real part of the eigenvalues is negative. We do have a steady state if at least one eigenvalue is 0 and the rest of the eigenvalues have negative real part. The system is unstable if at least one eigenvalue has a positive real part. For stability the trace of the systemโ€™s matrix must be negative. The reverse is not true. Obviously a negative trace does not guarantee stability.

21 Applications of matrix diagonalization cont
Applications of matrix diagonalization cont. First order differential equations ๐‘‘๐‘ข ๐‘‘๐‘ก =๐ด๐‘ข Consider a first-order differential equation ๐‘‘๐‘ข ๐‘‘๐‘ก =๐ด๐‘ข. Recall the matrix ๐‘† which contains the eigenvectors of ๐ด defined previously. I set ๐‘ข(๐‘ก)=๐‘†๐‘ฃ(๐‘ก) and therefore the differential equation becomes: ๐‘† ๐‘‘๐‘ฃ(๐‘ก) ๐‘‘๐‘ก =๐ด๐‘†๐‘ฃ(๐‘ก)โ‡’ ๐‘‘๐‘ฃ ๐‘ก ๐‘‘๐‘ก = ๐‘† โˆ’1 ๐ด๐‘†๐‘ฃ(๐‘ก)=ฮ›๐‘ฃ(๐‘ก) This is an interesting result which can be found in various engineering applications. I start from a system of equations ๐‘‘๐‘ข(๐‘ก) ๐‘‘๐‘ก =๐ด๐‘ข(๐‘ก) which are coupled (or dependent or correlated) and I end up with a set of equations which are decoupled and easier to solve. From the relationship ๐‘‘๐‘ฃ(๐‘ก) ๐‘‘๐‘ก =ฮ›๐‘ฃ(๐‘ก) we get ๐‘ฃ ๐‘ก = ๐‘’ ฮ›๐‘ก ๐‘ฃ(0) and ๐‘ข ๐‘ก = ๐‘†๐‘’ ฮ›๐‘ก ๐‘† โˆ’1 ๐‘ข(0) with ๐‘’ ๐ด๐‘ก = ๐‘†๐‘’ ฮ›๐‘ก ๐‘† โˆ’1 Question: What is the exponential of a matrix? See next slide.

22 Applications of matrix diagonalization cont
Applications of matrix diagonalization cont. Second order homogeneous differential equations How do I change the second order homogeneous differential equation ๐‘ฆ โ€ฒโ€ฒ (๐‘ก)+๐‘ ๐‘ฆ โ€ฒ (๐‘ก)+๐‘˜๐‘ฆ(๐‘ก)=0 to two first order ones? I use the equation above and the additional equation ๐‘ฆ โ€ฒ ๐‘ก = ๐‘ฆ โ€ฒ (๐‘ก). I define ๐‘ข(๐‘ก)= ๐‘ฆ โ€ฒ (๐‘ก) ๐‘ฆ(๐‘ก) and from the two available equations I form the matrix representation shown below. ๐‘ข โ€ฒ (๐‘ก)= ๐‘ฆ โ€ฒโ€ฒ (๐‘ก) ๐‘ฆ โ€ฒ (๐‘ก) = โˆ’๐‘ โˆ’๐‘˜ ๐‘ฆ โ€ฒ (๐‘ก) ๐‘ฆ(๐‘ก) ๐‘ข โ€ฒ (๐‘ก)= โˆ’๐‘ โˆ’๐‘˜ 1 0 ๐‘ข(๐‘ก)

23 Applications of matrix diagonalization cont
Applications of matrix diagonalization cont. Higher order homogeneous differential equations How do I change the ๐‘› th order homogeneous differential equation ๐‘ฆ ๐‘› (๐‘ก)+ ๐‘ 1 ๐‘ฆ ๐‘›โˆ’1 (๐‘ก)+โ€ฆ+ ๐‘ ๐‘›โˆ’1 ๐‘ฆ(๐‘ก)=0 to ๐‘› first order ones? I define ๐‘ข(๐‘ก)= ๐‘ฆ ๐‘›โˆ’1 (๐‘ก) โ‹ฎ ๐‘ฆ โ€ฒ (๐‘ก) ๐‘ฆ(๐‘ก) and ๐‘› additional equations ๐‘ฆ ๐‘– ๐‘ก = ๐‘ฆ ๐‘– ๐‘ก , ๐‘–=1,โ€ฆ,(๐‘›โˆ’1) and therefore, I obtain the matrix representation below: ๐‘ข โ€ฒ (๐‘ก)= ๐‘ฆ ๐‘› (๐‘ก) ๐‘ฆ ๐‘›โˆ’1 (๐‘ก) โ‹ฎ ๐‘ฆ โ€ฒโ€ฒ (๐‘ก) ๐‘ฆ โ€ฒ (๐‘ก) = โˆ’ ๐‘ 1 โˆ’ ๐‘ 2 โ€ฆ โˆ’ ๐‘ ๐‘›โˆ’2 โˆ’ ๐‘ ๐‘›โˆ’ โ‹ฎ 0 โ€ฆ โ‹ฎ โ€ฆ โ‹ฎ 0 โ‹ฎ โ€ฆ 0 โ€ฆ ๐‘ข(๐‘ก) As previously analysed the solution of this system is explicitly affiliated to the eigenvalues of the systemโ€™s matrix.

24 Diagonal matrix exponentials
The exponential ๐‘’ ฮ›๐‘ก of a diagonal matrix is given by: ๐‘’ ฮ›๐‘ก = ๐‘’ ๐œ† 1 ๐‘ก 0 โ€ฆ ๐‘’ ๐œ† 2 ๐‘ก โ€ฆ 0 โ‹ฎ โ‹ฎ โ‹ฑ โ€ฆ โ‹ฎ ๐‘’ ๐œ† ๐‘› ๐‘ก with ๐œ† ๐‘– being the elements of the diagonal. Furthermore, ฮ› ๐‘ก = ๐œ† 1 ๐‘ก 0 โ€ฆ ๐œ† 2 ๐‘ก โ€ฆ 0 โ‹ฎ โ‹ฎ โ‹ฑ โ€ฆ โ‹ฎ ๐œ† ๐‘› ๐‘ก As we already showed lim ๐‘กโ†’โˆž ๐‘’ ฮ›๐‘ก =0 if Re ๐œ† ๐‘– <0, โˆ€๐‘– lim ๐‘กโ†’โˆž ฮ› ๐‘ก =0 if ๐œ† ๐‘– <1, โˆ€๐‘–

25 Matrix exponentials ๐‘’ ๐ด๐‘ก
Consider the exponential ๐‘’ ๐ด๐‘ก . The Taylor series expansion is ๐‘’ ๐‘ฅ = 0 โˆž ๐‘ฅ ๐‘› ๐‘›! . Similarly, ๐‘’ ๐ด๐‘ก =๐ผ+๐ด๐‘ก+ (๐ด๐‘ก) (๐ด๐‘ก) โ€ฆ+ ๐ด๐‘ก ๐‘› ๐‘›! +โ€ฆ ๐‘’ ๐ด๐‘ก =๐ผ+๐‘†ฮ› ๐‘† โˆ’1 ๐‘ก+ ๐‘† ฮ› 2 ๐‘† โˆ’1 ๐‘ก ๐‘† ฮ› 3 ๐‘† โˆ’1 ๐‘ก โ€ฆ+ ๐‘† ฮ› ๐‘› ๐‘† โˆ’1 ๐‘ก ๐‘› ๐‘›! +โ€ฆ=๐‘† ๐‘’ ฮ›๐‘ก ๐‘† โˆ’1 The assumption built-in to the above formula is that ๐ด must be diagonalizable. Furthermore, note that 1 1โˆ’๐‘ฅ = 0 โˆž ๐‘ฅ ๐‘› . For matrices we have (๐ผโˆ’๐ด๐‘ก) โˆ’1 =๐ผ+๐ด๐‘ก+ (๐ด๐‘ก) 2 + (๐ด๐‘ก) 3 +โ€ฆ This sum converges if the magnitudes of the eigenvalues of matrix ๐ด๐‘ก are of magnitude less than 1, i.e., ๐œ†(๐ด๐‘ก) <1. This statement is proven using diagonalization.

26 Stochastic matrices Consider a matrix ๐ด with the following properties:
It is square All entries are positive and real. The elements of each column or each row or both each column and each row add up to 1. Based on the above, a matrix that exhibits the above properties will have all entries โ‰ค 1. This is called a stochastic matrix. Stochastic matrices are also called Markov, probability, transition, or substitution matrices. The entries of a stochastic matrix usually represent a probability. Stochastic matrices are widely used inย probability theory,ย statistics,ย mathematical financeย andย linear algebra, as well asย computer science.

27 Stochastic matrices. Types.
There are several types of stochastic matrices: Aย right stochastic matrixย is a matrix of nonnegative real entries, with each rowโ€™s elements summing to 1. Aย left stochastic matrixย is a matrix of nonnegative real entries, with each columnโ€™s elements summing to 1. Aย doubly stochastic matrixย is a matrix of nonnegative real entries with each rowโ€™s and each columnโ€™s elements summing to 1. A stochastic matrix often describes aย so called Markov chainย  ๐‘‹ ๐‘ก ย over aย finiteย state spaceย ๐‘†. Generally, an ๐‘›ร—๐‘› stochastic matrix is related to ๐‘› โ€œstatesโ€. If theย probabilityย of moving from stateย ๐‘–ย to stateย ๐‘— isย  ๐‘ƒ ๐‘Ÿ ๐‘— ๐‘– = ๐‘ ๐‘–๐‘— , the stochastic matrix ๐‘ƒย is given by usingย  ๐‘ ๐‘–๐‘— ย as theย  ๐‘– th ย row andย  ๐‘— th ย column element: ๐‘ƒ= ๐‘ 11 ๐‘ 12 ๐‘ 21 ๐‘ โ€ฆ ๐‘ 1๐‘› โ€ฆ ๐‘ 2๐‘› โ‹ฎ โ‹ฎ ๐‘ ๐‘›1 ๐‘ ๐‘›2 โ‹ฑ โ‹ฎ โ€ฆ ๐‘ ๐‘›๐‘› Depending on the particular problem, the above matrix can be formulated in such a way so that it is either right or left stochastic.

28 Products of stochastic matrices. Stochastic vectors.
An example of a left stochastic matrix is the following: ๐ด= You can prove that if ๐ด and ๐ต are stochastic matrices of any type, then ๐ด๐ต is also a stochastic matrix of the same type. Consider two left stochastic matrices ๐ด and ๐ต with elements ๐‘Ž ๐‘–๐‘— and ๐‘ ๐‘–๐‘— respectively, and ๐ถ=๐ด๐ต with elements ๐‘ ๐‘–๐‘— . Let us find the sum of the elements of the ๐‘— th column of ๐ถ: ๐‘–=1 ๐‘› ๐‘ ๐‘–๐‘— = ๐‘–=1 ๐‘› ๐‘˜=1 ๐‘› ๐‘Ž ๐‘–๐‘˜ ๐‘ ๐‘˜๐‘— = ๐‘˜=1 ๐‘› ๐‘–=1 ๐‘› ๐‘Ž ๐‘–๐‘˜ ๐‘ ๐‘˜๐‘— = ๐‘˜=1 ๐‘› ๐‘ ๐‘˜๐‘— ๐‘–=1 ๐‘› ๐‘Ž ๐‘–๐‘˜ = 1โˆ™1=1 Based on the above any power of a stochastic matrix is a stochastic matrix. Furthermore, a vector with real, nonnegative entries ๐‘ ๐‘˜ , for which all the ๐‘ ๐‘˜ add up to 1, is called a stochastic vector. For a stochastic matrix, every column or row or both is a stochastic vector. I am interested in the eigenvalues and eigenvectors of a stochastic matrix.

29 Stochastic matrices and their eigenvalues
I would like to prove that ๐œ†=1 is always an eigenvalue of a stochastic matrix. Consider again a left stochastic matrix ๐ด. Since the elements of each column of ๐ด add up to 1, the elements of each row of ๐ด ๐‘‡ should add up to 1. Therefore, ๐ด ๐‘‡ โ‹ฎ 1 = โ‹ฎ 1 =1โˆ™ โ‹ฎ 1 Therefore, 1 is an eigenvalue of ๐ด ๐‘‡ . As already shown, the eigenvalues of ๐ด and ๐ด ๐‘‡ are the same, which implies that 1 is also an eigenvalue of ๐ด. Since det ๐ดโˆ’๐ผ =0, the matrix ๐ดโˆ’๐ผ is singular, which means that there is a vector ๐‘ฅ for which ๐ดโˆ’๐ผ ๐‘ฅ=๐ŸŽโ‡’๐ด๐‘ฅ=๐‘ฅ A vector of the null space of ๐ดโˆ’๐ผ is the eigenvector of ๐ด that corresponds to eigenvalue ๐œ†=1.

30 Stochastic matrices and their eigenvalues cont.
I would like now to prove that the eigenvalues of a stochastic matrix have magnitude smaller or equal to 1. Assume that ๐‘ฃ= ๐‘ฃ 1 ๐‘ฃ 2 โ‹ฎ ๐‘ฃ ๐‘› is an eigenvector of a right stochastic matrix ๐ด with an associated eigenvalue ๐œ† >1. Then ๐ด๐‘ฃ=๐œ†๐‘ฃ implies ๐‘—=1 ๐‘› ๐ด ๐‘–๐‘— ๐‘ฃ ๐‘— =๐œ† ๐‘ฃ ๐‘– . We can see that ๐‘—=1 ๐‘› ๐ด ๐‘–๐‘— ๐‘ฃ ๐‘— โ‰ค ๐‘ฃ max ๐‘—=1 ๐‘› ๐ด ๐‘–๐‘— =๐‘ฃ max . All the elements of the vector ๐ด๐‘ฃ are smaller or equal to ๐‘ฃ max . If ๐œ† >1 at least one element of the vector ๐œ†๐‘ฃ (which is equal to ๐ด๐‘ฃ) will be greater than ๐‘ฃ max (๐œ† ๐‘ฃ max ). Based on the above, the assumption of an eigenvalue being larger than 1 can not be valid. For left stochastic matrices we use ๐ด ๐‘‡ in the proof above, and the fact that the eigenvalues of ๐ด and ๐ด ๐‘‡ are the same.

31 An application of stochastic matrices: First order systems
Consider again the first order system described by an equation of the form ๐‘ข ๐‘˜ = ๐ด ๐‘˜ ๐‘ข 0 , where ๐ด is now a stochastic matrix. Previously, we managed to write ๐‘ข ๐‘˜ = ๐ด ๐‘˜ ๐‘ข 0 = ๐‘ 1 ๐œ† 1 ๐‘˜ ๐‘ฅ 1 + ๐‘ 2 ๐œ† 2 ๐‘˜ ๐‘ฅ 2 +โ€ฆ where ๐œ† ๐‘– and ๐‘ฅ ๐‘– are the eigenvalues and eigenvectors of matrix ๐ด, respectively. Note that the above relationship requires a complete set of eigenvectors. If ๐œ† 1 =1 and ๐œ† ๐‘– <1, ๐‘–>1 then the steady state of the system is ๐‘ 1 ๐‘ฅ 1 (which is part of the initial condition ๐‘ข 0 ). I will use an example where ๐ด is a 2ร—2 matrix. Generally, an ๐‘›ร—๐‘› stochastic matrix is related to ๐‘› โ€œstatesโ€. Assume that the 2 โ€œstatesโ€ are 2 UK cities. I take London and Oxford. I am interested in the population of the two cities and how it evolves. I assume that people who inhabit these two cities move between them only. ๐‘ข ox ๐‘ข lon ๐‘ก=๐‘˜+1 = ๐‘ข ox ๐‘ข lon ๐‘ก=๐‘˜ It is now obvious that the column elements are positive and also add up to 1 because they represent probabilities.

32 Application of stochastic matrices (cont.)
I assume that ๐‘ข ox ๐‘ข lon ๐‘ก=๐‘˜=0 = Then ๐‘ข ox ๐‘ข lon ๐‘˜=1 = = What is the population of the two cities after a long time? Consider the matrix The eigenvalues are ๐œ† 1 =1 and ๐œ† 2 =0.7. (Notice that the second eigenvalue is found by the trace of the matrix.) The eigenvectors of this matrix are ๐‘ฅ 1 = and ๐‘ฅ 2 = โˆ’1 1 . ๐‘ข ox ๐‘ข lon ๐‘˜ = ๐‘ 1 ๐œ† 1 ๐‘˜ ๐‘ 2 ๐œ† 2 ๐‘˜ โˆ’1 1 = ๐‘ ๐‘ ๐‘˜ โˆ’1 1 I find ๐‘ 1 , ๐‘ 2 from the initial condition ๐‘ข ox ๐‘ข lon ๐‘˜=0 = = ๐‘ ๐‘ 2 โˆ’1 1 and therefore, ๐‘ 1 = and ๐‘ 2 = ๐‘ข ox ๐‘ข lon ๐‘˜โ†’โˆž = ๐‘˜โ†’โˆž โˆ’1 1 =

33 Application of stochastic matrices (cont.)
Stochastic models facilitate the modeling of various real life engineering applications. An example is the modeling of the movement of people without gain or loss: total number of people is conserved.

34 Symmetric matrices In this lecture we will be interested in symmetric matrices. In case of real matrices, symmetry is defined as ๐ด= ๐ด ๐‘‡ . In case of complex matrices, symmetry is defined as ๐ด โˆ— = ๐ด ๐‘‡ or ๐ด โˆ— ๐‘‡ =๐ด. A matrix which possesses this property is called Hermitian. We can also use the symbol ๐ด ๐ป =๐ด โˆ— ๐‘‡ . We will prove that the eigenvalues of a symmetric matrix are real. The eigenvectors of a symmetric matrix can be chosen to be orthogonal. If we also choose them to have a magnitude of 1, then the eigenvectors can be chosen to form an orthonormal set of vectors. However, the eigenvectors of a symmetric matrix that correspond to different eigenvalues are orthogonal (prove is given in subsequent slide). For a random matrix with independent eigenvectors we have ๐ด=๐‘†ฮ› ๐‘† โˆ’1 . For a symmetric matrix with orthonormal eigenvectors we have ๐ด=๐‘„ฮ› ๐‘„ โˆ’1 = ๐‘„ฮ› ๐‘„ ๐‘‡

35 Real symmetric matrices
Problem: Prove that the eigenvalues of a symmetric matrix occur in complex conjugate pairs. Solution: Consider ๐ด๐‘ฅ=๐œ†๐‘ฅ. If we take complex conjugate in both sides we get (๐ด๐‘ฅ) โˆ— = (๐œ†๐‘ฅ) โˆ— โ‡’๐ด โˆ— ๐‘ฅ โˆ— =๐œ† โˆ— ๐‘ฅ โˆ— If ๐ด is real then ๐ด ๐‘ฅ โˆ— =๐œ† โˆ— ๐‘ฅ โˆ— . Therefore, if ๐œ† is an eigenvalue of ๐ด with corresponding eigenvector ๐‘ฅ then ๐œ† โˆ— is an eigenvalue of ๐ด with corresponding eigenvector ๐‘ฅ โˆ— .

36 Real symmetric matrices cont.
Problem: Prove that the eigenvalues of a symmetric matrix are real. Solution: We proved that if ๐ด is real then ๐ด ๐‘ฅ โˆ— =๐œ† โˆ— ๐‘ฅ โˆ— . If we take transpose in both sides we get ๐‘ฅ โˆ— ๐‘‡ ๐ด ๐‘‡ = ๐œ† โˆ— ๐‘ฅ โˆ— ๐‘‡ โ‡’ ๐‘ฅ โˆ— ๐‘‡ ๐ด= ๐œ† โˆ— ๐‘ฅ โˆ— ๐‘‡ We now multiply both sides from the right with ๐‘ฅ and we get ๐‘ฅ โˆ— ๐‘‡ ๐ด๐‘ฅ= ๐œ† โˆ— ๐‘ฅ โˆ— ๐‘‡ ๐‘ฅ We take now ๐ด๐‘ฅ=๐œ†๐‘ฅ. We now multiply both sides from the left with ๐‘ฅ โˆ— ๐‘‡ and we get ๐‘ฅ โˆ— ๐‘‡ ๐ด๐‘ฅ= ๐œ† ๐‘ฅ โˆ— ๐‘‡ ๐‘ฅ. From the above we see that ๐œ† ๐‘ฅ โˆ— ๐‘‡ ๐‘ฅ= ๐œ† โˆ— ๐‘ฅ โˆ— ๐‘‡ ๐‘ฅ and since ๐‘ฅ โˆ— ๐‘‡ ๐‘ฅโ‰ 0, we see that ๐œ†= ๐œ† โˆ— .

37 Real symmetric matrices cont.
Problem: Prove that the eigenvectors of a symmetric matrix which correspond to different eigenvalues are always perpendicular. Solution: Suppose that ๐ด๐‘ฅ= ๐œ† 1 ๐‘ฅ and ๐ด๐‘ฆ= ๐œ† 2 ๐‘ฆ with ๐œ† 1 โ‰ ๐œ† 2 . ๐œ† 1 ๐‘ฅ ๐‘‡ ๐‘ฆ= ๐‘ฅ ๐‘‡ ๐œ† 1 ๐‘ฆ= ๐ด๐‘ฅ ๐‘‡ ๐‘ฆ= ๐‘ฅ ๐‘‡ ๐ด๐‘ฆ= ๐‘ฅ ๐‘‡ ๐œ† 2 ๐‘ฆ The conditions ๐‘ฅ ๐‘‡ ๐œ† 1 ๐‘ฆ= ๐‘ฅ ๐‘‡ ๐œ† 2 ๐‘ฆ and ๐œ† 1 โ‰ ๐œ† 2 give ๐‘ฅ ๐‘‡ ๐‘ฆ=0. The eigenvectors ๐‘ฅ and ๐‘ฆ are perpendicular.

38 Complex matrices. Complex symmetric matrices.
Let us find which complex matrices have real eigenvalues and orthogonal eigenvectors. Consider ๐ด๐‘ฅ=๐œ†๐‘ฅ with ๐ด possibly complex. If we take complex conjugate in both sides we get (๐ด๐‘ฅ) โˆ— = (๐œ†๐‘ฅ) โˆ— โ‡’๐ด โˆ— ๐‘ฅ โˆ— =๐œ† โˆ— ๐‘ฅ โˆ— If we take transpose in both sides we get ๐‘ฅ โˆ— ๐‘‡ ๐ด โˆ— ๐‘‡ = ๐œ† โˆ— ๐‘ฅ โˆ— ๐‘‡ We now multiply both sides from the right with ๐‘ฅ we get ๐‘ฅ โˆ— ๐‘‡ ๐ด โˆ— ๐‘‡ ๐‘ฅ= ๐œ† โˆ— ๐‘ฅ โˆ— ๐‘‡ ๐‘ฅ We take now ๐ด๐‘ฅ=๐œ†๐‘ฅ. We now multiply both sides from the left with ๐‘ฅ โˆ— ๐‘‡ and we get ๐‘ฅ โˆ— ๐‘‡ ๐ด๐‘ฅ= ๐œ† ๐‘ฅ โˆ— ๐‘‡ ๐‘ฅ. From the above we see that if ๐ด โˆ— ๐‘‡ =๐ด then ๐œ† ๐‘ฅ โˆ— ๐‘‡ ๐‘ฅ= ๐œ† โˆ— ๐‘ฅ โˆ— ๐‘‡ ๐‘ฅ and since ๐‘ฅ โˆ— ๐‘‡ ๐‘ฅโ‰ 0, we see that ๐œ†= ๐œ† โˆ— . If ๐ด โˆ— ๐‘‡ =๐ด the matrix is called Hermitian.

39 Complex vectors and matrices
Consider a complex column vector ๐‘ง= ๐‘ง 1 ๐‘ง 2 โ€ฆ ๐‘ง ๐‘› ๐‘‡ . Its length is ๐‘ง โˆ— ๐‘‡ ๐‘ง= ๐‘–=1 ๐‘› ๐‘ง ๐‘– As already mentioned, when we both transpose and conjugate we can use the symbol ๐‘ง ๐ป =๐‘ง โˆ— ๐‘‡ (Hermitian). The inner product of two complex vectors is ๐‘ฆ โˆ— ๐‘‡ ๐‘ฅ= ๐‘ฆ ๐ป ๐‘ฅ. For complex matrices the symmetry is defined as ๐ด โˆ— ๐‘‡ =๐ด. As already mentioned, these are called Hermitian matrices. They have real eigenvalues and perpendicular eigenvectors. If these are complex we check their length using ๐‘ž ๐‘– โˆ— ๐‘‡ ๐‘ž ๐‘– and also ๐‘„ โˆ— ๐‘‡ ๐‘„=๐ผ. Example: Consider the matrix ๐ด= 2 3+๐‘– 3โˆ’๐‘– 5 Eigenvalues are found from: 2โˆ’๐œ† 5โˆ’๐œ† โˆ’ 3+๐‘– 3โˆ’๐‘– =0 โ‡’ ๐œ† 2 โˆ’7๐œ†+10โˆ’ 9โˆ’3๐‘–+3๐‘–โˆ’ ๐‘– 2 =0โ‡’๐œ† ๐œ†โˆ’7 =0

40 Eigenvalue sign We proved that:
The eigenvalues of a symmetric matrix, either real or complex, are real. The eigenvectors of a symmetric matrix can be chosen to be orthogonal. The eigenvectors of a symmetric matrix that correspond to different eigenvalues are orthogonal. Do not forget the definition of symmetry for complex matrices. It can be proven that the signs of the pivots are the same as the signs of the eigenvalues. Just to remind you: Product of pivots=Product of eigenvalues=Determinant


Download ppt "Maths for Signals and Systems Linear Algebra in Engineering Lectures 10-12, Tuesday 1st and Friday 4th November2016 DR TANIA STATHAKI READER (ASSOCIATE."

Similar presentations


Ads by Google