III. Multi-Dimensional Random Variables and Application in Vector Quantization

© Tallal Elshabrawy 2

3

4

5

6

7 Karhunen-Loeve Decomposition v1v1 u2u2 X=[X 1, X 2 ] X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let u 1, u 2 (unit vectors) be a new co-ordinate system such that u 1, u 2 are the eigen vectors of the covariance matrix R X u1u1 v2v2 u 1, u 2 are orthogonal to each other For distinct eigen vectors, values

© Tallal Elshabrawy 8 Karhunen-Loeve Decomposition v1v1 u2u2 X=[X 1, X 2 ] X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let u 1, u 2 (unit vectors) be a new co-ordinate system such that u 1, u 2 are the eigen vectors of the covariance matrix R X u1u1 v2v2 Representation of X over u 1, u 2 Projection of X over u 1 unit vector u 1 Projection of X over u 2 unit vector u 2

© Tallal Elshabrawy 9 Karhunen-Loeve Decomposition v1v1 u2u2 X=[X 1, X 2 ] X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let u 1, u 2 (unit vectors) be a new co-ordinate system such that u 1, u 2 are the eigen vectors of the covariance matrix R X u1u1 v2v2 Diagonalization of R X

© Tallal Elshabrawy 10 Karhunen-Loeve Decomposition v1v1 u2u2 X=[X 1, X 2 ] X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let u 1, u 2 (unit vectors) be a new co-ordinate system such that u 1, u 2 are the eigen vectors of the covariance matrix R X u1u1 v2v2 Diagonalization of R X

© Tallal Elshabrawy 11 Karhunen-Loeve Decomposition v1v1 u2u2 X=[X 1, X 2 ] X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let u 1, u 2 (unit vectors) be a new co-ordinate system such that u 1, u 2 are the eigen vectors of the covariance matrix R X u1u1 v2v2 Diagonalization of R X

© Tallal Elshabrawy 12 K-L Transformation of 2-D Random Variable v1v1 u2u2 X X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let Y = [Y 1, Y 2 ] be the transformation of the random variable X over the co- ordinate system u 1, u 2 u1u1 v2v2 X1X1 X2X2 Y Y1Y1 Y2Y2

© Tallal Elshabrawy 13 K-L Transformation of 2-D Random Variable v1v1 u2u2 X X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let Y = [Y 1, Y 2 ] be the transformation of the random variable X over the co- ordinate system u 1, u 2 u1u1 v2v2 X1X1 X2X2 Y Y1Y1 Y2Y2 Define E[X 1 ] E[X 2 ] E[Y 1 ] E[Y 2 ]

© Tallal Elshabrawy 14 K-L Transformation of 2-D Random Variable v1v1 u2u2 X X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let Y = [Y 1, Y 2 ] be the transformation of the random variable X over the co- ordinate system u 1, u 2 u1u1 v2v2 X1X1 X2X2 Y Y1Y1 Y2Y2 Covariance Matrix R X

© Tallal Elshabrawy 15 K-L Transformation of 2-D Random Variable v1v1 u2u2 X X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let Y = [Y 1, Y 2 ] be the transformation of the random variable X over the co- ordinate system u 1, u 2 u1u1 v2v2 X1X1 X2X2 Y Y1Y1 Y2Y2 Covariance Matrix R Y

© Tallal Elshabrawy 16 K-L Transformation of 2-D Random Variable v1v1 u2u2 X X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let Y = [Y 1, Y 2 ] be the transformation of the random variable X over the co- ordinate system u 1, u 2 u1u1 v2v2 X1X1 X2X2 Y Y1Y1 Y2Y2 Covariance Matrix R Y

© Tallal Elshabrawy 17 K-L Transformation of 2-D Random Variable v1v1 u2u2 X X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let Y = [Y 1, Y 2 ] be the transformation of the random variable X over the co- ordinate system u 1, u 2 u1u1 v2v2 X1X1 X2X2 Y Y1Y1 Y2Y2 Y 1 and Y 2 are UNCORRELATED Covariance Matrix R Y

© Tallal Elshabrawy 18 K-L Transformation of 2-D Random Variable Therefore using principle component analysis, it is possible to transform a random variable X with components X 1 and X 2 that are correlated into another random variable Y whose components Y 1 and Y 2 are uncorrelated v1v1 u2u2 X u1u1 v2v2 X1X1 X2X2 Y Y1Y1 Y2Y2 u 1, u 2 are eigen vectors of R X λ 1, λ 2 are eigen values of R X

© Tallal Elshabrawy 19 Two-Dimensional Gaussian Distribution In the previous slides, we have talked about the transformed random variable Y whose components are uncorrelated and have mean m Y and covariance matrix R Y. What would be the distribution of Y. Well this depends on what is the distribution of X Generally X and Y do not follow a Gaussian distribution. However, if X is a two-dimensional Gaussian distribution then Y as well would be a two-dimensional Gaussian distribution

© Tallal Elshabrawy 20 Two-Dimensional Gaussian Distribution Suppose we have a two-dimensional random variable Y whose components Y 1 and Y 2 are independent and Gaussian

© Tallal Elshabrawy 21 Two-Dimensional Gaussian Distribution Suppose we have a two-dimensional random variable Y whose components Y 1 and Y 2 are independent and Gaussian

© Tallal Elshabrawy 22 Two-Dimensional Gaussian Distribution Suppose we have a two-dimensional random variable Y whose components Y 1 and Y 2 are independent and Gaussian

© Tallal Elshabrawy 23 Two-Dimensional Gaussian Distribution Suppose we have a two-dimensional random variable Y whose components Y 1 and Y 2 are independent and Gaussian This formula is valid for any multi- dimensional Gaussian random variable whether its components are correlated or not

© Tallal Elshabrawy 24 Two-Dimensional Gaussian Distribution

© Tallal Elshabrawy 25 Two-Dimensional Gaussian Distribution

© Tallal Elshabrawy 26 Two-Dimensional Gaussian Distribution

© Tallal Elshabrawy 27 Two-Dimensional Gaussian Distribution

© Tallal Elshabrawy 28 Two-Dimensional Gaussian Distribution

© Tallal Elshabrawy 29 Two-Dimensional Gaussian Distribution

© Tallal Elshabrawy 30 Two-Dimensional Gaussian Distribution

© Tallal Elshabrawy 31 Two-Dimensional Gaussian Distribution

© Tallal Elshabrawy 32 Two-Dimensional Gaussian Distribution

© Tallal Elshabrawy 33 Two-Dimensional Gaussian Distribution

© Tallal Elshabrawy 34 Two-Dimensional Gaussian Distribution

© Tallal Elshabrawy 35 Two-Dimensional Gaussian Distribution More Energy Less Energy Y 1 and Y 2 Correlated

© Tallal Elshabrawy 36 Two-Dimensional Gaussian Distribution Equal Energy By rotating the axis, the resultant transformed random variables are uncorrelated