Mathematical Preliminaries

37 Matrix Theory Vectors nth element of vector u : u(n) Matrix mth row and nth column of A : a(m,n) column vector

38 Row-ordered form of a matrix Column-ordered form of a matrix Lexicographic Ordering(Stacking operation)

39

40 Transposition and conjugation rules Toeplitz matrices Circulant matrices

41 Linear convolution using Toeplitz matrix

42 (Toepliz matrix)

43 N-point circular convolution : h(n) N x(n) N Circular convolution using circulant matrix

44 (circulant matrix) Circular convolution + zero padding linear convolution Circular convolution with the period : the same result with that of linear convolution

45 (ex) Linear convolution as a Toeplitz matrix operation (ex) Circular convolution as a circulant matrix operation

46 Orthogonal and unitary matrices Orthogonal : Unitary : Positive definiteness and quadratic forms is called positive definite, if is a Hermitian matrix and is called positive semidefinite(nonnegative), if is a Hermitian matrix and Theorem if is a symmetric positive definite matrix, then all its eigenvalues are positive and the determinant of satisfies

47 Diagonal forms For any Hermitian matrix there exists a unitary matrix such that Eigenvalue and eigenvector : diagonal matrix containing the the eigenvalues of : eigenvalue : eigenvector

48 (ex) m n m n n m y(m,n) Column ; Stacking Operation Block Matrices Block matrices : elements are matrices

49 where block matrix Let xn and yn be the column vector, then

50 (ex) Definition Properties(Table2.7) Kronecker Products

51 Separable transformation Transformation on an NXM image row-ordered form Consider the transformation, if : matrix form : vector form

52

53 Definitions Random signal : a sequence of random variables Mean : Variance : Covariance : Cross covariance : Autocorrelation : Cross correlation : Random Signals

54 : Nx1 vector : NxN matrix : mean vector : covariance matrix Gaussian(or Normal) distribution Gaussian random processes Gaussian random process if the joint probability density of any finite sub-sequence is a Gaussian distribution : covariance matrix Representation for an NX1 vector

55 Stationary process Strict-sense stationary if the joint density of any partial sequence is the same as that of the shifted sequence Wide-sense stationary if Gaussian process : wide-sense = strict sense : covariance matrix is Toeplitz

56 Orthogonal : Independent : Uncorrelated : (ex) Covariance matrix of a first-order stationary Markov sequence u(n) : Toeplitz Markov processes p-th order Markov

57 Karhunen-Loeve(KL) transform KL transform of Property The elements of y(k) are orthogonal is called the KL transform matrix The rows of are the conjugate eigenvectors of : NxN unitary matrix

58 Definitions Discrete random field Each sample of a 2-D sequence is a random variable Mean : Covariance : White noise field Symmetry Discrete Random Field

59 Separable and isotropic image covariance functions Separable Separable stationary covariance function Nonseparable exponential function (Nonstationary case) (Stationary case) (isotropic or circularly symmetric) Estimation mean and autocorrelation

60 2-D case Average power SDF(spectral density function) Definition Fourier transform of autocorrelation function 1-D case

61 (ex) the SDF of stationary white noise field

62 Estimate the random variable x by a suitable function g(y), such that is min. but the integrand is non-negative ; it is sufficient to minimize for every y Estimation Theory Mean square estimates

63 minimum mean square estimate (MMSE) also unbiased estimator ◆ Theorem Let y △ and x be jointly Gaussian with zero mean. The MMSE estimation is, where a i is chosen, such that ∀ all k = 1, 2, …, N (Pf) The random variable are jointly Gaussian. But the first one is uncorrelated with all the rest, it is independent of them. Thus, the error is independent of the random vector y.

64 where: estimation error yields, n = 1, 2, …, N

65 The estimation error is minimized if, n = 1, 2, …, N orthogonality principle If x and {y(n)} are independent If zero mean Gaussian random variables : linear combination of {y(n)} is determined by solving linear equations

66 Orthogonality principle The minimum mean square estimation error vector is orthogonal to every random variable functionally related to the observations, i.e., for any Since is a function of substitute matrix notation,

67 Minimum MSE : If x,y(n) are nonzero mean r.v. If x,y(n) are non-Gaussian, the results still give the best linear mean square estimate.

68 Information Theory Information Entropy For a binary source, i.e.,

69 Let x be a discrete r.v. with S x ={1, 2, …, K} △ {x=k}{x=k} uncertainty of A k is low, if p k is close to one, with p k =P r [x=k] let event A k and it is high, if p k is small. uncertainty of event : if P r (x=k) = 1 entropy : unit : bit when the logarithm is base 2 Information Theory

70 Consider the event A k, describing the emission of symbol s k by the source with probability p k 1) if p k =1 and p i =0 ∀ all i≠k no surprise ⇒ no information when s k is emitted by the source 2) if r k is low more surprise ⇒ information when s k is emitted by the source ; amount of information gained after observing the event s k ; average information per source symbol Entropy as a measure of information

71 16 balls : 2 balls “3”, 2 balls “4” 1 ball “5”, “6”, “7”, “8” 4 balls “1”, 4 balls “2” Question : Find out the number of the ball through a series of yes/no questions. x=1 ? yes x=2 ? yes x=7 ? yes Ex) 1) x=8 x=1 x=2 x=7 no the average number of question asked :

72 x≤2 ? yes x≤4 ? yes x=7 ? yes 2) x=8 x=1 x=2 x=7 no x≤6 ? yes x=1 ? yes no x=3 x=4 x=3 ? yes no x=5 x=6 x=5 ? yes no ⇒ The problem of designing the series of questions to identify x is exactly the same as the problem of encoding the output of information source.

73 x= yes / yes ⇒ 1 1 x= yes / no ⇒ 1 0 x= no / yes / yes ⇒ x= no / yes / no ⇒ x= no / no / yes / yes ⇒ x= no / no / yes / no ⇒ x= no / no / no / yes ⇒ x= no / no / no / no ⇒ bit / symbol variable length code p k ⇒ Huffman code ⇒ short code to frequency source symbol average number of bits required to identify the outcome of x ⇒ entropy of x represent the max. long code to rare source symbol

74 Noiseless Coding Theorem (1948, Shannon) min(R) = H(x) +ε bit / symbol when R is the transmission rate and ε is a positive quantity that can be arbitrarily close to zero by sophisticated coding procedure utilizing an appropriate amount of encoding delay.

75 Rate distortion function Distortion x : Gaussian r.v of variance y : reproduced value Rate distortion function of x Rate distortion function for a Gaussian source For a fixed average distortion D : Gaussian r.v.’s : reproduced values where is determined by solving