1. CWIT 2005 1 Robust Entropy Rate for Uncertain Sources: Applications to Communication and Control Systems Charalambos D. Charalambous Dept. of Electrical and Computer Engineering University of Cyprus, Cyprus. Also with the School of Information Technology and Engineering University of Ottawa, Canada. and Alireza Farhadi School of Information Technology and Engineering University of Ottawa, Canada.

2. 2 CWIT 2005 Overview Robust Entropy  Solution and Relations to Renyi Entropy Examples  Uncertainty in Distribution  Uncertainty in Frequency Response Stabilization of Communication-Control Systems

3. CWIT 2005 3 Applications

4. 4 CWIT 2005 Applications: Robust Lossless Coding Theorem [1] Source words of block length produced by a discrete uncertain memory less source with Shannon entropy can be encoded into code words of block length from a coding alphabet of size with arbitrary small probability of error if Uncertain Source EncoderDecoderDestination Uniform Source Encoder

5. 5 CWIT 2005 Applications: Observability and Stabilizability of Networked Control Systems [2] Definition. The uncertain source is uniform asymptotically observable in probability if there exists an encoder and decoder such that where is the joint density of is the source density uncertainty set. and are fixed and.

6. 6 CWIT 2005 Applications: Observability and Stabilizability of Networked Control Systems [2] Definition. The uncertain controlled source is uniform asymptotically stabilizable in probability if there exists encoder, decoder and controller such that For and small enough, a necessary condition for uniform observability and stabilizability in probability is

7. CWIT 2005 7 Problem Formulation, Solution and Relations

8. 8 CWIT 2005 Problem Formulation  Let be the space of density functions defined on, be the true unknown source density, belonging to the uncertainty set. Let also be the fixed nominal source density.  Definition. The robust entropy of observed process having density function is defined by where is the Shannon entropy and

9. 9 CWIT 2005 Problem Formulation Definition. In Previous definition, if represent a sequence of R.V.’s with length of source symbols produced by the uncertain source with joint density, the robust entropy rate is defined by provided the limit exits.

10. 10 CWIT 2005 Solution to the Robust Entropy  Let the uncertain set be defined by where is the relative entropy and is fixed.  Lemma. Given a fixed nominal density and the uncertainty defined above, the solution to the robust entropy is given by where the minimizing in above is the unique solution of.

11. 11 CWIT 2005 Solution to the Robust Entropy Corollary [1]. Suppose, is uniform Probability Mass Function (PMF), that is When and consequently Correspond to PMF’s, that is the previous solution is reduced to

12. 12 CWIT 2005 Solution to the Robust Entropy Rate Corollary. Let be a sequence with length of source symbols with uncertain joint density function and let we exchange with. Then, the robust entropy rate is given by where the minimizing in above is the unique solution of.

13. 13 CWIT 2005 Robust Entropy Rate of Uncertain Sources with Nominal Gaussian Source Density Example. From Previous result follows that if the nominal source density is -dimensional Gaussian density function with mean and covariance where is the unique solution of the following nonlinear equation

14. 14 CWIT 2005 Relation among Renyi, Tsallis and Shannon Entropies Definition. For and, the Renyi entropy [3] is defined by Moreover, the Tsallis entropy [4] is defined by We have the following relation among Shannon, Renyi and Tsallis entropies.

15. 15 CWIT 2005 Relation between Robust Entropy and Renyi Entropy The robust entropy found previously is related to the Renyi entropy as follows. Let ; then Moreover,

16. CWIT 2005 16 Examples

17. 17 CWIT 2005 Examples: Partially Observed Gauss Markov Nominal Source Assumptions. Let and are Probability Density Functions (PDF’s) corresponding to a sequence with length of symbols produced by uncertain and nominal sources respectively. Let also the uncertain source and nominal source are related by

18. 18 CWIT 2005 Example: Partially Observed Gauss Markov Nominal Source Nominal Source. The nominal density is induced by a partially observed Gauss Markov nominal source defined by Assumptions. unobserved process, observed process, is iid, is iid, are mutually independent is detectable is stabilizable and

19. 19 CWIT 2005 Partially Observed Gauss Markov Nominal Source: Lemmas Lemma. Let be a stationary Gaussian process with power spectral density. Let and assume exists. Then

20. 20 CWIT 2005 Partially Observed Gauss Markov Nominal Source: Lemmas Lemma. For the partially observed Gauss Markov nominal source whereis the unique positive semi-definite solution of the following Algebraic Ricatti equation

21. 21 CWIT 2005 Partially Observed Gauss Markov Nominal Source: Robust Entropy Rate Proposition. The robust entropy rate of an uncertain source with corresponding partially observed Gauss Markov nominal source is is the Shannon entropy rate of the nominal source,is given in previously mentioned Lemma and is the unique solution of

22. 22 CWIT 2005 Partially Observed Gauss Markov Nominal Source: Remarks Remark. For the scalar case with, after solving, we obtain Remark. The case corresponds to. Letting, we have

23. 23 CWIT 2005 Example: Partially Observed Controlled Gauss Markov Nominal Source The nominal source is defined via a partially observed controlled Gauss Markov system given by Assumptions. The uncertain source and nominal source are related by is stabilizing matrix, is unobserved process, is observed process, is iid are mutually independent and

24. 24 CWIT 2005 Partially Observed Controlled Gauss Markov Nominal Source: Robust Entropy Rate Proposition. Using Body integral formula [5] (e.g., relation between the sensivity transfer function and the unstable eigenvalues of system), the robust entropy rate of family of uncertain sources with corresponding partially observed controlled Gauss Markov nominal source is are eigenvalues of the system matrix and is the unique solution of

25. 25 CWIT 2005 Example: Uncertain Sources in Frequency Domain Defining Spaces. Let and be the space of scalar bounded, analytic function of. This space is equipped with the norm defined by Assumptions. Suppose the uncertain source is obtained by passing a stationary Gaussian random process, whit known power spectral density, through an uncertain linear filter where belongs to the following additive uncertainty model

26. 26 CWIT 2005 Uncertain Sources in Frequency Domain: Robust Entropy Rate Results. The observed process is Gaussian random process, and the Shannon entropy rate of observed process is Subsequently, the robust entropy rate is defined via and the solution is given by [6]

27. CWIT 2005 27 Applications in Stabilizability of Networked Control Systems

28. 28 CWIT 2005 Observability and Stabilizability of Networked Control Systems Definition. The uncertain source is uniform asymptotically observable in probability if there exists an encode and decoder such that Moreover, a controlled uncertain source is uniform asymptotically stabilizable in probability if there exists an encoder, decoder and controller such that

29. 29 CWIT 2005 Necessary Condition for Observability and Stabilizability of Networked Control Systems [2] Proposition. Necessary condition for uniform asymptotically observability and stabilizability in probability is is the covariance matrix of Gaussian distribution that satisfies

30. 30 CWIT 2005 References [1] C. D. Charalambous and Farzad Rezaei, Robust Coding for Uncertain Sources, Submitted to IEEE Trans. On Information Theory, 2004. [2] C. D. Charalambous and Alireza Farhadi, Mathematical Theory for Robust Control and Communication Subject to Uncertainty, Preprint, 2005. [3] A. Renyi, “On Measures of Entropy and Information”, in Proc. 4 th Berkeley Symp. Mathematical Statistics and Probability, vol. I, Berkeley, CA, pp. 547-561, 1961.

31. 31 CWIT 2005 References [4] C. Tsallis, Possible Generalization of Boltzmann-Gibbs Statistics, Journal of Statistics Physics, vol. 52, pp. 479, 1998. [5] B. F. Wu, and E. A. Jonckheere, A Simplified Approach to Bode’s Theorem for Continuous-Time and Discrete-Time Systems, IEEE Trans on AC, vol. 37, No. 11, pp. 1797- 1802, Nov 1992. [6] C. D. Charalambous and Alireza Farhadi, Robust Entropy and Robust Entropy Rate for Uncertain Sources: Applications in Networked Control Systems, Preprint, 2005.