1. Physical Fuctuomatics (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 1st Review of probabilistic information processing Kazuyuki Tanaka Graduate School of Information Sciences kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/ Webpage: http://www.smapip.is.tohoku.ac.jp/~kazu/PhysicalFluctuomatics/2010/

2. Physical Fuctuomatics (Tohoku University) 2 Textbooks Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese). Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., 2009 (in Japanese).

3. Physical Fuctuomatics (Tohoku University) 3 References of the present lecture K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), Journal of Physics A: Mathematical and General, vol.35, no.37, pp.R81-R150, 2002. Y. Kabashima and D. Saad: Statistical mechanics of low- density parity-check codes (Topical Review), J. Phys. A, vol.37, no.6, pp.R1-R43, 2004. H. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001. M. Opper and D. Saad D (eds): Advanced Mean Field Methods --- Theory and Practice, MIT Press, 2001. C. M. Bishop: Pattern Recognition and Machine Learning, Springer, 2006. M. J. Wainwright and M. I. Jordan: Graphical Models, Exponential Families, and Variational Inference, now Publishing Inc, 2008. M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2009.

4. Physical Fuctuomatics (Tohoku University) 4 Benefit of Information & Communications Technology Ubiquitous Computing Ubiquitous Internet Benefit of Information & Communications Technology Demand for Intelligence It cannot be satisfied only with it being only cheap and being quick.

5. Physical Fuctuomatics (Tohoku University) 5 Field of Information Processing Information processing according to theories Inference from propositions Realization by progress of computational processing capacity Information processing in real world Diversity of reason in phenomenon Compete data is not necessarily obtained. It is difficult to extract and select some important information from a lot of data. Uncertainty caused by the gap of knowing simply and understanding actually. We hope to deal successfully with such uncertainty. Information processing for numerical calculations Definite Procedure has been given for each calculation.

6. Physical Fuctuomatics (Tohoku University) 6 Computer for next generations Required Capacity Capability to sympathize with a user （ Knowledge) Capability to put failure and experience to account in the next chance （ Learning ） How should we deal successfully with the uncertainty caused by the gap of knowing simply and understanding actually? Formulation of knowledge and uncertainty Realization of information processing data with uncertainty

7. Physical Fuctuomatics (Tohoku University) 7 Computational model for information processing in data with uncertainty Probabilistic Inference Probabilistic model with graphical structure （ Bayesian network ） Medical diagnosis Failure diagnosis Risk Management Probabilistic information processing can give us unexpected capacity in a system constructed from many cooperating elements with randomness. Inference system for data with uncertainty modeling Node is random variable. Arrow is conditional probability. Mathematical expression of uncertainty =>Probability and Statistics Graph with cycles Important aspect

8. Physical Fuctuomatics (Tohoku University) 8 Computational Model for Probabilistic Information Processing Probabilistic Information Processing Probabilistic Model Bayes Formula Algorithm Monte Carlo Method Markov Chain Monte Carlo Method Randomized Algorithm Genetic Algorithm Approximate Method Belief Propagation Mean Field Method Randomness and Approximation

9. Physical Fuctuomatics (Tohoku University) 9 Probabilistic Image Processing Noise Reduction by Probabilistic Image Processing K. Tanaka: J. Phys. A, vol.35, 2002. A. S. Willsky: Proceedings of IEEE, vol.90, 2002. 192 202 190 202 219 120 100 218 110 192 202 190 202 173 120 100 218 110 Modeling of Probabilistic Image Processing based on Conventional Filters Markov Random Filed Model Probabilistic Image Processing The elements of such a digital array are called pixels. At each point, the intensity of light is represented as an integer number or a real number in the digital image data. Algorithm Conventional Filter

10. Physical Fuctuomatics (Tohoku University) 10 Probabilistic Image Processing Degraded Image (Gaussian Noise ） Probabilistic Image Processing Lowpass FilterWiener FilterMedian Filter MSE:520 MSE: 2137 MSE:860MSE:767MSE:1040 K. Tanaka: J. Phys. A, vol.35, 2002. A. S. Willsky: Proceedings of IEEE, vol.90, 2002.

11. Physical Fuctuomatics (Tohoku University) 11 Error Correcting Code Y. Kabashima and D. Saad: J. Phys. A, vol.37, 2004. High Performance Decoding Algorithm 010 000001111100000001001011100001 0 1 0 code 010 error decode Parity Check Code Turbo Code, Low Density Parity Check (LDPC) Code majority rule Error Correcting Codes

12. 14 January, 2010 Hokkaido University GCOE Tutorial (Sapporo ） 12 Error Correcting Codes and Belief Propagation

13. 14 January, 2010 Hokkaido University GCOE Tutorial (Sapporo ） 13 Error Correcting Codes and Belief Propagation

14. 14 January, 2010 Hokkaido University GCOE Tutorial (Sapporo ） 14 Error Correcting Codes and Belief Propagation Code Word

15. 14 January, 2010 Hokkaido University GCOE Tutorial (Sapporo ） 15 Error Correcting Codes and Belief Propagation 11 0 1 0 0 Received Word Code Word Binary Symmetric Channel

16. 14 January, 2010 Hokkaido University GCOE Tutorial (Sapporo ） 16 Error Correcting Codes and Belief Propagation

17. 14 January, 2010 Hokkaido University GCOE Tutorial (Sapporo ） 17 Error Correcting Codes and Belief Propagation

18. 14 January, 2010 Hokkaido University GCOE Tutorial (Sapporo ） 18 Error Correcting Codes and Belief Propagation

19. 14 January, 2010 Hokkaido University GCOE Tutorial (Sapporo ） 19 Error Correcting Codes and Belief Propagation Fundamental Concept for Turbo Codes and LDPC Codes

20. Physical Fuctuomatics (Tohoku University) 20 CDMA Multiuser Detectors in Mobile Phone Communication Relationship between mobile phone communication and spin glass theory T. Tanaka, IEEE Trans. on Information Theory, vol.48, 2002 Signals of User A Spreading Code Sequence Wireless Communication Received Data Decode Spreading Code Sequence Probabilistic model for decoding can be expressed in terms of a physical model for spin glass phenomena Noise Coded Signals of Other Users Coded Signals of User A

21. Physical Fuctuomatics (Tohoku University) 21 Artificial Intelligence Bayesian Network Probabilistic inference system Practical algorithms by means of belief propagation J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988).

22. Physical Fuctuomatics (Tohoku University) 22 Main Interests Information Processing: Data Physics: Material, Natural Phenomena System of a lot of elements with mutual relation Common Concept between Information Sciences and Physics Material Molecule Materials are constructed from a lot of molecules. Molecules have interactions of each other. ０,1 101101 110001 01001110111010 10001111100001 10000101000000 11101010111010 1010 Bit Data Data is constructed from many bits A sequence is formed by deciding the arrangement of bits. A lot of elements have mutual relation of each other Some physical concepts in Physical models are useful for the design of computational models in probabilistic information processing.

23. Physical Fuctuomatics (Tohoku University) 23 Horizon of Computation in Probabilistic Information Processing Compensation of expressing uncertainty using probability and statistics It must be calculated by taking account of both events with high probability and events with low probability. Computational Complexity It is expected to break throw the computational complexity by introducing approximation algorithms.

24. Physical Fuctuomatics (Tohoku University) 24 What is an important point in computational complexity? How should we treat the calculation of the summation over 2 N configuration? N fold loops If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40.

25. Physical Fuctuomatics (Tohoku University) 25 Why is a physical viewpoint effective in probabilistic information processing? Matrials are constructed from a lot of molecules. (10 23 molecules exist in 1 mol.) Molecules have intermolecular forces of each other Theoretical physicists always have to treat such multiple summation. Development of Approximate Methods Probabilistic information processing is also usually reduced to multiple summations or integrations. Application of physical approximate methods to probabilistic information processing

26. Physical Fuctuomatics (Tohoku University) 26 Academic Circulation Academic Circulation between Physics and Information Sciences Physics Information Sciences Understanding and prediction of properties of materials and natural phenomena Extraction and processing of information in data Common Concept Statistical Mechanical Informatics Probabilistic Information Processing Statistical Sciences

27. Physical Fuctuomatics (Tohoku University) 27 Summary of the present lecture Probabilistic information processing Examples of probabilistic information processing Common concept in physics and information sciences Application of physical modeling and approximations Future Lectures Fundamental theory of probability and statistics Linear model Graphical model.