1. Graduate School of Information Sciences, Tohoku University
Physical Fluctuomatics Applied Stochastic Process th Probabilistic information processing by means of graphical model
Kazuyuki Tanaka
Graduate School of Information Sciences, Tohoku University
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

2. Physical Fuctuomatics (Tohoku University)
Textbooks
Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 7.
K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), Journal of Physics A: Mathematical and General, vol.35, no.37, pp.R81-R150, 2002, Section 4.
Physical Fuctuomatics (Tohoku University)

3. Contents Introduction Probabilistic Image Processing
Gaussian Graphical Model
Statistical Performance Analysis
Concluding Remarks
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

4. Contents Introduction Probabilistic Image Processing
Gaussian Graphical Model
Statistical Performance Analysis
Concluding Remarks
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

5. Markov Random Fields for Image Processing
are One of Probabilistic Methods for Image processing.
S. Geman and D. Geman (1986): IEEE Transactions on PAMI
Image Processing for Markov Random Fields (MRF) (Simulated Annealing, Line Fields)
J. Zhang (1992): IEEE Transactions on Signal Processing
Image Processing in EM algorithm for Markov Random Fields (MRF) (Mean Field Methods)
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

6. Markov Random Fields for Image Processing
In Markov Random Fields, we have to consider not only the states with high probabilities but also ones with low probabilities.
Hyperparameter Estimation
Statistical Quantities
Estimation of Image
In Markov Random Fields, we have to estimate not only the image but also hyperparameters in the probabilistic model.
We have to perform the calculations of statistical quantities repeatedly.
We can calculate statistical quantities by adopting the Gaussian graphical model as a prior probabilistic model and by using Gaussian integral formulas.
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

7. Purpose of My Talk
Review of formulation of probabilistic model for image processing by means of conventional statistical schemes.
Review of probabilistic image processing by using Gaussian graphical model (Gaussian Markov Random Fields) as the most basic example.
K. Tanaka: Statistical-Mechanical Approach to Image Processing (Topical Review), J. Phys. A: Math. Gen., vol.35, pp.R81-R150, 2002.
Section 2 and Section 4 are summarized in the present talk.
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

8. Contents Introduction Probabilistic Image Processing
Gaussian Graphical Model
Statistical Performance Analysis
Concluding Remarks
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

9. Image Representation in Computer Vision
Digital image is defined on the set of points arranged on a square lattice.
The elements of such a digital array are called pixels.
We have to treat more than 100,000 pixels even in the digital cameras and the mobile phones.
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

10. Image Representation in Computer Vision
At each point, the intensity of light is represented as an integer number or a real number in the digital image data.
A monochrome digital image is then expressed as a two-dimensional light intensity function and the value is proportional to the brightness of the image at the pixel.
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

11. Noise Reduction by Conventional Filters
The function of a linear filter is to take the sum of the product of the mask coefficients and the intensities of the pixels.
Smoothing Filters
192
202
190
192
202
190
202
219
120
202
173
120
100
218
110
100
218
110
It is expected that probabilistic algorithms for image processing can be constructed from such aspects in the conventional signal processing.
Markov Random Fields
Probabilistic Image Processing
Algorithm
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

12. Bayes Formula and Bayesian Network
Prior Probability
Data-Generating Process
Bayes Rule
Posterior Probability A
Event B is given as the observed data.
Event A corresponds to the original information to estimate.
Thus the Bayes formula can be applied to the estimation of the original information from the given data. B
Bayesian
Network
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

13. Image Restoration by Probabilistic Model
Assumption 1: The degraded image is randomly generated from the original image by according to the degradation process.
Assumption 2: The original image is randomly generated by according to the prior probability.
Noise
Transmission
Original Image
Degraded Image
Bayes Formula
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

14. Image Restoration by Probabilistic Model
The original images and degraded images are represented by f = (f1,f2,…,f|V|) and g = (g1,g2,…,g|V|), respectively.
Original
Image
Degraded
Image
Position Vector of Pixel i i i
fi: Light Intensity of Pixel i
in Original Image
gi: Light Intensity of Pixel i
in Degraded Image
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

15. Probabilistic Modeling of Image Restoration
Assumption 1: A given degraded image is obtained from the original image by changing the state of each pixel to another state by the same probability, independently of the other pixels. fi gi fi gi or
Random Fields
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

16. Probabilistic Modeling of Image Restoration
Assumption 2: The original image is generated according to a prior probability. Prior Probability consists of a product of functions defined on the neighbouring pixels. i j
Random Fields
Product over All the Nearest Neighbour Pairs of Pixels
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

17. Prior Probability for Binary Image
It is important how we should assume the function F(fi,fj) in the prior probability. i j
We assume that every nearest-neighbour pair of pixels take the same state of each other in the prior probability.
Probability of Neigbouring Pixel i =
> p j
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

18. Prior Probability for Binary Image=
> p j
Probability of Nearest Neigbour Pair of Pixels ？
Which state should the center pixel be taken when the states of neighbouring pixels are fixed to the white states?
>
Prior probability prefers to the configuration with the least number of red lines.
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

19. Prior Probability for Binary Image=
>
？-？
Which state should the center pixel be taken when the states of neighbouring pixels are fixed as this figure?
> =
>
Prior probability prefers to the configuration with the least number of red lines.
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

20. What happens for the case of large umber of pixels?
Patterns with
both ordered states
and disordered states
are often generated near the critical point.
Covariance between the nearest neghbour pairs of pixels p
lnp
small p
large p
Sampling by Marko chain Monte Carlo
Critical Point
(Large fluctuation)
Disordered State
Ordered State
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

21. Pattern near Critical Point of Prior Probability
We regard that patterns generated near the critical point are similar to the local patterns in real world images.
Covariance between the nearest neghbour pairs of pixels
ln p
small p
large p
similar
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

22. Bayesian Image Analysis
Degraded Image
Prior Probability
Original
Image
Degradation Process
Posterior Probability
V：Set of All the pixels
E：Set of all the nearest neighbour pairs of pixels
Image processing is reduced to calculations of averages, variances and co-variances in the posterior probability.
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

23. Estimation of Original Image
We have some choices to estimate the restored image from posterior probability.
In each choice, the computational time is generally exponential order of the number of pixels.
(1)
Maximum A Posteriori (MAP)
estimation
(2)
Maximum posterior
marginal (MPM) estimation
In the image restoration, we usually have to estimate the hyperparameters alpha and p.
In statistics, the maximum likelihood estimation is often employed.
In the standpoint of maximum likelihood estimation, the hyperparameters are determined so as to maximize the marginal likelihood defined by marginalize the joint probability for the original image and degraded image with respect to the original image.
The marginal likelihood is expressed in terms of the partition functions of the a priori probabilistic model and the a posteriori probabilistic model.
We can calculate these partition functions approximately by using the Bethe approximation.
(3)
Thresholded Posterior
Mean (TPM) estimation
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

24. Contents Introduction Probabilistic Image Processing
Gaussian Graphical Model
Statistical Performance Analysis
Concluding Remarks
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

25. Bayesian Image Analysis by Gaussian Graphical Model
Prior
Probability
V:Set of all
the pixels
Patterns are generated by MCMC.
E:Set of all the nearest-neighbour pairs of pixels
Markov Chain Monte Carlo Method
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

26. Bayesian Image Analysis by Gaussian Graphical Model
Degradation Process is assumed to be the additive white Gaussian noise.
V: Set of all
the pixels
Gaussian Noise n
Original Image f
Degraded Image g
Degraded image is obtained by adding a white Gaussian noise to the original image.
Histogram of Gaussian Random Numbers
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

27. Bayesian Image Analysis
Degraded Image
Prior Probability
Original
Image
Degradation Process
Posterior Probability
V：Set of All the pixels
E：Set of all the nearest neighbour pairs of pixels
Image processing is reduced to calculations of averages, variances and co-variances in the posterior probability.
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

28. Bayesian Image Analysis
A Posteriori Probability
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

29. Statistical Estimation of Hyperparameters
Hyperparameters a, s are determined so as to maximize the marginal likelihood Pr{G=g|a,s} with respect to a, s.
Original Image
Degraded Image
In the image restoration, we usually have to estimate the hyperparameters alpha and p.
In statistics, the maximum likelihood estimation is often employed.
In the standpoint of maximum likelihood estimation, the hyperparameters are determined so as to maximize the marginal likelihood defined by marginalize the joint probability for the original image and degraded image with respect to the original image.
The marginal likelihood is expressed in terms of the partition functions of the a priori probabilistic model and the a posteriori probabilistic model.
We can calculate these partition functions approximately by using the Bethe approximation.
Marginalized with respect to F
Marginal Likelihood
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

30. Bayesian Image Analysis
A Posteriori Probability
|V|x|V| matrix
Gaussian Graphical Model
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

31. Average of Posterior Probability
Gaussian Integral formula
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

32. Bayesian Image Analysis by Gaussian Graphical Model
Posterior Probability
V:Set of all
the pixels
Average of the posterior probability can be calculated by using the multi-dimensional Gauss integral Formula
E:Set of all the nearest-neghbour pairs of pixels
|V|x|V| matrix
Multi-Dimensional Gaussian Integral Formula
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

33. Statistical Estimation of Hyperparameters
Original Image
Degraded Image
In the image restoration, we usually have to estimate the hyperparameters alpha and p.
In statistics, the maximum likelihood estimation is often employed.
In the standpoint of maximum likelihood estimation, the hyperparameters are determined so as to maximize the marginal likelihood defined by marginalize the joint probability for the original image and degraded image with respect to the original image.
The marginal likelihood is expressed in terms of the partition functions of the a priori probabilistic model and the a posteriori probabilistic model.
We can calculate these partition functions approximately by using the Bethe approximation.
Marginalized with respect to F
Marginal Likelihood
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

34. Calculations of Partition Function
Gaussian Integral formula
(A is a real symmetric and positive definite matrix.)
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

35. Exact expression of Marginal Likelihood in Gaussian Graphical Model
Multi-dimensional Gauss integral formula
We can construct an exact EM algorithm.
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

36. Bayesian Image Analysis by Gaussian Graphical Model
Iteration Procedure in Gaussian Graphical Model
In the image restoration, we usually have to estimate the hyperparameters alpha and p.
In statistics, the maximum likelihood estimation is often employed.
In the standpoint of maximum likelihood estimation, the hyperparameters are determined so as to maximize the marginal likelihood defined by marginalize the joint probability for the original image and degraded image with respect to the original image.
The marginal likelihood is expressed in terms of the partition functions of the a priori probabilistic model and the a posteriori probabilistic model.
We can calculate these partition functions approximately by using the Bethe approximation.
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

37. Image Restoration by Markov Random Field Model and Conventional Filters
MSE
Statistical Method
315
Lowpass Filter
(3x3)
388
(5x5)
413
Median Filter
486
445
Original Image
Degraded Image
V:Set of all
the pixels
Restored
Image
Finally, we show only the results for the gray-level image restoration.
For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters.
MRF
(3x3) Lowpass
(5x5) Median
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

38. Contents Introduction Probabilistic Image Processing
Gaussian Graphical Model
Statistical Performance Analysis
Concluding Remarks
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

39. Sample Average of Mean Square Error Additive White Gaussian Noise
Performance Analysis
Sample Average of Mean Square Error
Additive White Gaussian Noise
Posterior Probability
Signal
Observed
Data
Estimated Results
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

40. Statistical Performance Analysis
Degraded Image
Original Image
Additive White Gaussian Noise
Restored Image
Posterior Probability
Additive White Gaussian Noise
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

41. Statistical Performance Analysis
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

42. Statistical Performance Estimation for Gaussian Markov Random Fields
= 0
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

43. Statistical Performance Estimation for Gaussian Markov Random Fields
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

44. Contents Introduction Probabilistic Image Processing
Gaussian Graphical Model
Statistical Performance Analysis
Concluding Remarks
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

45. Summary
Formulation of probabilistic model for image processing by means of conventional statistical schemes has been summarized.
Probabilistic image processing by using Gaussian graphical model has been shown as the most basic example.
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

46. References
K. Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) . K. Tanaka: Statistical-Mechanical Approach to Image Processing (Topical Review), J. Phys. A, 35 (2002). A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, Proceedings of IEEE, 90 (2002).
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

47. Problem 5-1: Derive the expression of the posterior probability Pr{F=f|G=g,a,s} by using Bayes formulas Pr{F=f|G=g,a,s} =Pr{G=g|F=f,s}Pr{F=f,a}/Pr{G=g|a,s}. Here Pr{G=g|F=f,s} and Pr{F=f,a} are assumed to be as follows:
[Answer]
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

48. Problem 5-2: Show the following equality.
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

49. Problem 5-3: Show the following equality.
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

50. Problem 5-4: Show the following equalities by using the multi-dimensional Gaussian integral formulas.
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

51. Problem 5-5: Derive the extremum conditions for the following marginal likelihood Pr{G=g|a,s} with respect to the hyperparameters a and s.
[Answer]
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

52. Problem 5-6: Derive the extremum conditions for the following marginal likelihood Pr{G=g|a,s} with respect to the hyperparameters a and s.
[Answer]
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

53. Problem 5-7: Make a program that generate a degraded image by the additive white Gaussian noise. Generate some degraded images from a given standard images by setting s=10,20,30,40 numerically. Calculate the mean square error (MSE) between the original image and the degraded image.
Gaussian Noise
Original Image
Degraded Image
K.Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd.,
Sample Program:
Histogram of Gaussian Random Numbers Fi -Gi~N(0,402)
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)

54. Repeat the following procedure until convergence
Problem 5-8: Make a program of the following procedure in probabilistic image processing by using the Gaussian graphical model and the additive white Gaussian noise.
Algorithm:
Repeat the following procedure until convergence
K.Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd.,
Sample Program:
Physical Fluctuomatics / Applied Stochansic Process (Tohoku University)