1. Outline Statistical Modeling and Conceptualization of Visual Patterns
S. C. Zhu, “Statistical modeling and conceptualization of visual patterns,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 6, 1-22, 2003

2. A Common Framework of Visual Knowledge Representation
Visual patterns in natural images
Natural images consist of an overwhelming number of visual patterns
Generated by very diverse stochastic processes
Comments
Any single image normally consists of a few recognizable/segmentable visual patterns
Scientifically, given that visual patterns are generated by stochastic processes, shall we model the underlying stochastic processes or model visual patterns presented in the observations from the stochastic processes?
November 18, 2018
Computer Vision

3. A Common Framework of Visual Knowledge Representation – cont.
November 18, 2018
Computer Vision

4. A Common Framework of Visual Knowledge Representation – cont.
The image analysis as an image parsing problem
Parse generic images into their constituent patterns (according to the underlying stochastic processes)
Perceptual grouping when applied to points, lines, and curves processes
Image segmentation when applied to region processes
Object recognition when applied to high level objects
November 18, 2018
Computer Vision

5. A Common Framework of Visual Knowledge Representation – cont.
November 18, 2018
Computer Vision

6. A Common Framework of Visual Knowledge Representation – cont.
Required components for parsing
Mathematical definitions and models of various visual patterns
Definitions and models are intrinsically recursive
Grammars (or called rules)
Which specifies the relationships among various patterns
Grammars should be stochastic in nature
A parsing algorithm
November 18, 2018
Computer Vision

7. Syntactical Pattern Recognition
November 18, 2018
Computer Vision

8. A Common Framework of Visual Knowledge Representation – cont.
Conceptualization of visual patterns
The concept of a pattern is an abstraction of some properties decided by certain “visual purposes”
They are feature statistics computed from
Raw signals
Some hidden descriptions inferred from raw signals
Mathematically, each pattern is equivalent to a set of observable signals governed by a statistical model
November 18, 2018
Computer Vision

9. A Common Framework of Visual Knowledge Representation – cont.
Statistical modeling of visual patterns
Statistical models are intrinsic representations of visual knowledge and image regularities
Due to noise and distortion in imaging process?
Due to noise and distortion in the underlying generative process?
Due to transformations in the underlying stochastic process?
Pattern theory
November 18, 2018
Computer Vision

10. A Common Framework of Visual Knowledge Representation – cont.
Statistical modeling of visual patterns - continued
Mathematical space for patterns and spaces
Depends on the forms
Parametric
Non-parametric
Attributed graphs
Different models
Descriptive models
Bottom-up, feature-based models
Generative models
Hidden variables for generating images in a top-down manner
November 18, 2018
Computer Vision

11. A Common Framework of Visual Knowledge Representation – cont.
Learning a visual vocabulary
Hierarchy of visual descriptions for general visual patterns
Vocabulary of visual description
Learning from an ensemble of natural images
Vocabulary is far from enough
Rich structures in physics
Large vocabulary in speech and language
November 18, 2018
Computer Vision

12. A Common Framework of Visual Knowledge Representation – cont.
Computational tractability
Computational heuristics for effective inference of visual patterns
Discriminative models
A framework
Discriminative probabilities are used as proposal probabilities that drive the Markov chain search for fast convergence and mixing
Generative models are top-down probabilities and the hidden variables to be inferred from posterior probabilities
November 18, 2018
Computer Vision

13. A Common Framework of Visual Knowledge Representation – cont.
Discussion
Images are generated by rendering 3D objects under some external conditions
All the images from one object form a low dimensional manifold in a high dimensional image space
Rendering can be modeled fairly accurately
Describing a 3D object requires a huge amount of data
Under this setting
A visual pattern simply corresponds to the manifold
Descriptive model attempts to characterize the manifold
Generative model attempts to learn the 3D objects and the rendering
November 18, 2018
Computer Vision

14. 3D Model-Based Recognition
November 18, 2018
Computer Vision

15. Literature Survey
To develop a generic vision system, regularities in images must be modeled
The study of natural image statistics
Ecologic influence on visual perception
Natural images have high-order (i.e., non-Gaussian) structures
The histograms of Gabor-type filter responses on natural images have high kurtosis
Histograms of gradient filters are consistent over a range of scales
November 18, 2018
Computer Vision

16. Natural Image Statistics Example
November 18, 2018
Computer Vision

17. Analytical Probability Models for Spectral Representation
Transported generator model (Grenander and Srivastava, 2000)
where
gi’s are selected randomly from some generator space G
the weigths ai’s are i.i.d. standard normal
the scales ri’s are i.i.d. uniform on the interval [0,L]
the locations zi’s as samples from a 2D homogenous Poisson process, with a uniform intensity l, and
the parameters are assumed to be independent of each other
November 18, 2018
Computer Vision

18. Analytical Probability Models - continued
Define
Model u by a scaled -density
November 18, 2018
Computer Vision

19. Analytical Probability Models - continued
November 18, 2018
Computer Vision

20. Analytical Probability Models - continued
November 18, 2018
Computer Vision

21. Analytical Probability Models - continued
November 18, 2018
Computer Vision

22. Analysis of Natural Image Components
Harmonic analysis
Decomposing various classes of functions by different bases
Including Fourier transform, wavelet transforms, edgelets, curvelets, and so on
November 18, 2018
Computer Vision

23. Sparse Coding
From S. C. Zhu
November 18, 2018
Computer Vision

24. Grouping of Natural Image Elements
Gestalt laws
Gestalt grouping laws
Should be interpreted as heuristics rather than deterministic laws
Nonaccidental property
November 18, 2018
Computer Vision

25. Illusion
November 18, 2018
Computer Vision

26. Illusion – cont.
November 18, 2018
Computer Vision

27. Ambiguous Figure
November 18, 2018
Computer Vision

28. Statistical Modeling of Natural Image Patterns
Synthesis-by-analysis
November 18, 2018
Computer Vision

29. Analog from Speech Recognition
November 18, 2018
Computer Vision

30. Modeling of Natural Image Patterns
Shape-from-X problems are fundamentally ill-posed
Markov random field models
Deformable templates for objects
Inhomogeneous MRF models on graphs
November 18, 2018
Computer Vision

31. Four Categories of Statistical Models
Descriptive models
Constructed based on statistical descriptions of the image ensembles
Homogeneous models
Statistics are assumed to be the same for all elements in the graph
Inhomogeneous models
The elements of the underlying graph are labeled and different features and statistics are used at different sites
November 18, 2018
Computer Vision

32. Variants of Descriptive Models
Casual Markov models
By imposing a partial ordering among the vertices of the graph, the joint probability can be factorized as a product of conditional probabilities
Belief propagation networks
Pseudo-descriptive models
November 18, 2018
Computer Vision

33. Generative Models
Use of hidden variables that can “explain away” the strong dependency in observed images
This requires a vocabulary
Grammars to generate images from hidden variables
Note that generative models can not be separated from descriptive models
The description of hidden variables requires descriptive models
November 18, 2018
Computer Vision

34. Discriminative Models
Approximation of posterior probabilities of hidden variables based on local features
Can be seen as importance proposal probabilities
November 18, 2018
Computer Vision

35. An Example
November 18, 2018
Computer Vision

36. Problem formation Input: a set of images Output: a probability model
Here, f(I) represents the ensemble of images in a given domain, we shall discuss the relationship between ensemble and probability later.
November 18, 2018
Computer Vision

37. Problem formation The model p approaches the true density
The Kullback-Leibler Divergence
November 18, 2018
Computer Vision

38. Maximum Likelihood Estimate
November 18, 2018
Computer Vision

39. Model Pursuit 1. What is W -- the family of models ?
2. How do we augment the space W?
November 18, 2018
Computer Vision

40. Two Choices of Models The exponential family – descriptive models
--- Characterize images by features and statistics
2. The mixture family -- generative models
--- Characterize images by hidden variables
Features are deterministic mathematical transforms of an image.
Hidden variables are stochastic and are inferred from an image.
November 18, 2018
Computer Vision

41. I: Descriptive Models
Step 1: extracting image features/statistics as transforms
For example:
histograms of Gabor filter responses.
Other features/statistics: Gabors, geometry, Gestalt laws, faces.
November 18, 2018
Computer Vision

42. I.I: Descriptive Models
Step 2: using features/statistics to constrain the model
Two cases:
On infinite lattice Z2 --- an equivalence class.
On any finite lattice --- a conditional probability model.
image space on Z2
image space on lattice L
November 18, 2018
Computer Vision

43. I.I Descriptive Model on Finite Lattice
Modeling by maximum entropy:
Subject to:
Remark: p and f have the same projected marginal statistics.
November 18, 2018
Computer Vision

44. Minimax Entropy Learning
For a Gibbs (max. entropy) model p, this leads to the
minimax entropy principle (Zhu,Wu, Mumford 96,97)
November 18, 2018
Computer Vision

45. FRAME Model FRAME model Filtering, random field, and maximum entropy
A well-defined mathematical model for textures by combining filtering and random field models
November 18, 2018
Computer Vision

46. I.I Descriptive Model on Finite Lattice
The FRAME model (Zhu, Wu, Mumford, 1996)
This includes all Markov random field models.
Remark: all known exponential models are from maxent., and maxent was proposed in Physics (Jaynes, 1957). The nice thing is that it provides a parametric model integrating features.
November 18, 2018
Computer Vision

47. I.I Descriptive Model on Finite Lattice
Two learning phases:
1. Choose information bearing features
-- augmenting the probability family.
2. Compute the parameter L by MLE
-- learning within a family.
November 18, 2018
Computer Vision

48. Maximum Entropy Maximum entropy
Is an important principle in statistics for constructing a probability distribution on a set of random variables
Suppose the available information is the expectations of some known functions n(x), that is
Let W be the set of all probability distributions p(x) which satisfy the constraints
November 18, 2018
Computer Vision

49. Maximum Entropy – cont. Maximum Entropy – continued
According to the maximum entropy principle, a good choice of the probability distribution is the one that has the maximum entropy
subject to
November 18, 2018
Computer Vision

50. Maximum Entropy – cont. Maximum Entropy – continued
By Lagrange multipliers, the solution for p(x) is
where
November 18, 2018
Computer Vision

51. Maximum Entropy – cont. Maximum Entropy – continued
are determined by the constraints
But a closed form solution is not available general
Numerical solutions
November 18, 2018
Computer Vision

52. Maximum Entropy – cont. Maximum Entropy – continued
The solutions are guaranteed to exist and be unique by the following properties
November 18, 2018
Computer Vision

53. Minimax Entropy Learning (cont.)
Intuitive interpretation of minimax entropy.
November 18, 2018
Computer Vision

54. Learning A High Dimensional Density
November 18, 2018
Computer Vision

55. Toy Example I
November 18, 2018
Computer Vision

56. Toy Example II
November 18, 2018
Computer Vision

57. FRAME Model Texture modeling
The features can be anything you want n(x)
Histograms of filter responses are a good feature for textures
November 18, 2018
Computer Vision

58. FRAME Model – cont. The FRAME algorithm Initialization
Input a texture image Iobs
Select a group of K filters SK={F(1), F(2), ...., F(K)}
Compute {Hobs(a), a = 1, ....., K}
Initialize
Initialize Isyn as a uniform white noise image
November 18, 2018
Computer Vision

59. FRAME Model – cont. The FRAME algorithm – continued The algorithm
Repeat
calculate Hsyn(a), a=1,..., K from Isyn and use it as
Update by
Apply Gibbs sampler to flip Isyn for w sweeps
until
November 18, 2018
Computer Vision

60. FRAME Model – cont. The Gibbs sampler November 18, 2018
Computer Vision

61. FRAME Model – cont. Filter selection
In practice, we want a small number of “good” filters
One way to do that is to choose filters that carry the most information
In other words, minimum entropy
November 18, 2018
Computer Vision

62. FRAME Model – cont. Filter selection algorithm Initialization
November 18, 2018
Computer Vision

63. FRAME Model – cont.
November 18, 2018
Computer Vision

64. Descriptive Models – cont.
November 18, 2018
Computer Vision

65. Existing Texture Features
November 18, 2018
Computer Vision

66. Existing Feature Statistics
November 18, 2018
Computer Vision

67. Most General Feature Statistics
November 18, 2018
Computer Vision

68. Julesz Ensemble – cont. Definition
Given a set of normalized statistics on lattice 
a Julesz ensemble W(h) is the limit of the following set as   Z2 and H  {h} under some boundary conditions
November 18, 2018
Computer Vision

69. Julesz Ensemble – cont. Feature selection
A feature can be selected from a large set of features through information gain, or the decrease in entropy
November 18, 2018
Computer Vision

70. Example: 2D Flexible Shapes
November 18, 2018
Computer Vision

71. A Random Field for 2D Shape
The neighborhood
Co-linearity, co-circularity,
proximity, parallelism,
symmetry, …
November 18, 2018
Computer Vision

72. A Descriptive Shape Model
Random 2D shapes sampled from a Gibbs model.
(Zhu, 1999)
November 18, 2018
Computer Vision

73. A Descriptive Shape Model
Random 2D shapes sampled from a Gibbs model.
November 18, 2018
Computer Vision

74. Example: Face Modeling
November 18, 2018
Computer Vision

75. Generative Models
Use of hidden variables that can “explain away” the strong dependency in observed images
This requires a vocabulary
Grammars to generate images from hidden variables
Note that generative models can not be separated from descriptive models
The description of hidden variables requires descriptive models
November 18, 2018
Computer Vision

76. Generative Models – cont.
November 18, 2018
Computer Vision

77. Philosophy of Generative Models?
World structure H
observer
Features Hidden variables
November 18, 2018
Computer Vision

78. Example of Generative Model: image coding
Random variables
Parameters: wavelets
Assumptions:
1. Overcomplete basis
2. High kurtosis for iid a, e.g.
November 18, 2018
Computer Vision

79. A Generative Model noise Image I (Zhu and Guo, 2000) occlusion
additive
November 18, 2018
Computer Vision

80. Example: Texton map One layer of hidden variables: the texton map
November 18, 2018
Computer Vision

81. Learning with Generative Model
1. A generative model from H to I
2. A descriptive model for H.
November 18, 2018
Computer Vision

82. Learning with Generative Model
Learning by MLE:
3. Stochastic inference
2. Minimax entropy learning
1. Regression, fitting.
November 18, 2018
Computer Vision

83. Stochastic Inference by DDMCMC
Goal: sampling H ~ p(H | Iobs; Q)
Method: a symphony algorithm by data driven
Markov chain Monte Carlo. (Zhu, Zhang and Tu 1999)
2. Importance proposal probability density q(H | I)
1. Posterior probability p(H | Iobs; Q)
computer vision
pattern recognition
November 18, 2018
Computer Vision

84. Example of A Generative Model
An observed image:
November 18, 2018
Computer Vision

85. Data Clustering The saliency maps used as proposal probabilities
November 18, 2018
Computer Vision

86. November 18, 2018
Computer Vision

87. November 18, 2018
Computer Vision

88. A Descriptive Model for Texton Map
November 18, 2018
Computer Vision

89. Example of A Generative Model
November 18, 2018
Computer Vision

90. Data Clustering
November 18, 2018
Computer Vision

91. A Descriptive Model on Texton Map
November 18, 2018
Computer Vision

92. November 18, 2018
Computer Vision

93. November 18, 2018
Computer Vision

94. Example of A Generative Model
November 18, 2018
Computer Vision

95. A Descriptive Model for Texton Map
November 18, 2018
Computer Vision