Outline Statistical Modeling and Conceptualization of Visual Patterns S. C. Zhu, “Statistical modeling and conceptualization of visual patterns,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 6, 1-22, 2003

A Common Framework of Visual Knowledge Representation Visual patterns in natural images Natural images consist of an overwhelming number of visual patterns Generated by very diverse stochastic processes Comments Any single image normally consists of a few recognizable/segmentable visual patterns Scientifically, given that visual patterns are generated by stochastic processes, shall we model the underlying stochastic processes or model visual patterns presented in the observations from the stochastic processes? November 18, 2018 Computer Vision

A Common Framework of Visual Knowledge Representation – cont. November 18, 2018 Computer Vision

A Common Framework of Visual Knowledge Representation – cont. The image analysis as an image parsing problem Parse generic images into their constituent patterns (according to the underlying stochastic processes) Perceptual grouping when applied to points, lines, and curves processes Image segmentation when applied to region processes Object recognition when applied to high level objects November 18, 2018 Computer Vision

A Common Framework of Visual Knowledge Representation – cont. November 18, 2018 Computer Vision

A Common Framework of Visual Knowledge Representation – cont. Required components for parsing Mathematical definitions and models of various visual patterns Definitions and models are intrinsically recursive Grammars (or called rules) Which specifies the relationships among various patterns Grammars should be stochastic in nature A parsing algorithm November 18, 2018 Computer Vision

Syntactical Pattern Recognition November 18, 2018 Computer Vision

A Common Framework of Visual Knowledge Representation – cont. Conceptualization of visual patterns The concept of a pattern is an abstraction of some properties decided by certain “visual purposes” They are feature statistics computed from Raw signals Some hidden descriptions inferred from raw signals Mathematically, each pattern is equivalent to a set of observable signals governed by a statistical model November 18, 2018 Computer Vision

A Common Framework of Visual Knowledge Representation – cont. Statistical modeling of visual patterns Statistical models are intrinsic representations of visual knowledge and image regularities Due to noise and distortion in imaging process? Due to noise and distortion in the underlying generative process? Due to transformations in the underlying stochastic process? Pattern theory November 18, 2018 Computer Vision

A Common Framework of Visual Knowledge Representation – cont. Statistical modeling of visual patterns - continued Mathematical space for patterns and spaces Depends on the forms Parametric Non-parametric Attributed graphs Different models Descriptive models Bottom-up, feature-based models Generative models Hidden variables for generating images in a top-down manner November 18, 2018 Computer Vision

A Common Framework of Visual Knowledge Representation – cont. Learning a visual vocabulary Hierarchy of visual descriptions for general visual patterns Vocabulary of visual description Learning from an ensemble of natural images Vocabulary is far from enough Rich structures in physics Large vocabulary in speech and language November 18, 2018 Computer Vision

A Common Framework of Visual Knowledge Representation – cont. Computational tractability Computational heuristics for effective inference of visual patterns Discriminative models A framework Discriminative probabilities are used as proposal probabilities that drive the Markov chain search for fast convergence and mixing Generative models are top-down probabilities and the hidden variables to be inferred from posterior probabilities November 18, 2018 Computer Vision

A Common Framework of Visual Knowledge Representation – cont. Discussion Images are generated by rendering 3D objects under some external conditions All the images from one object form a low dimensional manifold in a high dimensional image space Rendering can be modeled fairly accurately Describing a 3D object requires a huge amount of data Under this setting A visual pattern simply corresponds to the manifold Descriptive model attempts to characterize the manifold Generative model attempts to learn the 3D objects and the rendering November 18, 2018 Computer Vision

3D Model-Based Recognition November 18, 2018 Computer Vision

Literature Survey To develop a generic vision system, regularities in images must be modeled The study of natural image statistics Ecologic influence on visual perception Natural images have high-order (i.e., non-Gaussian) structures The histograms of Gabor-type filter responses on natural images have high kurtosis Histograms of gradient filters are consistent over a range of scales November 18, 2018 Computer Vision

Natural Image Statistics Example November 18, 2018 Computer Vision

Analytical Probability Models for Spectral Representation Transported generator model (Grenander and Srivastava, 2000) where gi’s are selected randomly from some generator space G the weigths ai’s are i.i.d. standard normal the scales ri’s are i.i.d. uniform on the interval [0,L] the locations zi’s as samples from a 2D homogenous Poisson process, with a uniform intensity l, and the parameters are assumed to be independent of each other November 18, 2018 Computer Vision

Analytical Probability Models - continued Define Model u by a scaled -density November 18, 2018 Computer Vision

Analytical Probability Models - continued November 18, 2018 Computer Vision

Analytical Probability Models - continued November 18, 2018 Computer Vision

Analytical Probability Models - continued November 18, 2018 Computer Vision

Analysis of Natural Image Components Harmonic analysis Decomposing various classes of functions by different bases Including Fourier transform, wavelet transforms, edgelets, curvelets, and so on November 18, 2018 Computer Vision

Sparse Coding From S. C. Zhu November 18, 2018 Computer Vision

Grouping of Natural Image Elements Gestalt laws Gestalt grouping laws Should be interpreted as heuristics rather than deterministic laws Nonaccidental property November 18, 2018 Computer Vision

Illusion November 18, 2018 Computer Vision

Illusion – cont. November 18, 2018 Computer Vision

Ambiguous Figure November 18, 2018 Computer Vision

Statistical Modeling of Natural Image Patterns Synthesis-by-analysis November 18, 2018 Computer Vision

Analog from Speech Recognition November 18, 2018 Computer Vision

Modeling of Natural Image Patterns Shape-from-X problems are fundamentally ill-posed Markov random field models Deformable templates for objects Inhomogeneous MRF models on graphs November 18, 2018 Computer Vision

Four Categories of Statistical Models Descriptive models Constructed based on statistical descriptions of the image ensembles Homogeneous models Statistics are assumed to be the same for all elements in the graph Inhomogeneous models The elements of the underlying graph are labeled and different features and statistics are used at different sites November 18, 2018 Computer Vision

Variants of Descriptive Models Casual Markov models By imposing a partial ordering among the vertices of the graph, the joint probability can be factorized as a product of conditional probabilities Belief propagation networks Pseudo-descriptive models November 18, 2018 Computer Vision

Generative Models Use of hidden variables that can “explain away” the strong dependency in observed images This requires a vocabulary Grammars to generate images from hidden variables Note that generative models can not be separated from descriptive models The description of hidden variables requires descriptive models November 18, 2018 Computer Vision

Discriminative Models Approximation of posterior probabilities of hidden variables based on local features Can be seen as importance proposal probabilities November 18, 2018 Computer Vision

An Example November 18, 2018 Computer Vision

Problem formation Input: a set of images Output: a probability model Here, f(I) represents the ensemble of images in a given domain, we shall discuss the relationship between ensemble and probability later. November 18, 2018 Computer Vision

Problem formation The model p approaches the true density The Kullback-Leibler Divergence November 18, 2018 Computer Vision

Maximum Likelihood Estimate November 18, 2018 Computer Vision

Model Pursuit 1. What is W -- the family of models ? 2. How do we augment the space W? November 18, 2018 Computer Vision

Two Choices of Models The exponential family – descriptive models --- Characterize images by features and statistics 2. The mixture family -- generative models --- Characterize images by hidden variables Features are deterministic mathematical transforms of an image. Hidden variables are stochastic and are inferred from an image. November 18, 2018 Computer Vision

I: Descriptive Models Step 1: extracting image features/statistics as transforms For example: histograms of Gabor filter responses. Other features/statistics: Gabors, geometry, Gestalt laws, faces. November 18, 2018 Computer Vision

I.I: Descriptive Models Step 2: using features/statistics to constrain the model Two cases: On infinite lattice Z2 --- an equivalence class. On any finite lattice --- a conditional probability model. image space on Z2 image space on lattice L November 18, 2018 Computer Vision

I.I Descriptive Model on Finite Lattice Modeling by maximum entropy: Subject to: Remark: p and f have the same projected marginal statistics. November 18, 2018 Computer Vision

Minimax Entropy Learning For a Gibbs (max. entropy) model p, this leads to the minimax entropy principle (Zhu,Wu, Mumford 96,97) November 18, 2018 Computer Vision

FRAME Model FRAME model Filtering, random field, and maximum entropy A well-defined mathematical model for textures by combining filtering and random field models November 18, 2018 Computer Vision

I.I Descriptive Model on Finite Lattice The FRAME model (Zhu, Wu, Mumford, 1996) This includes all Markov random field models. Remark: all known exponential models are from maxent., and maxent was proposed in Physics (Jaynes, 1957). The nice thing is that it provides a parametric model integrating features. November 18, 2018 Computer Vision

I.I Descriptive Model on Finite Lattice Two learning phases: 1. Choose information bearing features -- augmenting the probability family. 2. Compute the parameter L by MLE -- learning within a family. November 18, 2018 Computer Vision

Maximum Entropy Maximum entropy Is an important principle in statistics for constructing a probability distribution on a set of random variables Suppose the available information is the expectations of some known functions n(x), that is Let W be the set of all probability distributions p(x) which satisfy the constraints November 18, 2018 Computer Vision

Maximum Entropy – cont. Maximum Entropy – continued According to the maximum entropy principle, a good choice of the probability distribution is the one that has the maximum entropy subject to November 18, 2018 Computer Vision

Maximum Entropy – cont. Maximum Entropy – continued By Lagrange multipliers, the solution for p(x) is where November 18, 2018 Computer Vision

Maximum Entropy – cont. Maximum Entropy – continued are determined by the constraints But a closed form solution is not available general Numerical solutions November 18, 2018 Computer Vision

Maximum Entropy – cont. Maximum Entropy – continued The solutions are guaranteed to exist and be unique by the following properties November 18, 2018 Computer Vision

Minimax Entropy Learning (cont.) Intuitive interpretation of minimax entropy. November 18, 2018 Computer Vision

Learning A High Dimensional Density November 18, 2018 Computer Vision

Toy Example I November 18, 2018 Computer Vision

Toy Example II November 18, 2018 Computer Vision

FRAME Model Texture modeling The features can be anything you want n(x) Histograms of filter responses are a good feature for textures November 18, 2018 Computer Vision

FRAME Model – cont. The FRAME algorithm Initialization Input a texture image Iobs Select a group of K filters SK={F(1), F(2), ...., F(K)} Compute {Hobs(a), a = 1, ....., K} Initialize Initialize Isyn as a uniform white noise image November 18, 2018 Computer Vision

FRAME Model – cont. The FRAME algorithm – continued The algorithm Repeat calculate Hsyn(a), a=1,..., K from Isyn and use it as Update by Apply Gibbs sampler to flip Isyn for w sweeps until November 18, 2018 Computer Vision

FRAME Model – cont. The Gibbs sampler November 18, 2018 Computer Vision

FRAME Model – cont. Filter selection In practice, we want a small number of “good” filters One way to do that is to choose filters that carry the most information In other words, minimum entropy November 18, 2018 Computer Vision

FRAME Model – cont. Filter selection algorithm Initialization November 18, 2018 Computer Vision

FRAME Model – cont. November 18, 2018 Computer Vision

Descriptive Models – cont. November 18, 2018 Computer Vision

Existing Texture Features November 18, 2018 Computer Vision

Existing Feature Statistics November 18, 2018 Computer Vision

Most General Feature Statistics November 18, 2018 Computer Vision

Julesz Ensemble – cont. Definition Given a set of normalized statistics on lattice  a Julesz ensemble W(h) is the limit of the following set as   Z2 and H  {h} under some boundary conditions November 18, 2018 Computer Vision

Julesz Ensemble – cont. Feature selection A feature can be selected from a large set of features through information gain, or the decrease in entropy November 18, 2018 Computer Vision

Example: 2D Flexible Shapes November 18, 2018 Computer Vision

A Random Field for 2D Shape The neighborhood Co-linearity, co-circularity, proximity, parallelism, symmetry, … November 18, 2018 Computer Vision

A Descriptive Shape Model Random 2D shapes sampled from a Gibbs model. (Zhu, 1999) November 18, 2018 Computer Vision

A Descriptive Shape Model Random 2D shapes sampled from a Gibbs model. November 18, 2018 Computer Vision

Example: Face Modeling November 18, 2018 Computer Vision

Generative Models Use of hidden variables that can “explain away” the strong dependency in observed images This requires a vocabulary Grammars to generate images from hidden variables Note that generative models can not be separated from descriptive models The description of hidden variables requires descriptive models November 18, 2018 Computer Vision

Generative Models – cont. November 18, 2018 Computer Vision

Philosophy of Generative Models ? World structure H observer Features Hidden variables November 18, 2018 Computer Vision

Example of Generative Model: image coding Random variables Parameters: wavelets Assumptions: 1. Overcomplete basis 2. High kurtosis for iid a, e.g. November 18, 2018 Computer Vision

A Generative Model noise Image I (Zhu and Guo, 2000) occlusion additive November 18, 2018 Computer Vision

Example: Texton map One layer of hidden variables: the texton map November 18, 2018 Computer Vision

Learning with Generative Model 1. A generative model from H to I 2. A descriptive model for H. November 18, 2018 Computer Vision

Learning with Generative Model Learning by MLE: 3. Stochastic inference 2. Minimax entropy learning 1. Regression, fitting. November 18, 2018 Computer Vision

Stochastic Inference by DDMCMC Goal: sampling H ~ p(H | Iobs; Q) Method: a symphony algorithm by data driven Markov chain Monte Carlo. (Zhu, Zhang and Tu 1999) 2. Importance proposal probability density q(H | I) 1. Posterior probability p(H | Iobs; Q) computer vision pattern recognition November 18, 2018 Computer Vision

Example of A Generative Model An observed image: November 18, 2018 Computer Vision

Data Clustering The saliency maps used as proposal probabilities November 18, 2018 Computer Vision

November 18, 2018 Computer Vision

November 18, 2018 Computer Vision

A Descriptive Model for Texton Map November 18, 2018 Computer Vision

Example of A Generative Model November 18, 2018 Computer Vision

Data Clustering November 18, 2018 Computer Vision

A Descriptive Model on Texton Map November 18, 2018 Computer Vision

November 18, 2018 Computer Vision

November 18, 2018 Computer Vision

Example of A Generative Model November 18, 2018 Computer Vision

A Descriptive Model for Texton Map November 18, 2018 Computer Vision