Outline Texture modeling - continued Julesz ensemble

Visual Perception Modeling FRAME Model – review FRAME model Filtering, random field, and maximum entropy A well-defined mathematical model for textures by combining filtering and random field models Maximum entropy is used when constructing the probability distribution on the image space Minimum entropy is used when selecting filters from a large bank of filters Together this is called min-max entropy principle 2/24/2019 Visual Perception Modeling

Visual Perception Modeling FRAME Model – review Maximum Entropy Distribution Given the expectations of some functions, the maximum entropy solution for p(x) is where 2/24/2019 Visual Perception Modeling

Visual Perception Modeling FRAME Model – review Maximum Entropy – continued are determined by the constraints Gradient ascend to maximize 2/24/2019 Visual Perception Modeling

Visual Perception Modeling Julesz Ensemble The original texture modeling question What features and statistics are characteristics of a texture pattern, so that texture pairs that share the same features and statistics cannot be told apart by pre-attentive human visual perception? --- Julesz, 1962 2/24/2019 Visual Perception Modeling

Summary of Existing Texture Features 2/24/2019 Visual Perception Modeling

Existing Feature Statistics 2/24/2019 Visual Perception Modeling

Most General Feature Statistics 2/24/2019 Visual Perception Modeling

Visual Perception Modeling Julesz Ensemble – cont. Definition Given a set of normalized statistics on lattice  a Julesz ensemble W(h) is the limit of the following set as   Z2 and H  {h} under some boundary conditions 2/24/2019 Visual Perception Modeling

Visual Perception Modeling Julesz Ensemble – cont. Feature selection A feature can be selected from a large set of features through information gain, or the decrease in entropy 2/24/2019 Visual Perception Modeling

Visual Perception Modeling Julesz Ensemble – cont. 2/24/2019 Visual Perception Modeling

Visual Perception Modeling Julesz Ensemble – cont. Sampling the Julesz ensemble In the Julesz ensemble, a texture type is defined as all the images sharing the observed statistics and features It is an inverse problem in order to generate texture images or verify the statistics The problem is again the dimensionality If the image size is 256x256 and each pixel can have 8 values, there are 865536 different images Markov chain Monte-Carlo algorithms 2/24/2019 Visual Perception Modeling

Visual Perception Modeling Julesz Ensemble – cont. Given observed feature statistics {H(a)obs}, we associate an energy with any image I as Then the corresponding Gibbs distribution is The q(I) can be sampled using a Gibbs sampler or other Markov chain Monte-Carlo algorithms 2/24/2019 Visual Perception Modeling

Image Synthesis Algorithm Compute {Hobs} from an observed texture image Initialize Isyn as any image, and T as T0 Repeat Randomly pick a pixel v in Isyn Calculate the conditional probability q(Isyn(v)| Isyn(-v)) Choose new Isyn(v) under q(Isyn(v)| Isyn(-v)) Reduce T gradually Until E(I) < e 2/24/2019 Visual Perception Modeling

A Texture Synthesis Example Observed image Initial synthesized image 2/24/2019 Visual Perception Modeling

A Texture Synthesis Example Image patch Energy Conditional probability Temperature 2/24/2019 Visual Perception Modeling

A Texture Synthesis Example - continued Average spectral histogram error 2/24/2019 Visual Perception Modeling

Texture Synthesis Examples - continued Observed image Synthesized image 2/24/2019 Visual Perception Modeling

Texture Synthesis Examples - continued Observed image Synthesized image 2/24/2019 Visual Perception Modeling

Texture Synthesis Examples - continued Mud image Synthesized image 2/24/2019 Visual Perception Modeling

Texture Synthesis Examples - continued Observed image Synthesized image 2/24/2019 Visual Perception Modeling

Texture Synthesis Examples - continued Observed image Synthesized image 2/24/2019 Visual Perception Modeling

Texture Synthesis Examples - continued Synthesized image Original cheetah skin patch 2/24/2019 Visual Perception Modeling

Texture Synthesis Examples - continued Observed image Synthesized image 2/24/2019 Visual Perception Modeling

Texture Synthesis Examples - continued Observed image Synthesized image 2/24/2019 Visual Perception Modeling

Texture Synthesis Examples - continued Observed image Synthesized image 2/24/2019 Visual Perception Modeling

An Synthesis Example for Fun 2/24/2019 Visual Perception Modeling

Comparison with Texture Synthesis Method - continued An example from Heeger and Bergen’s algorithm Cross image Heeger and Bergen’s Our result 2/24/2019 Visual Perception Modeling

Visual Perception Modeling Julesz Ensemble – cont. Remarks The results shown here are based on histograms of filter responses However, the Julesz ensemble applies to any features/statistics of your choice You can also define Julesz ensemble for images other than textures 2/24/2019 Visual Perception Modeling

Visual Perception Modeling Julesz Ensemble – cont. Applications This essentially provides a framework to systematically verify the sufficiency of chosen features/statistics Normally, features/statistics are evaluated empirically. In other words, features are evaluated on a limited number of images 2/24/2019 Visual Perception Modeling