Ch.8 Efficient Coding of Visual Scenes by Grouping and Segmentation Bayesian Brain Tai Sing Lee and Alan L. Yuille Heo, Min-Oh

Contents Introduction Computational Theories for Scene Segmentation  Weak-membrane model A Computational Algorithm for the Weak-Membrane Model Generalization of the Weak-Membrane Model  Region competition model  Affinity-based model  Integration segmentation with shape properties Biological Evidence  Go on to the next speaker…

Introduction Conjecture  Areas V1 and V2 compute a segmentation for more compact and parsimonious encoding of images  The neural processes are representative of neural mechanisms that operate in other areas of the brain for performing other higher-level tasks.

Computational Theories for Scene Segmentation Choosing the representation W of the regions which best fits the image data D based on Minimum Description Length (MDL) principle. Taking logarithm Encoding cost

Computational Theories for Scene Segmentation Weak-Membrane Model Data term: Gaussian white noise Smoothness term: variation on the estimated image intensity is smooth within each region Penalty term: On the length of the boundaries d(x,y) : intensity values of images (input image) u(x,y) : unobserved smoothed version of d(x,y) B : set of the boundaries between regions E(u,B) : encoding cost Mumford D. and Shah J, Optimal approximations by piecewise smooth functions and associated variational problems. Communications on Pure and Applied Mathematics, 42: , 1989

Computational Theories for Scene Segmentation Reformulation  to deal with boundaries easily d(x,y) : intensity values of images (input image) u(x,y) : unobserved smoothed version of d(x,y) l(x,y) : line process variables which take on values in [0,1] E(u,l) : encoding cost Ambrosio L, Tortorelli VM, On the approximations of free discontinuity problems. Preprints di Matermatica, 86, Pisa, Italy: Scuola Normale Superiore, 1990

A Computational Algorithm for the Weak-Membrane Model

Continuation methods  At large p, the energy function is convex.  As p approaches zero, the energy function transform back to the original function which can have many local minima.  Strategy (successive gradual relaxation)  Initialize p 0 with large value, perform steepest descent.  And decrease p to p 1, do it again.  Repeat the process.  Empirically it yields good results

A Computational Algorithm for the Weak-Membrane Model The steepest descent equations  The system relaxes to an equilibrium as p decrease from 1 to 0  As p decrease, the boundary responses contract spatially to the exact location R u : positive rate constant w.r.t u R l : positive rate constant w.r.t l

A Computational Algorithm for the Weak-Membrane Model

Segmentation of an image by the weak-membrane model

Generalization of the Weak-Membrane Model Natural Images have …  Texture  Shade  Color  Shape  Material Properties  Lighting conditions How can we segment images with these properties?  Region competition model  Affinity-based model  Integrate segmentation with the estimation of 3D shape properties

Generalization of the Weak-Membrane Model Region competition models R r : Region sets for each model a r : model type index variable θ r : the parameters of the model Tu Z, Zhu SC, Image segmentation by data-driven Markov chain Monte Carlo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5), 2002

Generalization of the Weak-Membrane Model Regions can be encoded as one of these types  Gaussian model of the intensity in the region  Shading model  the image intensity follows a simple parameterized form  Simple texture/clutter model

Generalization of the Weak-Membrane Model Affinity-based Model  Affinity weights w ij between different image pixels v i and v j  Define a graph with image pixels and the weights.  Assigning a label to each image pixel so that pixels with the same labels define a region n: the number of pixels k : the number of labels : Yu SX, Shi J, Multiclass Spectral Clustering. Proc. of the 9 th International Conference on Computer Vision, , 2003

Generalization of the Weak-Membrane Model Integration segmentation with shape properties  Additional constraint on the surface normal  Occlusion border Ω : a subregion of the image d(x, y) : the intensity of the image at location (x, y) : reflectance function based on standard Lambertian model is surface gradient at position (x, y) is the light source

Generalization of the Weak-Membrane Model Example: Surface interpolation process  (a) input image  (b) initial estimate of surface by needle map  (c) rendering with (b)  (d) final estimate of surface orientations  (e) shaded rendering

Biological Evidence Let me introduce the next speaker!