Image Segmentation Techniques Advanced Multimedia Databases Jeff Strickrott

Content Introduction Image segmentation methods Summary Template Transformation based Probabilistic modeling Summary 11/19/2018 Image Segmentation

Introduction What is the goal behind building multimedia databases? We want to ask questions based on content. Of the form: what images show the red car in front of the house? Why do we need to segment image? To represent content! Content Represented by objects or spatial regions in the image. To support region/content based queries. 11/19/2018 Image Segmentation

Template Based Segmentation A template is a generalization of the features of some object. Spatially distributed. Features can be pixel: intensity, color, relations between regions, etc. Detect object through correlation (match object to template). Detect objects the same size as template. Requires domain knowledge (to create template). Result: Provides semantic segmentation of image. 11/19/2018 Image Segmentation

Face Templates Examples Scasselati Ratio-Metric Template [Scaz 98] Liu’s Intensity Based Template [Liu 00] Two templates for real-time face detection: Average intensity & Ratio-metric 11/19/2018 Image Segmentation

Using Templates Algorithm (Naive): Select region of image. Slide template over region, moving “i” pixels at a time. Compute correspondence for each location. Locations with high correspondence are locations of objects. Resize (Downsample) to find faces of different sizes. Repeat Dynamic programming can be used to make search more efficient (Liu 00). 11/19/2018 Image Segmentation

Results Liu’s & Wang’s Face Temp. Video Scene Segmentation: on new face start new scene. 11/19/2018 Image Segmentation

Transformation Based Segmentation: Image Mapping Map Image from intensity space to new feature space. Reason: Easier to detect relevant information about objects in the image. Transform Examples: Fourier Transform, DCT, Wavelet. 11/19/2018 Image Segmentation

Wavelet Image Representation Maps 2D image into varying spatial and frequency resolution space. Haar Basis function encodes relationship between neighboring pixel intensities (edges). O(n), n = the number of pixels. Linear transform (invertible/lossless). 11/19/2018 Image Segmentation

Poggio’s People Detection System Generate Templates of Objects in Domain from over-complete Haar wavelet. Averaged over transformed images of objects to get relevant features (multiple views). Spatial correspondence between transform coefficients and object location. Template encodes relationship between regions in image (absolute or ratio of intensities). Encode spatial relationship. 11/19/2018 Image Segmentation

Standard Haar Wavelet L = Lowpass H = Highpass XYz X Row Y Column Z level 11/19/2018 Image Segmentation

People Templates Each square one wavelet Black: Relevant edge Gray: random patterns Black: Relevant edge information [Poggio 00] 11/19/2018 Image Segmentation

Detection System Algorithm: Proven also on faces and cars. Image transformed. Template matching done in wavelet space. Shift of wavelet coefficient = 4 pixels in image. Image must be resized to find objects not the size of template. Proven also on faces and cars. Real time people detection in complex scene. [Poggio 00] 11/19/2018 Image Segmentation

Probabilistic Modeling Generic Model, No Segmentation. We know nothing about image domain. Need generic criteria that captures local and global information. Should perform well on texture and high color images. Should work on sections of images. No assumptions about similarity metric or dimension of feature space. 11/19/2018 Image Segmentation

Retrieval as Classification Problem Model content of images as class(es) in some set of classes. Bayesian classifier used to minimize the misclassification of image content. Solve problem: g*(X) = argmaxi P(X | Y=i) P(Y =i). X can be an image, subset of image, other types of data (text, audio, etc.). Will work on any type of features used to represent a class. 11/19/2018 Image Segmentation

Embedded Mixture Model Vasconcelos showed feature representation choice is major factor in minimizing classification error. Proposed solution to Bayesian min problem: DCT image in blocks (separate or overlapping) Model as Mixture of Gaussian distributions, this is a weighted sum of Gaussian probability densities. 11/19/2018 Image Segmentation

EMM Performance Can model content from images with complex color and texture features (outperforms histogram based and texture based models for retrieval). Works on compressed images. Supports region based queries. Classification algorithm runs in O(C2) time. C on the order of 8-16. Prior probability term P(Y =i) can be used for learning via relevance feedback. Expectation Maximization algorithm used for learning weight, mean and covariance parameters (w, m,S). More work must be done to make this efficient. 11/19/2018 Image Segmentation

Region Based Retrieval Content based Retrieval on a Region of an image. Database of mixed objects (in color). Results of query on Onion [Vasc 00] 11/19/2018 Image Segmentation

Summary We have investigated three techniques for segmenting images that require varying amounts of domain knowledge. Closest similarity with segmentation methods discussed in class are the clustering and stochastic modeling methods. Two methods work in real-time while fourth designed for generic domains and offline classification. 11/19/2018 Image Segmentation

References [Liu 00] Liu, Z. and Wang, Y., "Face Detection and Tracking in Video Using Dynamic Programming," ICIP-2000, Vancouver, Canada. [Poggio 00] Papageorgiou, C. and Poggio, T., A Trainable System for Object Detection International journal of computer vision, vol. 38, pp. 15-35, 2000. [Scaz 98] Scassellati, B., "Eye Finding via Face Detection for a Foveated, Active Vision System," AAAI 98 [Stau 99] Stauffer, C. and Grimson, W. E. L., "Adaptive Background Mixture Models for Real-Time Tracking," IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 246, 1999. [Vasc 00]Vasconcelos, N. and Lippman, A., "A Probabilistic Architecture for Content-based Image Retrieval," IEEE Conference on Computer Vision and Pattern Recognition (ICPR 2000), Hilton Head Island, SC, 2000. 11/19/2018 Image Segmentation

Segmentation Methods Color Histogram Split and Merge Region Growing segment in histogram space no domain knowledge Split and Merge Segment image into regions () Region Growing Grow regions around seed point based on similarity measure. Similarity criteria Clustering Stochastic 11/19/2018 Image Segmentation