Segmentation Lecture 04 Computer Vision

Material Citations Dr George Stockman Professor Emeritus, Michigan State University Dr Mubarak Shah Professor, University of Central Florida The Robotics Institute Carnegie Mellon University Dr David A. Forsyth Professor, University of Illinois at Urbana- Champaign Dr Kristen Grauman Associate Professor, University of Texas at Austin

Suggested Readings Chapter 10 Linda G. Shapiro and George Stockman, “Computer Vision”, Upper Saddle River, NJ, Prentice Hall, 2001 . Chapter 15 David A. Forsyth and Jean Ponce, “Computer Vision A Modern Approach”, 2nd edition, Prentice Hall, Inc., 2003. Chapter 5 Richard Szeliski , “Computer Vision: Algorithms and Applications”, Springer; 2011.  Chapter 3 Mubarak Shah, “Fundamentals of Computer Vision”.

Recap (Edge Detection) Marr-Hildreth and Canny edge detectors Gaussian smoothing Compute derivatives In x and y directions Find gradient magnitude Threshold gradient magnitude Difference between Marr-Hildreth and Canny Marr-Hildreth use 2nd order derivative Marr-Hildreth thresholds slope of zero-crossings

Marr-Hildreth Edge Detector Image 2g(x,y) Find zero-crossings compute slope Threshold

Canny Edge Detector gx(x,y) gy(x,y) Image Gradient magnitude direction Non-maximum suppression Hysteresis thresholding

Segmentation Thresholding Edge-based methods Region-based techniques Clustering K-Means etc. Graph partitioning Graph cuts etc. Connectivity-preserving Active Contour Model etc. And more

Segmentation – Some applications Object detection Pedestrian detection Face detection Brake light detection Locate objects in satellite images (roads, forests, crops, etc.) Medical imaging Locate tumors and other pathologies Measure tissue volumes Diagnosis, study of anatomical structure Surgery planning Virtual surgery simulation Intra-surgery navigation

Segmentation – Some applications Recognition Tasks Face recognition Fingerprint recognition Iris recognition Traffic control systems Video surveillance

Segmentation - Example application Object Recognition using region properties Training For all training samples of each model object Segment the image Compute region properties (features) Recognition Given an image of unknown object, Compute its feature vector Compare with the training set

Segmentation - Example application MPEG4 Compression - Object Based Compression Advantages of OBC High compression ratio Allows insertion deletion of objects How does it work? Find objects (Object Segmentation) Code objects and their locations Build mosaics of globally static objects Render scene at receiver

Region Segmentation

Layer Representation

Segmentation Find set of regions R1, R2, ….,Rn such that All pixels in region i satisfy some similarity constraint

Similarity Constraints All pixels in any sub-image must have the same gray levels All pixels in any sub-image must not differ more than some threshold The standard deviation of gray levels in any sub-image must be small

Simple Segmentation

Image Histogram Histogram graphs the number of pixels with a particular gray level as a function of the image of gray levels. gray level number of pixels

Segmentation using Histogram - Simple case

Segmentation using Histogram - Simple case

Realistic Histograms Bimodal Histograms – Having two peaks Not realistic Real (noise)

Realistic Histograms Smooth out noise Convolve histogram by averaging or 1D Gaussian filter peak valley Identify peaks and valleys by detecting local maxima and local minima

Segmentation using Histogram Compute the histogram of a given image Smooth the histogram by averaging peaks and valleys in the histogram Detect good peaks by applying thresholds at the valleys Segment the image into several binary images using thresholds at the valleys Apply connected component algorithm to each binary image and find connected regions.

Good Peaks Peakiness = Sharpness of a peak = Smaller this ratio -sharper the peak. When ration = 1, it’s a rectangle Sharpness of a peak = Ht. of valley to ht. of peak = W = Width of the peak in grey level range from valley to valley P = Height of the peak Va, Vb = Height of valleys N = Number of pixels in the image covered by the peak If peakiness is greater then some threshold then it will be used for segmentation

Segmentation using Histograms Select the valleys as thresholds Apply threshold to histogram Label the pixels within the range of a threshold with same label, i.e., a, b, c … or 1, 2, 3 …

Segmentation using Histograms Have only used the grey level information for segmentation Histograms do not give any spatial information For connected group of pixels we need to apply connected component algorithm

Connectedness x 4 connected 8 connected 6 connected

Connected Components Disjoint segments with same labels need to be split may be added to segment c

Recursive Connected Component Algorithm Scan the binary image left to right, top to bottom If there is an unlabeled pixel with a value of ‘1’ assign a new label to it Recursively check the neighbors of the pixel in step 2 and assign the same label if there are unlabeled with a value of ‘1’ Stop when all the pixels of value ‘1’ have been labeled

Example - Detecting Finger Tips (marked white)

Example - Segmenting a bottle image 93 peaks

Example - Segmenting a bottle image Smoothed histogram (averaging using mask Of size 5) 54 peaks (once smoothing) After peakiness 7 Smoothed histogram 21 peaks (twice) After peakiness 7 Smoothed histogram 11 peaks (three times) After peakiness 4

Example - Segmenting a bottle image Region from Peak 1 (0,40) Region from Peak 2 (40, 116) Region from Peak 3 (116,243) Region from Peak 4 (243,255)

How to determine the three main intensities? Example - Segmenting Kristen Grauman, University of Texas at Austin How to determine the three main intensities? Kristen Grauman, University of Texas at Austin

Segmentation and Grouping Purpose Obtain a compact representation from an image/motion sequence/set of tokens Tokens: Whatever we need to group (pixels, points, surface elements, etc.) Major approaches Forming image segments by grouping (or clustering) Collect together tokens that “belong together” Fitting a model - Fitting a line to collection of points and associating the lines - Associate a model with tokens (e.g. correspondence between pixels of the two shots of same scene) - Issues Which model? Which token goes to which element?

Segmentation and Grouping Partitioning: Top down segmentation – Decomposition of large sets into smaller ones according to their eligibility to the selected model (or criterion) • Decompose an image into regions that have coherent color or texture • Decompose a video sequence into segments of video showing about the same stuff (static area) from same view point – Tokens belong together because they lie on the same object Grouping: Bottom up segmentation – Tokens belong together because they are locally coherent

Basic ideas of grouping in humans Human visual illusions Figure-ground discrimination Grouping can be seen in terms of allocating some elements to a figure, some to ground It is difficult to differentiate between figure and ground

Basic ideas of grouping in humans Human visual illusions Gestalt properties Gestalt: A whole or a group Muller-Lyer effect: Elements in a collection of elements can have properties that result from relationships Gestalt Qualitat: The set of relationship that makes it a group Gestalt factors: A series of factors affect whether elements should be grouped together A human visual system uses them for grouping Muller-Lyer effect

Gestalt factors Not grouped Proximity Similarity Common fate Similar motion tendency Some criteria that tend to cause tokens to be grouped.

Gestalt factors Similarity Some criteria that tend to cause tokens to be grouped. http://chicagoist.com/attachments/chicagoist_alicia/GEESE.jpg http://wwwdelivery.superstock.com/WI/223/1532/PreviewComp/SuperStock_1532R-0831.jpg

Gestalt factors Symmetry Some criteria that tend to cause tokens to be grouped. http://seedmagazine.com/content/article/beauty_is_in_the_processing-time_of_the_beholder/

Gestalt factors Proximity Some criteria that tend to cause tokens to be grouped. http://www.capital.edu/CapitalError.aspx?aspxerrorpath=/Resources/Images/outside6_035.jpg

Gestalt factors Common Fate Some criteria that tend to cause tokens to be grouped. Image credit: Arthus-Bertrand (via F. Durand)

Gestalt factors Parallelism Symmetry Continuity Closure Parallel curves or tokens tend to be grouped together Tokens or curves that lead symmetric groups tend to be grouped together Tokens that lead to continuous curves tend to be grouped together More such Tokens or curves that tend to lead to closed curves tend to be grouped together

Visual illusions - Grouping in humans Occlusions and their effects Can you understand what this picture is about? Hint: There are five numerals Occlusion cues seem to be very important in grouping

Visual illusions - Grouping in humans What numbers do you see? but easy in this Continuity factor can be used

Visual illusions - Grouping in humans Illusory contours; an object appears to be occluding

Visual illusions - Grouping in humans Elevator buttons with high probability to arrive at the wrong floor was high Elevator buttons with high probability to arrive at the right floor - Only because buttons are properly segmented