The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz

The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) n Blobs Need for blobs Extracting blobs –Thresholding, region growing, mean-shift –Minimum spanning trees n Object extraction Intelligent scissors (also known as live-wire) Snakes Region+boundary methods n Local, greedy, and global processing Extra Reading: Sonka at.al. Ch. 5 Gonzalez and Woods, Ch. 10

The University of Ontario HW assignment 1 n 2D image segmentation Live-wire (based on Dijkstra algorithm for shortest paths on a graph) Implement and experiment on two images Bonus for comparing with local/greedy methods Some useful code libraries and sample images will be posted within a couple of days

The University of Ontario Goal: Extract “Blobs” n What are “blobs”? Regions of an image that are somehow coherent n Why? Object extraction, object removal, compositing, etc. …but are “blobs” objects? No, not in general

The University of Ontario Blob’s coherence n Simplest way to define blob coherence is as similarity in brightness or color: The tools become blobs The house, grass, and sky make different blobs

The University of Ontario Why is this useful? AIBO RoboSoccer (VelosoLab)

The University of Ontario Ideal Segmentation

The University of Ontario Result of Segmentation

The University of Ontario Thresholding n Basic segmentation operation: mask(x,y) = 1 if im(x,y) > T mask(x,y) = 0 if im(x,y) < T n T is threshold User-defined Or automatic n Same as histogram partitioning:

The University of Ontario Sometimes works well… What are potential Problems?

The University of Ontario …but more often not Adaptive thresholding

The University of Ontario Region growing n Is this same as global threshold? Start with initial set of pixels K (initial seed(s)) Add to pixels p in K their neighbors q if |Ip-Iq| < T Repeat until nothing changes

The University of Ontario Region growing

The University of Ontario What can go wrong with region growing ? Region growth may “leak” through a single week spot in the boundary

The University of Ontario Region growing See region leaks into sky due to a weak boundary between them

The University of Ontario Due to Pedro Felzenszwalb and Daniel Huttenlocher Motivating example This image has three perceptually distinct regions Difference along border between A and B is less then differences within C A CB Q: Where would image thresholding fail? Q: Where would region growing fail? A: Region A would be divided in two sets and region C will be split into a large number of arbitrary small subsets A: Either A and B are merged or region C is split into many small subsets Also, B and C are merged

The University of Ontario Color-Based Blob Segmentation n Automatic Histogram Partitioning Given image with N colors, choose K Each of the K colors defines a region –not necessarily contiguous Performed by computing color histogram, looking for modes This is what happens when you downsample image color range, for instance in Photoshop

The University of Ontario Finding Modes in a Histogram n How Many Modes Are There? Easy to see, hard to compute

The University of Ontario Mean Shift [ Comaniciu & Meer] 1.Initialize random seed, and fixed window 2.Calculate center of gravity ‘x’ of the window (the“mean”) 3.Translate the search window to the mean 4.Repeat Step 2 until convergence Iterative Mode Search x o x x mode

The University of Ontario Mean-Shift

The University of Ontario Mean-shift results More ExamplesMore Examples:

The University of Ontario Issues: n Although often useful, all these approaches work only some of the time, and are considered rather “hacky”. n Can’t even handle our tiger: Problem is that blobs != objects!

The University of Ontario Extracting objects n How could this be done?

The University of Ontario Extracting objects n Many approaches proposed color cues region cues contour cues n We will consider a few of these n Today: Intelligent Scissors (contour-based) –E. N. Mortensen and W. A. Barrett, Intelligent Scissors for Image Composition, in ACM Computer Graphics (SIGGRAPH `95), pp , 1995Intelligent Scissors for Image Composition

The University of Ontario Intelligent Scissors

The University of Ontario Intelligent Scissors n Approach answers a basic question Q: how to find a path from seed to mouse that follows object boundary as closely as possible? A: define a path that stays as close as possible to edges

The University of Ontario Intelligent Scissors n Basic Idea Define edge score for each pixel –edge pixels have low cost Find lowest cost path from seed to mouse seed mouse Questions How to define costs? How to find the path?

The University of Ontario Path Search (basic idea) n Graph Search Algorithm Computes minimum cost path from seed to all other pixels Note: diagonal “paths” are scaled by a factor. … Why?

The University of Ontario Good segmentation method should have Invariance to Image Rotation After object rotation L L Path’s cost along the top boundaryPath’s cost along the top-left boundary After diagonal links are adjusted by image gradient scores

The University of Ontario Instead of nodes, image gradient scores can be assigned directly to graph edges n Graph node for every pixel p link between every adjacent pair of pixels, p,q cost c for each link n Note: each link has a cost this is a little different than the figure before where each pixel (graph node) had a cost p q c n Treat the image as a graph

The University of Ontario Defining the costs n Treat the image as a graph n Want to hug image edges: how to define cost of a link? p q c the link should follow the intensity edge –want intensity to change rapidly to the link c  - |difference of intensity to the link| T T

The University of Ontario Defining the costs p q c n c can be computed using a cross-correlation filter assume it is centered at p n Also typically scale c by its length set c = (max-|filter response|) –where max = maximum |filter response| over all pixels in the image

The University of Ontario Defining the costs p q c 1 w 11 n c can be computed using a cross-correlation filter assume it is centered at p n Also typically scale c by its length set c = (max-|filter response|) –where max = maximum |filter response| over all pixels in the image Why no scalar?

The University of Ontario (see Cormen et.al. “Introduction to Algorithms”, p.595) Dijkstra’s shortest path algorithm link cost 1.init node costs to , set p = seed point, cost(p) = 0 2.expand p as follows: for each of p’s neighbors q that are not expanded set cost(q) = min( cost(p) + c pq, cost(q) ) ALGORITHM

The University of Ontario Dijkstra’s shortest path algorithm init node costs to , set p = seed point, cost(p) = 0 2.expand p as follows: for each of p’s neighbors q that are not expanded set cost(q) = min( cost(p) + c pq, cost(q) ) –if q’s cost changed, make q point back to p put q on the ACTIVE list (if not already there) ALGORITHM

The University of Ontario Dijkstra’s shortest path algorithm init node costs to , set p = seed point, cost(p) = 0 2.expand p as follows: for each of p’s neighbors q that are not expanded set cost(q) = min( cost(p) + c pq, cost(q) ) –if q’s cost changed, make q point back to p put q on the ACTIVE list (if not already there) 3.set r = node with minimum cost on the ACTIVE list 4.repeat Step 2 for p = r ALGORITHM

The University of Ontario Dijkstra’s shortest path algorithm init node costs to , set p = seed point, cost(p) = 0 2.expand p as follows: for each of p’s neighbors q that are not expanded set cost(q) = min( cost(p) + c pq, cost(q) ) –if q’s cost changed, make q point back to p put q on the ACTIVE list (if not already there) 3.set r = node with minimum cost on the ACTIVE list 4.repeat Step 2 for p = r ALGORITHM

The University of Ontario Dijkstra’s shortest path algorithm init node costs to , set p = seed point, cost(p) = 0 2.expand p as follows: for each of p’s neighbors q that are not expanded set cost(q) = min( cost(p) + c pq, cost(q) ) –if q’s cost changed, make q point back to p put q on the ACTIVE list (if not already there) 3.set r = node with minimum cost on the ACTIVE list 4.repeat Step 2 for p = r ALGORITHM

The University of Ontario Path Search (basic idea) A B Dijkstra algorithm - processed nodes (distance to A is known) - active nodes (front) - active node with the smallest distance value

The University of Ontario Dijkstra’s shortest path algorithm n Properties It computes the minimum cost path from the seed to every node in the graph. This set of minimum paths is represented as a tree Running time, with N pixels: –O(N 2 ) time if you use an active list –O(N log N) if you use an active priority queue (heap) –takes < second for a typical (640x480) image Once this tree is computed once, we can extract the optimal path from any point to the seed in O(N) time. –it runs in real time as the mouse moves What happens when the user specifies a new seed?

The University of Ontario Livewire extensions n Directed graphs n Restricted search space Restricted domain (e.g. near a priori model) Restricted backward search n Different edge weight functions Image-Edge strength Image-Edge Curvature Proximity to known approximate model/boundary n Multi-resolution processing

The University of Ontario Results