Qualifying Exam: Contour Grouping Vida Movahedi Supervisor: James Elder Supervisory Committee: Minas Spetsakis, Jeff Edmonds York University Summer 2009

Contents Introduction Preliminary Concepts –Pre-processing –Gestalt cues Methods –Local & Heuristic –Local & Probabilistic –Global Saliency Evaluation Conclusion & open problems

Contents Introduction Preliminary Concepts –Pre-processing –Gestalt cues Methods –Local & Heuristic –Local & Probabilistic –Global Saliency Evaluation Conclusion & open problems

Introduction Segmentation Partition an image into regions, each corresponding to an object or entity Figure-Ground segmentation

Segmentation Methods Regional Segmentation –Use regional info, optimize labelling of regional tokens, e.g. clustering –Depending on uniformity in object region Active Contour Models –Use regional (external) & boundary (internal) info, optimize deformation of model –Sensitivity to initialization, too smooth Contour Grouping –Use boundary info (& regional info), optimize grouping of contour fragments

Problem Definition Input: Color image Goal:Figure-ground segmentation Method: Contour Grouping Other available info: None - No motion, stereo or video information - No user interactions - No assumptions on object types, shapes, color, etc. - No assumptions on background or lighting conditions

Challenges High-dimensional data space, lots of information, many cues Unknown cue integration Global optimization in a non-convex multidimensional space Camera, imaging, quantization noise Clutter in natural scenes Occluded or overlapping objects

Contents Introduction Preliminary Concepts –Pre-processing –Gestalt cues Methods –Local & Heuristic –Local & Probabilistic –Global Saliency Evaluation Conclusion & open problems

Steps Image Grouping Algorithm Saliency Computations Optimization Algorithm Figure/Ground Segmentation Pre-processing Edge Detection Line /Curve Approximation Learned Parameters or Distributions

Pre-processing Image  Edge Map  Line Map  Contour

Gestalt Cues How is grouping done in human vision? Proximity Similarity –Brightness –Contrast Good continuation –Parallelism –Co-circularity

Contents Introduction Preliminary Concepts –Pre-processing –Gestalt cues Methods –Local & Heuristic –Local & Probabilistic –Global Saliency Evaluation Conclusion & open problems

Grouping Methods Local Heuristic methods –Defining a heuristic cost for contour hypotheses, find the optimal one Local Probabilistic methods –Find posterior probability of contour hypotheses given cues, find the optimal one Global methods –An extra step of calculating global saliencies based on local measures

Contents Introduction Preliminary Concepts –Pre-processing –Gestalt cues Methods –Local & Heuristic –Local & Probabilistic –Global Saliency Evaluation Conclusion & open problems

Local & Heuristic Example: Ratio Contour Method (Wang et. al, PAMI’05) Detected/ virtual fragments Contour cost= curvature & gap per unit length Graph model Alternate cycle

Local & Heuristic Example: Ratio Contour Method (Wang et. al, PAMI’05) Edge/ Link costs Ratio Contour Algorithm

Sample Results for RC method ImageRC RRC RC Image (from Stahl & Wang, TIP’07) (from Wang et al., PAMI’05)

Contents Introduction Preliminary Concepts –Pre-processing –Gestalt cues Methods –Local & Heuristic –Local & Probabilistic –Global Saliency Evaluation Conclusion & open problems

Local & Probabilistic (Elder et al., PAMI’03) Bayesian Rule: Contour saliency= posterior probability of contour Assumptions: –Markov Chain Assumption –Independence of evidence from cues –Comparing contours of same length

Local & Probabilistic (Elder et al., PAMI’03) Graph Model Node weight Link weight Shortest path/cycle Approximate search

Sample Results for Probabilistic Methods (from Estrada & Elder- CVPRW’06)

Contents Introduction Preliminary Concepts –Pre-processing –Gestalt cues Methods –Local & Heuristic –Local & Probabilistic –Global Saliency Evaluation Conclusion & open problems

Global Model Local weights Global weights

Global Saliency Edge/Link Affinity Based on collinearity, proximity, etc. Edge/ Link Saliency Relative number of closed random walks which visit that edge/link (Mahamud et al., PAMI’03) Shown to be relevant to the eigenvalues and eigenvectors of the affinity matrix Grouping based on global saliency

Some Results of the Untangling method (from Zhu; Song; Shi- ICCV’07)

Contents Introduction Preliminary Concepts –Pre-processing –Gestalt cues Methods –Local & Heuristic –Local & Probabilistic –Global Saliency Evaluation Conclusion & open problems

Evaluation Empirical discrepancy methods The output of algorithms is compared with a reference segmentation or ground truth Requirements – A ground truth dataset – An error measure

SOD: Salient Object Dataset Based on Berkeley Segmentation Dataset (BSD) 300 images, randomly showing 818 segmentations (half of BSD) to each of 7 subjects 12,110 object boundaries obtained

Region-based Error Measures Example Not sensitive to some large shape features (e.g., spikes, wiggles)

Boundary-based Error Measures Not sensitive to object topology and some large shape features (e.g., loop-backs, wiggles)

Mixed Error Measures Example Not sensitive to some large shape features. Does not respect ordering along contours. p j, j=1..N fn are pixels in the false negative region (R B -R A ) q k, k=1..N fp are pixels in the false positive region (R A -R B )

Contour Mapping Measure Based upon a matching between all points on the two boundaries Monotonically non-decreasing Allowing one-to-one, many-to- one, and one-to-many matching Error= average distance between matched pairs Dynamic Programming Contour Mapping Distance=7.73

Contents Introduction Preliminary Concepts –Pre-processing –Gestalt cues Methods –Local & Heuristic –Local & Probabilistic –Global Saliency Evaluation Conclusion & open problems

Conclusion & Open Problems Cue selection and combination Grouping Model –Global saliency –Probabilistic models Optimization Algorithms Hierarchical and multi-scale algorithms Quantitative evaluation