Fast, Multiscale Image Segmentation: From Pixels to Semantics Ronen Basri The Weizmann Institute of Science Joint work with Achi Brandt, Meirav Galun, Eitan Sharon

Camouflage

Camouflage Malik et al.’s “Normalized cuts”

Our Results

Segmentation by Weighted Aggregation A multiscale algorithm: Optimizes a global measure Returns a full hierarchy of segments Linear complexity Combines multiscale measurements: –Texture –Boundary integrity

The Pixel Graph Couplings (weights) reflect intensity similarity Low contrast – strong coupling High contrast – weak coupling

Normalized-cut Measure Minimize:

Saliency Measure Minimize:

Multiscale Computation of Ncuts Our objective is to rapidly find the segments (0-1 partitions) that optimize  For single-node cuts we simply evaluate  For multiple-node cuts we perform “soft contraction” using coarsening procedures from algebraic multigrid solvers of PDEs.

Coarsening the Graph Suppose we can define a sparse mapping such that for all minimal states

Coarse Energy Then P T WP, P T LP define a new (smaller) graph

Recursive Coarsening

Representative subset

Recursive Coarsening For a salient segment :, sparse interpolation matrix

Weighted Aggregation aggregate

Hierarchical Graph  Pyramid of graphs  Soft relations between levels  Segments emerge as salient nodes at some level of the pyramid

Importance of Soft Relations

Physical Motivation Our algorithm is motivated by algebraic multigrid solutions to heat or electric networks u - temperature/potential a(x, y) – conductivity At steady state largest temperature differences are along the cuts AMG coarsening is independent of f

Determine the Boundaries ,0,0,…,0 P

Hierarchy in SWA

Texture Examples

Filters (From Malik and Perona) Oriented filters Center- surround

A Chicken and Egg Problem Problem: Coarse measurements mix neighboring statistics Solution: Support of measurements is determined as the segmentation process proceeds Hey, I was here first

Texture Aggregation Aggregates assumed to capture texture elements Compare neighboring aggregates according to the following statistics: –Multiscale brightness measures –Multiscale shape measures –Filter responses Use statistics to modify couplings

Recursive Computation of Measures Given some measure of aggregates at a certain level (e.g., orientation) At every coarser level we take a weighted sum of this measure from previous level The result can be used to compute the average, variance or histogram of the measure Complexity is linear

Use Averages to Modify the Graph

Adaptive vs. Rigid Measurements Averaging Our algorithm - SWA Original Geometric

Adaptive vs. Rigid Measurements Interpolation Geometric Original Our algorithm - SWA

Adaptive vs. Rigid Measurements

Texture Aggregation Fine (homogeneous) Coarse (heterogeneous)

Multiscale Variance Vector

Variance: Avoid Mixing aggregationSliding window

Leopard

More Leopards…

And More…

Birds

More Animals

Boat

Malik’s Ncuts

Key Differences Optimize a global measure (like Malik’s Ncuts) Hierarchy with soft relations (unlike agglomerative/graph contraction) Combine texture measurements while avoiding the “chicken and egg problem”

Complexity Every level contains about half the nodes of the previous level: Total #nodes 2 X #pixels All connections are local, cleaning small weights Top-down sharpening: constant number of levels Linear complexity Implementation: 5 seconds for 400x400

Average intensity Texture Shape Representation

Relevance to Biology? Layers of retinotopic maps Feed-forward (fine-to-coarse) progression Feedback refinement Edges determine both boundaries and textures Segmentation determined by combinations of cues Coarse nodes represent abstract shapes and properties

Matching (with Chen Brestel)

More…

Data: Filippi 30 slices, 180x220 in 3 minutes MRI Data

MS Lesion Detection Tagged Our results Data: Filippi

Tagged Our results

Data: Filippi Tagged Our results

Data: Filippi Tagged Our results

2D Segmentation Data: Filippi

3D Segmentation

Cell Movement

Summary image segments Shape properties Leopard