Segmentation Slides Credit: Jim Rehg, G.Tech. Christopher Rasmussen, UD John Spletzer, Lehigh Also, Slides adopted from material provided by David Forsyth and Trevor Darrell

Segmentation and Grouping Obtain a compact representation from an image/motion sequence/set of tokens Should support application Broad theory is absent at present Grouping (or clustering) –collect together tokens that “belong together” Fitting –associate a model with tokens –issues which model? which token goes to which element? how many elements in the model?

What is Segmentation? Clustering image elements that “belong together” –Partitioning Divide into regions/sequences with coherent internal properties (k-means) –Grouping Identify sets of coherent tokens in image (model fit, Hough) Tokens: Whatever we need to group –Pixels –Features (corners, lines, etc.) –Larger regions (e.g., arms, legs, torso) –Discrete objects (e.g., people in a crowd) –Etc.

General ideas tokens –whatever we need to group (pixels, points, surface elements, etc., etc.) top down segmentation –tokens belong together because they lie on the same object bottom up segmentation –tokens belong together because they are locally coherent These two are not mutually exclusive

Why do these tokens belong together?

Basic ideas of grouping in humans Gestalt properties –elements in a collection of elements can have properties that result from relationships (Muller-Lyer effect) gestaltqualitat –A series of factors affect whether elements should be grouped together Gestalt factors

Gestalt Theory of Grouping Psychological basis for why/how things are grouped bottom-up and top- down Figure-ground discrimination –Grouping can be seen in terms of allocating tokens to figure or ground Factors affecting token coherence –Proximity –Similarity: Based on color, texture, orientation (aka parallelism), etc. –Common fate: Parallel motion –Common region: Tokens that lie inside the same closed region tend to be grouped together. –Closure: Tokens or curves that tend to lead to closed curves tend to be grouped together. –Symmetry: Curves that lead to symmetric groups are grouped together –Continuity: Tokens that lead to “continuous” — as in “joining up nicely,” rather than in the formal sense — curves tend to be grouped –Familiar Configuration: Tokens that, when grouped, lead to a familiar object—e.g., the top-down recognition that allows us to see the dalmation from Forsyth & Ponce

Technique: Background Subtraction If we know what the background looks like, it is easy to identify “interesting bits” Applications –Person in an office –Tracking cars on a road –surveillance Approach: –use a moving average to estimate background image –subtract from current frame –large absolute values are interesting pixels trick: use morphological operations to clean up pixels

Video Example

Subtraction and Thresholding

Finding Subparts

Pfinder Example

Outline Clustering basics –k -means clustering (partitioning) Hough transform (grouping)

Basic Approaches to Clustering Unknown number of clusters –Agglomerative clustering Start with as many clusters as tokens and selectively merge –Divisive clustering Start with one cluster for all tokens and selectively split Known number of clusters –Selectively change cluster memberships of tokens Merging/splitting/rearranging stops when threshold on token similarity is reached –Within cluster: As similar as possible –Between clusters: As dissimilar as possible

Feature Space Every token is identified by a set of salient visual characteristics called features (akin to gestalt grouping factors). For example: –Position –Color –Texture –Motion vector –Size, orientation (if token is larger than a pixel) The choice of features and how they are quantified implies a feature space in which each token is represented by a point Token similarity is thus measured by distance between points (aka “feature vectors”) in feature space

k -means Clustering Initialization: Given k categories, N points in feature space. Pick k points randomly; these are initial cluster centers (means) ¹ 1, …, ¹ k. Repeat the following: 1.Assign all N points to clusters by nearest ¹ i (make sure no cluster is empty) 2.Recompute mean ¹ i of each cluster from C i member points 3.If no mean has changed more than some ¢, stop Effectively carries out gradient descent on

Example: 3-means Clustering from Duda et al. Convergence in 3 steps The mean changes, and so are the clusters.

Example: k -means Clustering 4 of 11 clusters using color alone from Forsyth & Ponce

Example: k -means Clustering 4 of 20 clusters using color and position from Forsyth & Ponce

Hough Transform (HT) Basic idea: Change problem from complicated pattern detection to peak finding in parameter space of the shape –Each pixel can lie on a family of possible shapes (e.g., for lines, the set of lines through that point) –Shapes with more pixels on them have more evidence that they are present in the image –Thus every pixel “votes” for a set of shapes and the one(s) with the most votes “win”—i.e., exist courtesy of Massey U.

The Hough Transform The general idea: –A line in the image can be parameterized by 2 variables –Each edge pixel (x,y) corresponds to a family of lines L(x,y) = {l 1,…,l n } –Pixel (x,y) votes for each l i L(x,y) –Edge pixels that form a line will each place one vote for the same (a i,b i ) – along with lots of other lines –Lines that are in the image will receive more votes than ones that are not

The Hough Transform “Each edge pixel (x,y) corresponds to a family of lines L(x,y) = {l 1,…,l n }”

The Hough Transform Pixel (x,y) votes for each l i L(x,y)

The Hough Transform Edge pixels that form a line will each place one vote for the same (a i,b i ) – along with lots of other lines

The Hough Transform Lines that are in the image will receive more votes than ones that are not

The Hough Transform The line feature detection problem is transformed into a peak detection algorithm! We need only find the lines with the most votes, and these correspond to the greatest Issue: Line representation How do we discretize a and b?

Solution: Polar Representation Instead of using the slope-intercept representation, we can use a polar representation where ρ corresponds to the normal distance to the line and θ the polar angle These parameters are bounded by which we can discretize to an appropriate resolution

HT for Line Finding Fixing an image pixel (x i, y i ) yields a set of points in line space f(r, µ)g corresponding to a sinusoidal curve described by Each point on curve in line space is a member of the set of lines through the pixel Collinear points yield curves that intersect in a single point courtesy of R. Bock

The Algorithm 1.Take as input an edge image E(i,j) 2.Define a resolution dθ and dρ for θ and ρ, respectively 3.Construct an accumulator array A(m,n)=0 where m=(1/dρ)*ρ max and n=2π/dθ

The Algorithm 4. For each pixel E(i,j)==255 do For θ = dθ:dθ:2*pi 1.ρ = i*cos(θ) + j*sin(θ) 2.if ρ<0, continue 3.Round ρ to the nearest dρ value 4.A(i’,j’)++ 5. Threshold A(i’,j’) to find all relevant lines

Example: HT for Line Finding Edge-detected image Accumulator array “De-Hough” of lines ¸ 70% of max courtesy of Massey U.

(Credit: Lehigh) Raw ImageEdge Image Segmented Lanes

pixels votes θ ρ x y

pixelsvotes θ ρ x y

pixelsvotes θ ρ x y

Mechanics of the Hough transform How big should the cells be? –If too big, we can’t distinguish between different lines; if too small, noise causes lines to be missed) How many lines? –Count the peaks in the array (thresholding) Who belongs to which line? –Tag the votes, or post-process

Hough Transform: Issues Noise –Points slightly off curve result in multiple intersections Can use larger bins, smooth accumulator array Non-maximum suppression a good idea to get unique peaks Dimensionality –Exponential increase in size of accumulator array as number of shape parameters goes up –HT works best for shapes with 3 or fewer variables (circles, ellipses, etc.) Since the HT is a voting algorithm, it is robust to noise and can detect broken, disconnected, or partially occluded line segments