1. Segmentation and Grouping
Obtain a compact representation from an image/motion sequence/set of tokens
Should support application
Broad theory is absent at present
Grouping (or clustering)
collect together tokens that “belong together”
Fitting
associate a model with tokens
issues
which model?
which token goes to which element?
how many elements in the model?
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

2. Computer Vision - A Modern Approach
General ideas
tokens
whatever we need to group (pixels, points, surface elements, etc., etc.)
top down segmentation
tokens belong together because they lie on the same object
bottom up segmentation
tokens belong together because they are locally coherent
These two are not mutually exclusive
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

3. Computer Vision - A Modern Approach
A disturbing possibility is that they all lie on a sphere -- but then, if we didn’t know that the tokens belonged together,
where did the sphere come from?
Why do these tokens belong together?
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

4. Computer Vision - A Modern Approach
A driving force behind the gestalt movement is the observation that it isn’t enough to think about pictures in terms of separating figure and ground (e.g. foreground and background). This is (partially) because there are too many different
possibilities in pictures like this one. Is a square with a hole in it the figure? or a white circle? or what?
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

5. Basic ideas of grouping in humans
Figure-ground discrimination
grouping can be seen in terms of allocating some elements to a figure, some to ground
impoverished theory
Gestalt properties
elements in a collection of elements can have properties that result from relationships (Muller-Lyer effect)
gestaltqualitat
A series of factors affect whether elements should be grouped together
Gestalt factors
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

6. Computer Vision - A Modern Approach
The famous Muller-Lyer illusion; the point is that the horizontal bar has properties that come only from its membership in
a group (it looks shorter in the lower picture, but is actually the same size) and that these properties can’t be discounted--- you can’t look at this figure and ignore the arrowheads and thereby make the two bars seem to be the same size.
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

7. Computer Vision - A Modern Approach
Some criteria that tend to cause tokens to be grouped.
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

8. Computer Vision - A Modern Approach
More such
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

9. Computer Vision - A Modern Approach
Occlusion cues seem to be very important in grouping. Most people find it hard to read the 5 numerals in
this picture
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

10. Computer Vision - A Modern Approach
but easy in this
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

11. Computer Vision - A Modern Approach
The story is in the book (figure 14.7)
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

12. Computer Vision - A Modern Approach
Illusory contours; a curious phenomenon where you see an object that appears to be occluding.
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

13. Technique: Shot Boundary Detection
Find the shots in a sequence of video
shot boundaries usually result in big differences between succeeding frames
Strategy:
compute interframe distances
declare a boundary where these are big
Possible distances
frame differences
histogram differences
block comparisons
edge differences
Applications:
representation for movies, or video sequences
find shot boundaries
obtain “most representative” frame
supports search
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

14. Technique: Background Subtraction
If we know what the background looks like, it is easy to identify “interesting bits”
Applications
Person in an office
Tracking cars on a road
surveillance
Approach:
use a moving average to estimate background image
subtract from current frame
large absolute values are interesting pixels
trick: use morphological operations to clean up pixels
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

15. Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

16. Computer Vision - A Modern Approach
See caption, figure 14.10
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

17. Computer Vision - A Modern Approach
See caption, figure 14.11
An important point here is that background subtraction works quite badly in the presence of high spatial frequencies, because when we
Estimate the background, we’re almost always going to smooth it.
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

18. Segmentation as clustering
Cluster together (pixels, tokens, etc.) that belong together
Agglomerative clustering
attach closest to cluster it is closest to
repeat
Divisive clustering
split cluster along best boundary
Point-Cluster distance
single-link clustering
complete-link clustering
group-average clustering
Dendrograms
yield a picture of output as clustering process continues
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

19. Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

20. Computer Vision - A Modern Approach
K-Means
Choose a fixed number of clusters
Choose cluster centers and point-cluster allocations to minimize error
can’t do this by search, because there are too many possible allocations.
Algorithm
fix cluster centers; allocate points to closest cluster
fix allocation; compute best cluster centers
x could be any set of features for which we can compute a distance (careful about scaling)
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

21. Computer Vision - A Modern Approach
Image
Clusters on intensity
Clusters on color
I gave each pixel the mean intensity or mean color of its cluster --- this is basically just vector quantizing the image intensities/colors. Notice that there is no requirement that clusters be spatially localized and they’re not.
K-means clustering using intensity alone and color alone
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

22. Computer Vision - A Modern Approach
Image
Clusters on color
K-means using color alone, 11 segments
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

23. Computer Vision - A Modern Approach
K-means using
color alone,
11 segments.
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

24. Computer Vision - A Modern Approach
K-means using colour and
position, 20 segments
Here I’ve represented each pixel as (r, g, b, x, y), which means that segments prefer to be spatially coherent. THese are just some of 20 segments.
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

25. Graph theoretic clustering
Represent tokens using a weighted graph.
affinity matrix
Cut up this graph to get subgraphs with strong interior links
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

26. Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

27. Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

28. Computer Vision - A Modern Approach
Measuring Affinity
Intensity
Distance
here c(x) is a vector of filter outputs. A natural thing to do is to square the outputs of a range of different filters at different scales and orientations, smooth the result, and rack these into a vector.
Texture
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

29. Scale affects affinity
This is figure 14.18
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

30. Computer Vision - A Modern Approach
Eigenvectors and cuts
Simplest idea: we want a vector a giving the association between each element and a cluster
We want elements within this cluster to, on the whole, have strong affinity with one another
We could maximize
But need the constraint
This is an eigenvalue problem - choose the eigenvector of A with largest eigenvalue
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

31. Computer Vision - A Modern Approach
Example eigenvector
points
eigenvector
matrix
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

32. Computer Vision - A Modern Approach
More than two segments
Two options
Recursively split each side to get a tree, continuing till the eigenvalues are too small
Use the other eigenvectors
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

33. Computer Vision - A Modern Approach
Normalized cuts
Current criterion evaluates within cluster similarity, but not across cluster difference
Instead, we’d like to maximize the within cluster similarity compared to the across cluster difference
Write graph as V, one cluster as A and the other as B
Maximize
i.e. construct A, B such that their within cluster similarity is high compared to their association with the rest of the graph
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

34. Computer Vision - A Modern Approach
Normalized cuts
Write a vector y whose elements are 1 if item is in A, -b if it’s in B
Write the matrix of the graph as W, and the matrix which has the row sums of W on its diagonal as D, 1 is the vector with all ones.
Criterion becomes
and we have a constraint
This is hard to do, because y’s values are quantized
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

35. Computer Vision - A Modern Approach
Normalized cuts
Instead, solve the generalized eigenvalue problem
which gives
Now look for a quantization threshold that maximises the criterion --- i.e all components of y above that threshold go to one, all below go to -b
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

36. Computer Vision - A Modern Approach
This is figure caption there explains it all.
Figure from “Image and video segmentation: the normalised cut framework”,
by Shi and Malik, copyright IEEE, 1998
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth

37. Computer Vision - A Modern Approach
F igure from “Normalized cuts and image segmentation,” Shi and Malik, copyright IEEE, 2000
This is figure 14.24, whose caption gives the story
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth