1. Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...

2. Computer Vision Aug 26/28-Introduction Sep 2/4CamerasRadiometry Sep 9/11Sources & ShadowsColor Sep 16/18Linear filters & edges(hurricane Isabel) Sep 23/25Pyramids & TextureMulti-View Geometry Sep30/Oct2StereoProject proposals Oct 7/9Tracking (Welch)Optical flow Oct 14/16-- Oct 21/23Silhouettes/carving(Fall break) Oct 28/30-Structure from motion Nov 4/6Project updateProj. SfM Nov 11/13Camera calibrationSegmentation Nov 18/20FittingProb. segm.&fit. Nov 25/27Matching templates(Thanksgiving) Dec 2/4Matching relationsRange data Dec ?Final project Tentative class schedule

3. Computer Vision Segmentation and Grouping Motivation: not information is evidence Obtain a compact representation from an image/motion sequence/set of tokens Should support application Broad theory is absent at present Grouping (or clustering) –collect together tokens that “belong together” Fitting –associate a model with tokens –issues which model? which token goes to which element? how many elements in the model?

4. Computer Vision General ideas tokens –whatever we need to group (pixels, points, surface elements, etc., etc.) top down segmentation –tokens belong together because they lie on the same object bottom up segmentation –tokens belong together because they are locally coherent These two are not mutually exclusive

5. Computer Vision Why do these tokens belong together?

6. Computer Vision

7. Computer Vision Basic ideas of grouping in humans Figure-ground discrimination –grouping can be seen in terms of allocating some elements to a figure, some to ground –impoverished theory Gestalt properties –elements in a collection of elements can have properties that result from relationships (Muller-Lyer effect) gestaltqualitat –A series of factors affect whether elements should be grouped together Gestalt factors

8. Computer Vision

9. Computer Vision

10. Computer Vision

11. Computer Vision

12. Computer Vision

13. Computer Vision

14. Computer Vision

15. Computer Vision Technique: Shot Boundary Detection Find the shots in a sequence of video –shot boundaries usually result in big differences between succeeding frames Strategy: –compute interframe distances –declare a boundary where these are big Possible distances –frame differences –histogram differences –block comparisons –edge differences Applications: –representation for movies, or video sequences find shot boundaries obtain “most representative” frame –supports search

16. Computer Vision Technique: Background Subtraction If we know what the background looks like, it is easy to identify “interesting bits” Applications –Person in an office –Tracking cars on a road –surveillance Approach: –use a moving average to estimate background image –subtract from current frame –large absolute values are interesting pixels trick: use morphological operations to clean up pixels

17. Computer Vision

18. Computer Vision

19. Computer Vision

20. Computer Vision

21. Computer Vision

22. Computer Vision

23. Computer Vision

24. Computer Vision

25. Computer Vision

26. Computer Vision

27. Computer Vision Segmentation as clustering Cluster together (pixels, tokens, etc.) that belong together Agglomerative clustering –attach closest to cluster it is closest to –repeat Divisive clustering –split cluster along best boundary –repeat Point-Cluster distance –single-link clustering –complete-link clustering –group-average clustering Dendrograms –yield a picture of output as clustering process continues

28. Computer Vision Simple clustering algorithms

29. Computer Vision

30. Computer Vision K-Means Choose a fixed number of clusters Choose cluster centers and point-cluster allocations to minimize error can’t do this by search, because there are too many possible allocations.

31. Computer Vision K-means clustering using intensity alone and color alone Image Clusters on intensity Clusters on color

32. Computer Vision K-means using color alone, 11 segments Image Clusters on color

33. Computer Vision K-means using color alone, 11 segments.

34. Computer Vision K-means using colour and position, 20 segments

35. Computer Vision Graph theoretic clustering Represent tokens using a weighted graph. –affinity matrix Cut up this graph to get subgraphs with strong interior links

36. Computer Vision

37. Computer Vision

38. Computer Vision

39. Computer Vision

40. Computer Vision

41. Computer Vision Measuring Affinity Intensity Texture Distance

42. Computer Vision Scale affects affinity

43. Computer Vision Eigenvectors and cuts Simplest idea: we want a vector a giving the association between each element and a cluster We want elements within this cluster to, on the whole, have strong affinity with one another We could maximize But need the constraint This is an eigenvalue problem - choose the eigenvector of A with largest eigenvalue

44. Computer Vision Example eigenvector points matrix eigenvector

45. Computer Vision

46. Computer Vision More than two segments Two options –Recursively split each side to get a tree, continuing till the eigenvalues are too small –Use the other eigenvectors

47. Computer Vision More than two segments

48. Computer Vision Normalized cuts Current criterion evaluates within cluster similarity, but not across cluster difference Instead, we’d like to maximize the within cluster similarity compared to the across cluster difference Write graph as V, one cluster as A and the other as B Maximize i.e. construct A, B such that their within cluster similarity is high compared to their association with the rest of the graph

49. Computer Vision

50. Computer Vision

51. Computer Vision Normalized cuts Write a vector y whose elements are 1 if item is in A, -b if it’s in B Write the matrix of the graph as W, and the matrix which has the row sums of W on its diagonal as D, 1 is the vector with all ones. Criterion becomes and we have a constraint This is hard to do, because y’s values are quantized

52. Computer Vision

53. Computer Vision Normalized cuts Instead, solve the generalized eigenvalue problem which gives Now look for a quantization threshold that maximises the criterion --- i.e all components of y above that threshold go to one, all below go to -b

54. Computer Vision Figure from “Image and video segmentation: the normalised cut framework”, by Shi and Malik, copyright IEEE, 1998

55. Computer Vision Figure from “Normalized cuts and image segmentation,” Shi and Malik, copyright IEEE, 2000

56. Computer Vision Image parsing: Unifying Segmentation, Detection and Recognition Tu, Chen, Yuille & Zhu ICCV’03 (Marr prize!)

57. Computer Vision Image parsing Generative models for each class some random samples for 5 and H some random face samples (PCA model)

58. Computer Vision DDMCMC Data Driven Markov Chain Monte Carlo

59. Computer Vision Image parsing

60. Computer Vision Image parsing

61. Computer Vision Next class: Fitting Reading: Chapter 15