1. Recognition of 3D Objects or, 3D Recognition of Objects Alec Rivers

2. Overview 3D object recognition was dead, now it’s coming back – These papers are within the last 2 years Doesn’t really work yet, but it’s just a beginning

3. Papers The Layout Consistent Random Field for Recognizing and Segmenting Partially Occluded Objects – CVPR 2006 3D LayoutCRF for Multi-View Object Class Recognition and Segmentation – CVPR 2007 3D Generic Object Categorization, Localization and Pose Estimation – ICCV 2007

4. The Layout Consistent Random Field for Recognizing and Segmenting Partially Occluded Objects John Winn Microsoft Research Cambridge Jamie Shotton University of Cambridge

5. Introduction Needed to understand next paper – It’s 2D What does it try to solve? – Recognize one class of object at one pose and one scale, but with occlusions Does it work? – Yes, really well, especially given occlusions

6. Introduction What is interesting about it? – Segments objects – Interesting methods No sliding windows – Multiple instances for free

7. Overview Instead of sparse parts at features, use a densely covering part grid [Fischler & Elschlager 73] [Winn & Shotton 06]

8. Recognizing New Image – Overview Walk through an example

9. Recognizing a New Image – Overview 1. Pixels guess their part

10. Recognizing a New Image – Overview 2. Maximize layout consistency

11. Layout Consistency Defined pairwise between two pixels: P I, P J => Bool Means pixels I, J could be part of one instance Toy example: Object: 1,2,3,4,5 Image: 2,3,4,5,0,0,1,2,3,4,5,2,3,4,5,0,0

12. Layout Consistency Defined pairwise between two pixels: P I, P J => Bool Means pixels I, J could be part of one instance Toy example: Object: 1,2,3,4,5 Image: 2,3,4,5,0,0,1,2,3,4,5,2,3,4,5,0,0 occlusion instance 2instance 3instance 1

13. Layout Consistency In 2D, consistent IFF their relative assignments could exist in a deformed regular grid Formally:

14. Overview 2. Maximize layout consistency

15. Layout Consistency 3. Find consistent regions; create instances Possible due to layout inconsistency at occluding borders

16. Overview 1. Pixels guess parts 2. Maximize layout consistency 3. Create instances [Winn & Shotton 06]

17. Implementation Details Trained on manually segmented data Crux of algorithm is conditional distribution – Like a probability for each possibility, or a score Algorithm is just finding maximum

18. Part Appearance Each pixel prefers parts that match surrounding image data Randomized decision trees – Multiple trees, each trained on a subset of the data – Node is maximal-information-gain binary test on two nearby pixels’ intensities – Leaf of node is histogram of part possibilities – Actual preference is average over all trees

19. Deformed Training Part Labelings Fits parts tighter 1. Label by grid 2. Learn from data 3. Apply to data 4. Set guesses as truth 5. Relearn

20. Part Layout Preference for layout consistency plus additional pairwise costs: Helps remove noise Align edges along image edges

21. Part Layout Return to toy example Just appearance: 1,2,0,4,5,0,0,1,2,3,3,4,0,0,1,0 With layout costs: 1,2,3,4,5,0,0,1,2,3,3,4,0,0,0,0 instance 2instance 1

22. Instance Layout Apply weak force trying to keep parts at sane positions relative to instance data (centroid, L/R flip) Toy example: 0,1,1,1,1,1,2,3,4,5 is bad!

23. Implementation Theoretically, finding global maximum of This is “MAP” estimation – MAP = Maximum A Posteriori In reality, using tricks to find a local maximum – α-expansion, annealed expansion move

24. Approximating MAP Estimation Global maximum is intractable α-expansion – Start with given configuration – For a given new label, ask each pixel: do you want to switch? – Can be solved efficiently with graph cuts Repeat over all part labels Annealed expansion move – Relabel grid, but offset to avoid local maxima

25. Results

27. Oh, snap!

28. Thoughts Bottom-up system is great – No sliding windows – Multiple instances for free Information about segment boundaries: occlusion vs. completion – Reason about complete segment boundaries?

29. John Winn 3D LayoutCRF for Multi-View Object Class Recognition and Segmentation Derek Hoiem Carnegie Mellon University Carsten Rother Microsoft Research Cambridge

30. Introduction What does it try to solve? – Extend LayoutCRF to be pose and scale invariant Does it work? – Improvements to LayoutCRF work; 3D information does little What is interesting about it? – One method for combining 2D methods with a 3D framework – The improvements to 2D are good

31. Overview Generate rough 3D model of class Parts created over 3D model

32. Overview Probability distribution

33. Refinements Part layout, instance layout take into account 3D position

34. Refinements New term: Instance cost

35. Instance Cost Eliminates false positives – LayoutCRF: object-background cost Explain multiple groups with one instance

36. Refinements New term: Instance appearance

37. Instance appearance Learn color distribution for each instance Separate groups of pixels: definitely object, definitely background Use these to learn colors Apply cost to non-standard-color pixels This would fail…

38. Implementation Details Parts are learned separately for each 45 o viewing range, and for different scales Instance layout is also discretized by viewpoint

39. Results – Comparison to LCRF A little better (+ 8% recall) BUT they actually turn off 3D information for this comparison Better segmentation

40. Results – PASCAL 2006 61% precision-recall – Previous best: 45% – But, reduced test set Without 3D: -5% Without color: -5%

41. Thoughts Color, instance costs very nice Shoehorns LCRF into 3D without much success LCRF is already somewhat viewpoint-invariant: segments can stretch

42. 3D Generic Object Categorization, Localization and Pose Estimation Silvio Savarese University of Illinois at Urbana-Champaign Fei-Fei Li Princeton University

43. Introduction What does it try to solve? – Multiclass pose-invariant, scale-invariant object recognition Does it work? – Not well. But it may be due to implementation Why is it interesting? – Attempt learn actual 3D structure of an object – Interesting data structure for 3D info

44. Overview – Data Structure Decompose object into large parts; find “canonical view” Relate parts by mutual appearance

45. Related Work – Aspect Graphs Represent stable views rather than parts Image [Khoh & Kovesi, 99] Aspect graph of a cube:

46. Data Structure for Cube Left Front Right Bottom Top Back

47. Related Work Constellation models Similar, but wraps around in 3D vs.

48. Implementation – Links Link from canonical P I to P J consists of Matrix defines transformation to observe P J when P I is viewed canonically A IJ is skew, t IJ is translation

49. Implementation – Links H IJ Part J canonical view Part I canonical view

50. Implementation – Links Part J canonical view H JI Part I canonical view

51. Overview Learn data structure from images (unsupervised) Apply to new image by recognizing parts and selecting model that best accounts for their appearances

52. Implementation – Learning Parts Tricky implementation! Part = collection of SIFT features For each pair of images of the same instance: 1. Find set M of shared SIFT features 2. RANSAC M to find a group of pairs that transform together 3. Group close-together parts of M into candidate parts

53. Background: What is RANSAC? Finds subset of data that is accounted for by some model; ignores outliers 1. Guess points 2. Fit model 3. Select matching points 4. Calculate error Repeat!

54. RANSAC In our case: find points for which a homographic transformation of the points in image I yield the points in image J

55. Implementation – Canonical Views Goal: front-facing view of part Construct directed graph – Direction means “more front-facing” Traverse to find canonical view How to go from pairwise-defined to graph?

56. Implementation Upshot: a collection of parts with canonical views and links

57. Recognizing a New Image 1. Extract SIFT features 2. Use scanning windows to get 5 best canonical part matches 3. For every pair of found parts, for each model, score how well the model accounts for their relative appearances 4. Select the model with the best score

58. Results Not stellar New test set – Overfit? – Comparison?

59. Results

60. Thoughts Low performance may make it useless as a system, but the data structure is very nice Implementation has a lot of tricky parts – Doesn’t seem to select great canonical parts – I wonder if there’s a simpler way – Are SIFT features the right choice?

61. Extremely Confusing Figure “Each dashed box indicates a particular view. A subset of the canonical parts is presented for each view. Part relationships are denoted by arrows.”

62. Overall Conclusions 3D is just starting out. Doesn’t work too well right now, but neither did MV at the beginning. LayoutCRF: – Nice method to learn 2D patches 3D Object Categorization: – Nice conceptual model relating 3D parts Possible to combine strengths of both?