1. 3D Shape Representation Tianqiang 04/01/2014

2. Image/video understanding Content creation Why do we need 3D shapes?

3. Image/video understanding [Chen et al. 2010] Why do we need 3D shapes?

4. Image/video understanding [Xiang et al. 2014 (ECCV)] [Chen et al. 2010] Why do we need 3D shapes?

5. Image/video understanding Need a 3D shape prior! Why do we need 3D shapes?

6. Image/video understanding Content creation [Yumer et al. 2010][Kalogerakis et al. 2012] Why do we need 3D shapes?

7. Content creation Why do we need 3D shapes? Need a 3D shape prior, again!

8. Papers to be covered in this lecture Recovering 3D from single images 3D Reconstruction from a Single Image [Rother et al. 08] Shared Shape Spaces [Prisacariu et al. 11] A Hypothesize and Bound Algorithm for Simultaneous Object Classification, Pose Estimation and 3D Reconstruction from a Single 2D Image [Rother et al. 11] Beyond PASCAL: A Benchmark for 3D Object Detection in the Wild [Xiang et al. 14 (WACV)] Inferring 3D Shapes and Deformations from Single Views [Chen et al. 10]

9. Papers to be covered in this lecture Tracking Simultaneous Monocular 2D Segmentation, 3D Pose Recovery and 3D Reconstruction [Prisacariu et al. 12] Monocular Multiview Object Tracking with 3D Aspect Parts [Xiang et al. 14 (ECCV)]

10. Papers to be covered in this lecture Shape synthesis A Probabilistic Model for Component-based Shape Synthesis [Kalogerakis et al. 12] Co-constrained Handles for Deformation in Shape Collections [Yumer et al. 14]

11. Recovering 3D from single images Input: a single image

12. Recovering 3D from single images Output: 3D shape, and pose or viewpoint

13. Outline A traditional generative model Shared shape space A benchmark for 3D object detection

14. Outline A traditional generative model Shared shape space A benchmark for 3D object detection

15. A Traditional Model : 3D shape (output) : Viewpoint (output) : Image (input)

16. : 3D shape (output) : Viewpoint (output) : Image (input) Projection likelihood Shape prior Viewpoint prior A Traditional Model

17. Shape prior Viewpoint prior A Traditional Model [Rother et al. 2008] [Xiang et al. 2014] [Chen et al. 2014] Projection likelihood

18. Rother et al. 2008 - Overview Voxel representation for 3D shape Viewpoint is known or enumerated Using an optics law to do projection

19. Rother et al. 2008 - Overview

20. Voxel occupancies Rother et al. 2008 – 3D Shape Prior (2D is just for illustration)

21. Rother et al. 2008 – Projection likelihood (Assuming the viewpoint is known)

22. Rother et al. 2008 – Projection likelihood Two issues: Effect is unrelated to travel distance in each voxel Dependent on the resolution of grids

23. Rother et al. 2008 – Projection likelihood Beer-lambert law:

24. Rother et al. 2008 – Optimization Possible solution: belief propagation But what if the graph is loopy?

25. Rother et al. 2008 – Optimization Possible solution: belief propagation But what if the graph is loopy? Down-sample the image!

26. Rother et al. 2008 – Experiments Learning shape prior: separately for each subclass eg. Right leg in front, left leg up, … Separate Together Degree = 0 60 120 180

27. Rother et al. 2008 – Experiments

28. Chen et al. 2010 - Overview Modeling intrinsic shape (phenotype) and pose variations Mesh representation for 3D shape. Only leveraging silhouettes in images

29. Chen et al. 2010 - Overview Pose Shape Viewpoint Projection Silhouette Latent variables Intrinsic shape

30. Pose Shape Viewpoint Projection Silhouette Intrinsic shape Chen et al. 2010 - Overview Latent variables

31. Gaussian Process Latent Variable Models Chen et al. 2010 – 3D Shape Prior

32. Gaussian Process Latent Variable Model Pose

33. Gaussian Process Latent Variable Model Intrinsic shape (phenotype)

34. Shape transfer

35. : projected silhouettes : samples on 3D mesh : camera parameters : input silhouettes Chen et al. 2010 – Projection

36. ProjectingMatching Chen et al. 2010 – Projection

37. Projecting: : projection : translation vector Chen et al. 2010 – Projection

38. : a similarity function : normalizer Matching: Chen et al. 2010 – Projection

39. Chen et al. 2010 – Optimization Approximation Adaptive-scale line search Multiple initialization

40. Chen et al. 2010 – Experiments

41. Outline A traditional generative model Shared shape space A benchmark for 3D object detection

42. Shared Shape Space Silhouette 3D shape Latent variable

43. Shared Shape Space Multimodality

44. Silhouette 3D shape Latent variable Cannot model multimodality Shared Shape Space

45. Silhouette 3D shape Latent variables Can model multimodality Similarity distribution Shared Shape Space

47. A little more detail: Hierarchy of latent manifolds Representation of 2D and 3D shapes

48. Hierarchy of Manifolds 10D 4D 2D

49. 2D/3D Discrete Cosine Transforms Advantages: Fewer dimensions than directly representing the images.

50. 2D/3D Discrete Cosine Transforms Image: 640X480 = 307200 dims DCT: 35X35 = 1225 dims

51. Shared Shape Space - Inference 10D 4D 2D InputOutput

52. Shared Shape Space - Inference Step 1: find a latent space point in which produces Y that best segments the image

53. Shared Shape Space - Inference Step 2: obtain a similarity function

54. Shared Shape Space - Inference Step 3: pick the peaks in, which are latent points in space

55. Shared Shape Space - Inference Step 4: take these points as initializations, and search for latent points in that best segments the image.

56. Shared Shape Space - Inference Step 5: obtain latent points in the upper layer and continue.

57. Shared Shape Space - Inference Step 5: obtain latent points in the upper layer and continue.

58. Shared Shape Space - Inference Step 5: obtain latent points in the upper layer and continue. Please refer to the paper to see how to find the latent points for best segmenting the image in each step! (Also use GPLVM)

59. Shared Shape Space - Results Tracking

60. Shared Shape Space - Results Shape from single images

61. Shared Shape Space - Results Gaze tracking

62. Outline A traditional generative model Shared shape space A benchmark for 3D object detection

63. A Benchmark: Beyond PASCAL Benchmark for what?

64. A Benchmark: Beyond PASCAL Advantage over previous benchmarks Background clutter Occluded/truncated objects Large number of classes and intra-class variation Large number of viewpoints

65. Advantages Background clutter Occluded/truncated objects Large number of classes and intra-class variation Large number of viewpoints

66. Background clutter Occluded/truncated objects Large number of classes and intra-class variation Large number of viewpoints Advantages

67. Background clutter Occluded/truncated objects Large number of classes and intra-class variation Large number of viewpoints

68. Advantages Background clutter Occluded/truncated objects Large number of classes and intra-class variation Large number of viewpoints

69. Statistics of azimuth distribution A Benchmark: Beyond PASCAL

70. Statistics of occlusion distribution

71. A Benchmark: Beyond PASCAL Statistics of occlusion distribution

72. A Benchmark: Beyond PASCAL

73. Papers to be covered in this lecture Tracking Simultaneous Monocular 2D Segmentation, 3D Pose Recovery and 3D Reconstruction [Prisacariu et al. 12] Monocular Multiview Object Tracking with 3D Aspect Parts [Xiang et al. 14 (ECCV)]

74. Tracking Input: a sequence of frames Output: segmentation, pose, 3D reconstruction per frame

75. Outline PaperShape representation Motion priorShape consistency Prisacariu et al. 12 DCT (same as Shared Shape Space) NoneConstrained to be the same 3D shape Xiang et al. 14Assembly of 3D aspect parts Gaussian priors

76. Outline PaperShape representation Motion priorShape consistency Prisacariu et al. 12 DCT (same as Shared Shape Space) NoneConstrained to be the same 3D shape Xiang et al. 14Assembly of 3D aspect parts Gaussian priors

77. Tracking with 3D Aspect Parts Frames3D aspect parts

78. Tracking with 3D Aspect Parts : positions of aspect parts : viewpoints : image To maximize the following posterior distribution:

79. Tracking with 3D Aspect Parts : positions of aspect parts : viewpoints : image To maximize the following posterior distribution:

80. : position of an aspect part Likelihood term Assumption of independency:

81. Likelihood term Using classifiers in both category and instance level:

82. Likelihood term Descriptor of a rectified aspect part Extracting Haar-like features for instance-level templates HOG

83. Tracking with 3D Aspect Parts : positions of aspect parts : viewpoints : image To maximize the following posterior distribution:

84. Tracking with 3D Aspect Parts : positions of aspect parts : viewpoints : image To maximize the following posterior distribution:

85. Motion prior term

86. Assumption of independency: Motion prior term

87. Pairwise term: Motion prior term

88. MCMC sampling Inference

90. Results - Tracking

91. Results – Viewpoint and 3D recovery

92. Papers to be covered in this lecture Shape synthesis A Probabilistic Model for Component-based Shape Synthesis [Kalogerakis et al. 12] Co-constrained Handles for Deformation in Shape Collections [Yumer et al. 14]

93. Papers to be covered in this lecture Shape synthesis A Probabilistic Model for Component-based Shape Synthesis [Kalogerakis et al. 12] Co-constrained Handles for Deformation in Shape Collections [Yumer et al. 14]

94. Probabilistic Model for Shape Synthesis (from Kalogerakis’ SIGGRAPH slides) Input Probabilistic model Output Learning stage Synthesis stage

95. Learning stage Input Probabilistic model Learning stage

96. R P(R)

97. R

98. R P(R) [ P( N l | R) ] Π l ∈ L NlNl L

99. R SlSl NlNl L P(R) [ P( N l | R) P ( S l | R ) ] Π l ∈ L

100. R SlSl NlNl L P(R) [ P( N l | R) P ( S l | R ) ] Π l ∈ L

101. ClCl R SlSl L P(R) [ P( N l | R) P ( S l | R ) P( D l | S l ) P( C l | S l ) ] Π l ∈ L DlDl NlNl

102. ClCl R SlSl L P(R) [ P( N l | R) P ( S l | R ) P( D l | S l ) P( C l | S l ) ] Π l ∈ L DlDl NlNl Height Width

103. ClCl R SlSl L P(R) [ P( N l | R) P ( S l | R ) P( D l | S l ) P( C l | S l ) ] Π l ∈ L DlDl NlNl Latent object style Latent component style

104. ClCl R SlSl L DlDl NlNl Learn from training data: latent styles lateral edges parameters of CPDs

105. Learning stage Given observed data O, find structure G that maximizes:

106. Learning stage Given observed data O, find structure G that maximizes: Assuming uniform prior over structures, maximize marginal likelihood:

107. Learning stage Given observed data O, find structure G that maximizes: Assuming uniform prior over structures, maximize marginal likelihood: (Cheeseman-Stutz approximation)

108. Synthesis Stage (from Kalogerakis’ SIGGRAPH slides) Probabilistic model Output Synthesis stage

109. Synthesizing a set of components Enumerate high-probability instantiations of the model

110. Source shapes Unoptimized new shape Optimized new shape Optimizing component placement

111. Results Source shapes (colored parts are selected for the new shape) New shape

112. R N1N1 N2N2 C1C1 C2C2 S1S1 S2S2 D1D1 D2D2 Results of alternative models: no latent variables

113. Results of alternative models: no part correlations R N1N1 N2N2 C1C1 C2C2 S1S1 S2S2 D1D1 D2D2

114. Papers to be covered in this lecture Shape synthesis A Probabilistic Model for Component-based Shape Synthesis [Kalogerakis et al. 12] Co-constrained Handles for Deformation in Shape Collections [Yumer et al. 14]

115. Co-Constrained Handles for Shape Deformation Input Constraints Output Learning stage Synthesis stage

116. Pipeline Initial Segmentation Initial Abstraction Clustering Co-constrained Abstraction Handles Constraints

117. Pipeline Initial Segmentation Initial Abstraction Clustering Co-constrained Abstraction Handles Constraints

118. Clustering Seed surface clustering Global Clustering

119. Clustering Seed surface clustering – Aligning semantically similar/identical segments Global Clustering – Creating larger clusters that join the surfaces from different segments – Based on geometric features of the surfaces, including PCA basis, normal and curvature signature of the surface.

120. Clustering Door handle Wheels Seed clustering Different clusters

121. Clustering Door handle Wheels Global clustering Same cluster

122. Pipeline Initial Segmentation Initial Abstraction Clustering Co-constrained Abstraction Handles Constraints

123. Co-constrained abstraction Planer, spherical, or cylindrical? Initial abstraction Co- abstraction

124. Identify constraints of DOFs Identify as constraints if variance is below certain threshold Translation Rotation Scaling

125. Pipeline Initial Segmentation Initial Abstraction Clustering Co-constrained Abstraction Handles Constraints

126. Correlation among deformation handles Estimate the conditional feature probability

127. Results

128. Summary 3D shape from image or video Using silhouettes, colors or features (e.g. HOG) in image Using voxel occupancies, aspect parts, level set, or meshes to represent 3D shapes Either generative model, shared shape space, or discriminative model

129. Summary Shape synthesis Generative model based on semantic parts, which models styles also. Modeling correlation between geometric features.