Presentation is loading. Please wait.

Presentation is loading. Please wait.

3D Point Capsule Networks Lifting Capsule Networks to Raw 3D Data

Similar presentations


Presentation on theme: "3D Point Capsule Networks Lifting Capsule Networks to Raw 3D Data"— Presentation transcript:

1 3D Point Capsule Networks Lifting Capsule Networks to Raw 3D Data
Yongheng Zhao,  Tolga Birdal,  Haowen Deng,  Federico Tombari CVPR Tutorial Sunday, June 16, 2019

2 representing 3d data octree graphs implicit surfaces
algebraic surfaces

3 NxD matrix of attributes
point clouds  NxD matrix of attributes …... Raw data: Efficient Sparse: Memory friendly Generic Arbitrary accuracy x, y, z r,g,b

4 why are point clouds hard?
 unstructured geometry  cannot be projected on a single plane (different manifolds exist)  basic representation is permutation dependent  sparse input : dense convolutions are wasteful  varying data density

5 consuming point clouds in networks: point-net
Nx3 point set X MLP local feature global feature Qi, Charles R., et al. "Pointnet: Deep learning on point sets for 3d classification and segmentation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

6 point-net pointnet

7 a new 3D auto-encoder D E Input Shape Latent Capsules Latent Feature
Reconstruction

8 "Learn to Fold a Napkin into Almost Any 3D Shape, Deeply"
upsample vs deform latent code deform (MLP) concatanate Fixed Grid Template "Learn to Fold a Napkin into Almost Any 3D Shape, Deeply" Yang, Yaoqing, et al. "Foldingnet: Point cloud auto-encoder via deep grid deformation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

9 a quick look at the decoders
Groueix, Thibault, et al. "A papier-mâché approach to learning 3d surface generation." Proceedings of the IEEE conference on computer vision and pattern recognition Yang, Yaoqing, et al. "Foldingnet: Point cloud auto-encoder via deep grid deformation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

10 architecture

11 capsules act locally

12 Input point cloud with part label Part label for the capsule
optionally supervise Reconstruction Latent Capsules capsule-part association Input point cloud with part label Part label for the capsule

13 Part label for the capsule
optionally supervise Latent Capsules Part prediction Conv Cross Entropy Loss Part label for the capsule

14 part segmentation D Latent Capsules Part prediction Conv

15 part segmentation

16 part segmentation

17 a rather new application: part interpolation/replacement
Part replacement A Capsule-Part Association Latent Capsules E Target shape Latent Capsules E Tail Wing Body A Tail Wing Body A Part interpolation Segmentation D Segmentation D Source shape

18 part interpolation/replacement

19 part interpolation / replacement

20

21 extracting invariant 3D local descriptors
Deng, Haowen, Tolga Birdal, and Slobodan Ilic. "Ppfnet: Global context aware local features for robust 3d point matching." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

22 reconstruction quality
transfer learning part segmentation with limited data

23 https://tinyurl.com/yxq2tmv3
to take home... representation matters and is unsolved. a rich latent space is desirable and can (to a certain extent) be achieved by capsules + dynamic routing. can we make capsules specialize on other 3D shape properties?


Download ppt "3D Point Capsule Networks Lifting Capsule Networks to Raw 3D Data"

Similar presentations


Ads by Google