1. Exploiting Segmentation for Robust 3D Object Matching Michael Krainin, Kurt Konolige, and Dieter Fox

2. Shape-based Pose Quality Metrics  Given an object model (triangle mesh) and a depth map,  Should not assume any object texture  Should be robust to clutter and occlusion 2

3. Outline  Progression of quality metrics  Iterative Closest Point  Beam-based Sensor Models  Segmentation Sensor Model  Key Contributions  Gradient-based optimization for beam-based models  Improves on ICP by reasoning about free-space  Novel sensor model framework  Uses segmentation to relax beam-independence assumptions  Explicitly reasons about surface extents 3

4. ICP Error Metric  Mean squared distance for pairs of closest points [Besl & McKay‘92]  No notion of amount of model or scene explained by the correspondences  Discards viewpoint/free-space information 4

5. xxx Beam-Based Sensor Models  Maximize data likelihood:  Typically assume pixel independence: 5 Measured Depth – Rendered Depth 0 l p(pixel | model, pose) x x x x

6. Beam Model Sensor Models  Beam-based sensor model properties:  Consider viewpoint  Prefer to explain more pixels using the model  Use in pose estimation:  2D particle-based localization [Thrun et al. ‘05]  Coarse to fine grid search for body tracking [Ganapathi et al. ‘10]  Annealed particle filters for vehicle detection and tracking [Petrovskaya & Thrun ‘09]  Has not been used with gradient-based optimization 6

7. Beam Model Optimization  We propose a gradient-based optimization  Levenberg-Marquardt using the g 2 o graph optimization framework [Kümmerle et al. ‘11]  Error function evaluations by re-rendering  OpenGL rendering, CUDA sensor model evaluation  1 ms per evaluation on a mid-range graphics card  ~500 evaluations per g 2 o optimization 7 ICP Beam Optimization observedrendered

8. Surface Extents  Can still match to the wrong surface  Idea: Use size of surfaces to rule out matches like these 8

9. Segmentation  (Over-)Segmentation as an estimate of surface extents  Connected components using depths and normals 9

10. Segmentation Sensor Model  Usual beam-based sensor model:  Segmentation model:  Allows consistent classification of entire segment as being generated from the model or not 10

11. Segmentation Sensor Model 11  Let S be a partition (segmentation) of D. Then given model M and pose T,  Let m i be an indicator for whether S i was generated from M  Segment pixels conditionally independent given classification

12. Error Functions Standard Beam-Based ModelSegmentation Model 12 x

13. Test Data  Recorded 46 cluttered scenes with ground truth poses 13

14. Pose Estimation Results  Optimization algorithms:  Random restart evaluation functions (upper bounded by 93% from previous result): ICPBeam Optimization Successful convergence [%] (exists a correct end pose among 20 random restarts) 86 ± 3 93 ± 2 ICP Mean Squared Error Standard Beam Model Segmentation Beam Model Successful matches [%] (determines correct end pose among 20 restarts) 79 ± 367 ± 4 85 ± 3 14

15. Robustness to Segmentation Granularity 15 Spam 1 of 50 degrees 8 degrees 12 degrees x x x (Fixed to 8 degrees in other experiments)

16. Conclusions  Contributions  Novel formulation for beam-based sensor models  Demonstration of gradient-based approach for optimizing beam-based sensor models  All code and test data freely available  Future Work  Detecting failed segmentations  Extensions such as color and normal direction  Directly optimizing the segmentation model 16

17. Evaluation Function Failure Cases 17 Standard Beam ModelSegmentation Beam Model

18. Object Confusion 18 AClCoOJSoSpTiToZ A212 Cl23 Co5 OJ51 So212 Sp5 Ti1111 To5 Z6 ICP MSE AClCoOJSoSpTiToZ A5 Cl5 Co212 OJ51 So41 Sp41 Ti4 To5 Z15 AClCoOJSoSpTiToZ A41 Cl5 Co41 OJ6 So5 Sp5 Ti4 To5 Z114 Standard Beam Model Segmentation Beam Model AbbrevObject AAll detergent ClClorox CoCoke OJOrange Juice SoSoup can SpSpam can TiTilex spray ToToothpaste ZZiploc bags