1. 3D Object Modelling and Classification Intelligent Robotics Research Centre (IRRC) Department of Electrical and Computer Systems Engineering Monash University, Australia Visual Perception and Robotic Manipulation Springer Tracts in Advanced Robotics Chapter 4 Geoffrey Taylor Lindsay Kleeman

2. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 2 Contents Introduction and motivation. Split-and-merge segmentation algorithm New method for surface type classification based on Gaussian image and convexity analysis Fitting geometric primitives Experimental results Conclusions

3. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 3 Introduction Motivation: enable a humanoid robot to perform ad hoc tasks in a domestic or office environment. Flexibility in an unknown environment requires data driven segmentation to support object classification. Metalman: an upper-torso humanoid robot

4. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 4 Introduction Object modelling in robotic applications: –CAD models (Kragić, 2001) –Generalized cylinders (Rao et al, 1989) –Non-parametric (Müller & Wörn, 2000) –Geometric primitives (Yang & Kak, 1986) Many domestic objects can be adequately modelled with geometric primitives. Colour/range data provided by robust stereoscopic light stripe scanner (Taylor et al, 2002).

5. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 5 Segmentation Basic techniques: –Region Growing: iteratively grow seed segments. –Split-and-Merge: find region boundaries. –Clustering: transform and group points. Region growing requires accurate range data for fitting primitives to small seed regions. Split-and-Merge maintains large regions that can be robustly fitted to primitives.

6. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 6 Segmentation Raw range/colour data from stereoscopic light stripe camera. Calculate normal vector and surface type for each range element.

7. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 7 Segmentation Remove range discontinuities and creases. Fit primitives. Compare best model to dominant surface type. Split poorly modelled regions by surface type and fit primitives again.

8. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 8 Segmentation Iteratively grow regions by adding unlabelled pixels that satisfy model. Merge regions using iterative boundary cost minimization to compensate for over- segmentation.

9. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 9 Segmentation Extract primitives and add texture using projected colour data. Use models for object classification, tracking and task planning

10. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 10 Surface Type Determine local shape of NxN element patch:

11. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 11 Classification methods Conventional method: –Fit surface, calculate mean and Gaussian curvature –Classify based on curvature sign ( > 0, < 0, = 0)  Sensitive to noise (second-order derivatives required)  Arbitrary approximating function introduces bias. Our novel method: –Based on convexity and principal curvatures. –Non-parametric (no approximating surface) –Robust to noise

12. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 12 Classification Number of non-zero principal curvatures Convexity convex concave neither Zero One Two

13. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 13 Principal Curvatures Determine number of principal curvatures from Gaussian image of surface patch. Surface representationGaussian image

14. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 14 Principal Curvatures Spread of normal vectors in Gaussian image of patch indicates non-zero principal curvature. plane ridge/valleypit/peak/saddle

15. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 15 Align central normal to z-axis. Measure spread in direction  using MMSE: Optimize with respect to  Two solutions: ( , e) max and ( , e) min Non-zero curvature when e max > e th or e min > e th Principal Curvatures min  max  y x a min

16. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 16 Convexity Convex Concave n0n0 n1n1 d n 1 x n 0 (n 1 x n 0 ) x d n1n1 n0n0 d n 1 x n 0 (n 1 x n 0 ) x d

17. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 17 For each element in patch, calculate: Let S=N cv /N cc, ratio of convex to concave elements. Global convexity given by dominant local property: Convexity convex (peak, ridge):S > S th concave (pit, valley):S < 1/S th neither (plane, saddle):1/S th < S < S th

18. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 18 Surface Type Summary principal curvatures convexity raw 3D scansurface type

19. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 19 Planes: – Principal component analysis Spheres, cylinders, cones: –Minimize distance to fitted surface: –Levenberg-Marquardt numerical optimization. –Initial estimate of parameters required. Choose model with minimum error, e < e th. Fitting Primitives

20. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 20 Cylinder Estimation Estimate cylinder axis from Gaussian image: min  y x a Cylindrical region and axis Gaussian image and direction of minimum spread a

21. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 21 Results Box, ball and cup: Raw colour/range scanDiscontinuities, surface type

22. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 22 Results Box, ball and cup: Region growing, mergingExtracted object models

23. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 23 Results Bowl, funnel and goblet: Raw colour/range scanDiscontinuities, surface type

24. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 24 Results Bowl, funnel and goblet: Region growing, mergingExtracted object models

25. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 25 Results Comparison with curvature-based method: Besl and Jain, 1988Non-parametric result

26. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 26 Conclusions Split-and-merge segmentation using surface type and geometric primitives is capable of modelling a variety of domestic objects using planes, spheres, cylinders and cones. New surface type classifier based on principal curvatures and convexity provides greater robustness than curvature-based methods without additional computational cost.