1. Final Class: Range Data registration CISC4/689 Credits: Tel-Aviv University

2. The Problem Align two partially- overlapping meshes given initial guess for relative transform

3. Data Types Point sets Line segment sets (polylines) Implicit curves : f(x,y,z) = 0 Parametric curves : (x(u),y(u),z(u)) Triangle sets (meshes) Implicit surfaces : s(x,y,z) = 0 Parametric surfaces (x(u,v),y(u,v),z(u,v)))

4. Motivation Shape inspection Motion estimation Appearance analysis Texture Mapping Labeling (atlas registration)

5. Motivation Range images registration

6. Range Scanners

7. Aligning 3D Data

8. Iterative Closest Point Algorithm Also called ICP algorithm proposed in 1992. Many variants have come into existence after the original algorithm proposed by Besl and Mackay.

9. Corresponding Point Set Alignment Let M be a model point set. Let S be a scene point set. We assume : 1.N M = N S. 2.Each point S i correspond to M i.

10. Corresponding Point Set Alignment The objective function : The alignment is :

11. Aligning 3D Data If correct correspondences are known, can find correct relative rotation/translation

12. Aligning 3D Data How to find correspondences: User input? Feature detection? Signatures? Alternative: assume closest points correspond

13. Aligning 3D Data How to find correspondences: User input? Feature detection? Signatures? Alternative: assume closest points correspond

14. Aligning 3D Data Converges if starting position “close enough“

15. Closest Point Given 2 points r 1 and r 2, the Euclidean distance is: Given a point r 1 and set of points A, the Euclidean distance is:

16. Finding Matches The scene shape S is aligned to be in the best alignment with the model shape M. The distance of each point s of the scene from the model is :

17. Finding Matches Finding each match is performed in O(N M ) worst case. Given correspondence, Y we can calculate alignment S is updated to be :

18. The Algorithm Init the error to ∞ Calculate correspondence Calculate alignment Apply alignment Update error If error > threshold Y = CP(M,S),e (rot,trans,d) S`= rot(S)+trans d` = d

19. Convergence Theorem Correspondence error : Alignment error:

20. ICP Variants Variants on the following stages of ICP have been proposed: 1.Selecting sample points (from one or both meshes) 2.Matching to points in the other mesh 3.Weighting the correspondences 4.Rejecting certain (outlier) point pairs 5.Assigning an error metric to the current transform 6.Minimizing the error metric w.r.t. transformation

21. ICP Variants 1.Selecting sample points (from one or both meshes). 2.Matching to points in the other mesh. 3.Weighting the correspondences. 4.Rejecting certain (outlier) point pairs. 5.Assigning an error metric to the current transform. 6.Minimizing the error metric w.r.t. transformation.

22. ICP Variants 1.Selecting sample points (from one or both meshes). 2.Matching to points in the other mesh using invariants. 3.Weighting the correspondences. 4.Rejecting certain (outlier) point pairs. 5.Assigning an error metric to the current transform. 6.Minimizing the error metric w.r.t. transformation.

23. ICP Variants 1.Selecting sample points (from one or both meshes). 2.Matching to points in the other mesh. 3.Weighting the correspondences. 4.Rejecting certain (outlier) point pairs. 5.Assigning an error metric to the current transform. 6.Minimizing the error metric w.r.t. transformation.

24. ICP Variants 1.Selecting sample points (from one or both meshes). 2.Matching to points in the other mesh. 3.Weighting the correspondences. 4.Rejecting certain (outlier) point pairs. 5.Assigning an error metric to the current transform. 6.Minimizing the error metric w.r.t. transformation.

25. Rejecting Pairs Inconsistent Pairs p1 p2 q2 q1

26. ICP Variants 1.Selecting sample points (from one or both meshes). 2.Matching to points in the other mesh. 3.Weighting the correspondences. 4.Rejecting certain (outlier) point pairs. 5.Assigning an error metric to the current transform. 6.Minimizing the error metric w.r.t. transformation.

27. Error metric and minimization Sum of squared distances between corresponding points. There exist closed form solutions for rigid body transformation : 1.SVD 2.Quaternions 3.Orthonoraml matrices 4.Dual quaternions.

28. 3D Surface-to-surface Motion Analysis Direct Shape-based method: –J. S. Duncan, et al. 1991 –J. Feldmar, et al. 1996 –Y. Wang, et al. 2000 –D. Meier, et al. 2002 Curvature differenceNormal difference Nonrigid Shape-based method: Nonrigid Shape-based method: Nonrigid shape relationship between the before-motion and after-motion surfaces is described by the undergoing nonrigid motion. Nonrigid shape relationship between the before-motion and after-motion surfaces is described by the undergoing nonrigid motion. C. Kambhamettu, et al. CVPR 1992 C. Kambhamettu, et al. CVPR 1992 C. Kambhamettu, et al. CVGIP:IU 1994 C. Kambhamettu, et al. CVGIP:IU 1994 C. Kambhamettu, et al. IVC 2003 C. Kambhamettu, et al. IVC 2003 P. Laskov, et al. PAMI 2003 P. Laskov, et al. PAMI 2003

29. 3D Surface-to-surface Motion Analysis Previous Nonrigid Shape-based Methods –A local coordinate system is constructed at each point of interest Defined motion has no explicit physical meaning –Each point of interest is looking for its corresponding point independently Motion consistency can not be guaranteed New Approach of Nonrigid Shape-based Method New Approach of Nonrigid Shape-based Method Nonrigid motion is modeled with a single spline-based motion field (GRBF) over the whole 3D surface. Nonrigid motion is modeled with a single spline-based motion field (GRBF) over the whole 3D surface. Nonrigid shape relationship is still described in the local coordinate system constructed at each point of interest Nonrigid shape relationship is still described in the local coordinate system constructed at each point of interest

30. Background At each point in the local coordinate system Before motionAfter motion First fundamental form Unit normal Discriminant Modulus of dilationMotion divergence

31. Background Assume orthogonal parameterization (F=0) Nonrigid shape relationship we used Nonrigid shape relationship for small deformation, additional assumption

32. From World to Local Coordinate Principal local coordinate system Motion transformation:

33. Problem Statement Motion estimation: recover the GRBF motion –What we want to know : –What we already know: –What we should do: Nonrigid relationship GRBF motion vector Least-square error

34. Paper bending Correspondence errors for five small-to-large paper bending deformations

35. Correspondence errors for five small-to-large smile deformations Correspondence errors for five small-to-large open-mouth deformations

36. Experiments Correspondences between frame 1 and Frame 6 is first estimated. Intermediate faces are reconstructed using linear interpolation, based on obtained correspondences between frame 1 and 6

37. Tongue Motion Analysis Sagittal+Coronal Sagittal+Axial 3D tongue

38. Tongue Motion Analysis A tagged MRI image. Tags are used for validation only.

39. Tongue Motion Analysis Correspondence errors for 11 tongue deformations

40. Evaluation of Structure and Nonrigid Motion (evaluation of both structure and motion) Torso Bullfight More..

41. Face Motion Application Facial Animation Parameters (FAPs) Facial Definition Parameters (FDPs) Drive a face:moviemovie Build and drive an Avatar:DemoDemo Face Anatomy Motion: Movie1Movie1 (US) Movie2Movie2 (MRI)

42. REVIEW www.cis.udel.edu/~chandra/courses.htm Exam will only take 1.30 min., though you are given 2 hours.

43. Thank you! Please complete Evaluations Come to my office and.. Take your mid-term2 Show project progress If you have to run to a class: show me quick progress, meet again. 5/20 is deadline for the project unless there is a great reason for extension for few more days. I prefer to give you an extra day or two rather than evaluating half finished product. Its easy to come show me the demo for project evaluation. However, project html reports will be gladly accepted.