1. Object Recognition & Model Based Tracking © Danica Kragic Tracking system

2. Object Recognition & Model Based Tracking © Danica Kragic Motivation Manipulating objects in domestic environments Localization / Navigation Object Recognition Servoing Tracking Grasping Pose estimation

3. Object Recognition & Model Based Tracking © Danica Kragic Steps Recognition (2D) Tracking (2D) Pose estimation (3D) : Initial pose estimation Where in the image … ? Where in the world … ?

4. Object Recognition & Model Based Tracking © Danica Kragic Initial Pose Estimation Recognition/Tracking Pose estimation (x,y) (X,Y,Z, , ,  )

5. Object Recognition & Model Based Tracking © Danica Kragic Example Objects

6. Object Recognition & Model Based Tracking © Danica Kragic Characteristics Simple geometry ( polyhedra, cones, cylinders ) Specular surfaces Background Illumination Slippery objects

7. Object Recognition & Model Based Tracking © Danica Kragic Characteristics Simple geometry wireframe models Specular surfaces - ll – Illumination - ll – Background - ll – Highly texture appearance Slippery objects power grasps

8. Object Recognition & Model Based Tracking © Danica Kragic Model Based Techniques Appearance based methods Geometry based methods 3D wireframe models Complete pose estimation Techniques from computer graphics used for rendering FUSION!

9. Object Recognition & Model Based Tracking © Danica Kragic Object Recognition Removes background, preserves object. Necessary to raise the signal to noise ratio, for the pose estimatior. Solved using color cooccurrence histograms.

10. Object Recognition & Model Based Tracking © Danica Kragic Pose Estimation An apperance based method is used to recognize the object, and estimate an initial pose. A geometric model based method is used to obtain an accurate pose. Algorithm combines the robustness of appearance based methods with the accuracy of feature based methods.

11. Object Recognition & Model Based Tracking © Danica Kragic

12. Object Recognition & Model Based Tracking © Danica Kragic

13. Object Recognition & Model Based Tracking © Danica Kragic Color Cooccurrence Histograms Apperance based method. Based on color cues only. Superior to standard color histograms. Invariant to translation and rotation. Robust towards scale changes.

14. Object Recognition & Model Based Tracking © Danica Kragic Building Color Cooccurrence Histograms All pairs of pixels within a certain radius contribute to the histogram. Example: 4x4 image with 3 colors, and a maximum radius of 3 pixels. Histogram:

15. Object Recognition & Model Based Tracking © Danica Kragic Building Color Cooccurrence Histograms When all pairs have been counted, the histogram is normalized. Each bin is divided with the total number of pixel pairs. Histogram: 50 %

16. Object Recognition & Model Based Tracking © Danica Kragic Color Cooccurrence Histograms - Matching A common histogram matching method is used. Reduces the effect of background noise, as unexpected colors will not penalize the match value.

17. Object Recognition & Model Based Tracking © Danica Kragic Color Quantization Before the histogram can be built, the colors in the image need to be quantized. This is done using k- means clustering. Red Green

18. Object Recognition & Model Based Tracking © Danica Kragic Color Quantization Images are normalized prior to quantization, in order to decrease the effect of varying lighting conditions. Only the red and green components are preserved. Performance equal to RGB and HSV. Red Green

19. Object Recognition & Model Based Tracking © Danica Kragic Color Constancy Problem If lighting conditions change, colors may ”fall out of” their original cluster, or even worse, into another one. Red Green green light

20. Object Recognition & Model Based Tracking © Danica Kragic Object Segmentation - Training The system was trained using both front and back sides of the objects. The background of the training images was manually removed before training.

21. Object Recognition & Model Based Tracking © Danica Kragic Object Segmentation A search window scans through the image, comparing the cooccurrence histogram with the stored histogram from the training images. The result is a vote matrix.

22. Object Recognition & Model Based Tracking © Danica Kragic Object Segmentation From the vote matrix, segmentation windows are contructed. Starting from the global maximum, adjacent rows and columns are added as long as the vote values give sufficient support.

23. Object Recognition & Model Based Tracking © Danica Kragic Object Segmentation - Results Out of 50 test images, 49 objects were successfully segmented. Average segmentation time was 1.7 s on a 500 MHz Sun station.

24. Object Recognition & Model Based Tracking © Danica Kragic Pose Estimation The geometric model based pose estimator requires an initial pose to converge. The initial pose is estimated using color cooccurrence histograms.

25. Object Recognition & Model Based Tracking © Danica Kragic Pose Estimation - Training 70 training images were used. The pose of the object varied over the training images. The correct pose of the object in the training image was stored, together with the cooccurrence histogram.

26. Object Recognition & Model Based Tracking © Danica Kragic Pose Estimation The object with the unknown pose is compared to each of the training examples. The result is a match value graph.

27. Object Recognition & Model Based Tracking © Danica Kragic Pose Estimation The match value graph is filtered using a Gaussian kernel. Superior method compared to a nearest-neighbor approach.

28. Object Recognition & Model Based Tracking © Danica Kragic Initial Pose Estimation Appearance based

29. Object Recognition & Model Based Tracking © Danica Kragic Principle Component Analisys Learning stage – compressing image set using eigenspace representation PCA PCAPCA Pose recognition stage – closest point search on appearance manifold PCAPCA Fitting stage – closest line search for pose refinement

30. Object Recognition & Model Based Tracking © Danica Kragic PCA – i(q) Pose Appearance Eigenstructure decomposition problem PCA

31. Object Recognition & Model Based Tracking © Danica Kragic PCA Implicit covariance matrix (conjugate gradient method) PCA

32. Object Recognition & Model Based Tracking © Danica Kragic PCA Pose determination PCA

33. Object Recognition & Model Based Tracking © Danica Kragic Initialization by PCA

34. Object Recognition & Model Based Tracking © Danica Kragic Geometric Model Based Pose Estimation Finally, the algorithm was integrated with the model based pose estimator.

35. Object Recognition & Model Based Tracking © Danica Kragic Geometric Model Based Pose Estimation

36. Object Recognition & Model Based Tracking © Danica Kragic Local refinement by tracking  H = (14 0 60 15 6 5) [mm, deg]

37. Object Recognition & Model Based Tracking © Danica Kragic Modeling

38. Object Recognition & Model Based Tracking © Danica Kragic Modeling

39. Object Recognition & Model Based Tracking © Danica Kragic Pose estimation DeMenthon and Davis 1995 Orthographic projection Iterative method No initial guess needed This step is followed by an extension of Lowe’s nonlinear approach (Canceroni, Araujo and Brown et al.)

40. Object Recognition & Model Based Tracking © Danica Kragic Tracking Lie algebra approach Rigid body motion SE(3) (6D Lie group)

41. Object Recognition & Model Based Tracking © Danica Kragic Image motion with: L - observed motion in an image point i

42. Object Recognition & Model Based Tracking © Danica Kragic Normal flow

43. Object Recognition & Model Based Tracking © Danica Kragic Rendering example

44. Object Recognition & Model Based Tracking © Danica Kragic 3D pose update The change in pose is estimated using least square approach: where  represents the quantities of Euclidian motion i

45. Object Recognition & Model Based Tracking © Danica Kragic 3D pose update

46. Object Recognition & Model Based Tracking © Danica Kragic Examples

47. Object Recognition & Model Based Tracking © Danica Kragic Examples

48. Object Recognition & Model Based Tracking © Danica Kragic Example

49. Object Recognition & Model Based Tracking © Danica Kragic Task 1 – Align and Track

50. Object Recognition & Model Based Tracking © Danica Kragic Task 1 – Align and Track

51. Object Recognition & Model Based Tracking © Danica Kragic Task 2 – Object Positioning

52. Object Recognition & Model Based Tracking © Danica Kragic

53. Object Recognition & Model Based Tracking © Danica Kragic

54. Object Recognition & Model Based Tracking © Danica Kragic Task 3 - Insertion

55. Object Recognition & Model Based Tracking © Danica Kragic Insertion task How much a-priori info can we used?

56. Object Recognition & Model Based Tracking © Danica Kragic Pick and Place