1. Chapter 5 Multi-Cue 3D Model- Based Object Tracking Geoffrey Taylor Lindsay Kleeman Intelligent Robotics Research Centre (IRRC) Department of Electrical and Computer Systems Engineering Monash University, Australia Visual Perception and Robotic Manipulation Springer Tracts in Advanced Robotics

2. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 2 Contents Motivation and Background Overview of proposed framework Kalman filter Colour tracking Edge tracking Texture tracking Experimental results

3. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 3 Introduction Research aim: –Enable a humanoid robot to manipulate a priori unknown objects in an unstructured office or domestic environment. Previous results: –Visual servoing –Robust 3D stripe scanning –3D segmentation, object modelling Metalman

4. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 4 Why Object Tracking? Metalman uses visual servoing to execute manipulations: control signals are calculated from observed relative pose of gripper and object. Object tracking allows Metalman to: –Handle dynamic scenes –Detect unstable grasps –Detect motion from accidental collisions –Compensate for calibration errors in kinematic and camera models

5. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 5 Why Multi-Cue? Individual cues only provide robust tracking under limited conditions: –Edges fail in low contrast, distracted by texture –Textures not always available, distracted by reflections –Colour gives only partial pose Fusion of multiple cues provides robust tracking in unpredictable conditions.

6. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 6 Multi-Cue Tracking Mainly applied to 2D feature-based tracking. Sequential cue tracking: –Selector (focus of attention) followed by tracker –Can be extended to multi-level selector/tracker framework (Tomaya and Hager 1999). Cue integration: –Voting, fuzzy logic (Kragić and Christensen 2001) –Bayesian fusion, probabilistic models –ICondensation (Isard and Blake 1998)

7. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 7 Proposed framework 3D Model-based tracking: models extracted using segmentation of range data from stripe scanner. Colour (selector), edges and texture (trackers) optimally fused in a Kalman filter framework. Colour + range scanTextured polygonal models

8. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 8 Kalman filter Optimally estimate object state x k given previous state x k-1 and new measurements y k. System state comprises pose and velocity screw: x k = [p k, v k ] T State Prediction (constant velocity dynamics): p * k = p k-1 + v k-1 ·  tv k = v k-1 State Update: x k = x * k + K k  [ y k - y * (x * k ) ] Need measurement function for each cue: y * (x * k )

9. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 9 Measurements For each new frame, predict object pose of and project model onto image to define region of interest (ROI): –only process within ROI to eliminate distractions and reduce computational expense Captured frame, predicted pose & ROI

10. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 10 Colour Tracking Colour filter created from RGB histogram of texture Image processing: –Apply filter to ROI –Calculate centroid of the largest connected blob Measurement prediction: –Project centroid of model vertices at predicted pose onto the image plane

11. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 11 Edge Tracking To avoid texture, only consider silhouette edges Image processing: –Extract directional edge pixels (Sobel masks) –Combine colour data to extract silhouette edges –Match pixels to projected model edge segments

12. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 12 Edge Tracking Fit line to matched points for each segment and extract angle and mean position Measurement prediction: –Project model vertices to image plane –For each model edge, calculate angle and distance to measured mean point

13. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 13 Texture Tracking Textures represented as 8×8 pixel templates with high spatial variation of intensity Image processing: –Render textured object in predicted pose –Apply feature detector (Shi & Tomasi 1994) –Extract templates, match to captured frame by SSD

14. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 14 Texture Tracking Apply outlier rejection: –Consistent motion vectors –Invertible matching Calculate the 3D position of texture features on the surface of the model Measurement prediction: –Project 3D surface features in current pose onto image plane

15. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 15 Experimental Results Three tracking scenarios: –Poor visual conditions –Occluding obstacles –Rotation about axis of symmetry Off-line processing of captured video sequence: –Direct comparison of tracking performance using edges only, texture only, and nultimodal fusion. –Actual processing rate is about 15 frames/sec

16. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 16 Poor Visual Conditions Colour, texture and edge tracking

17. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 17 Poor Visual Conditions Texture onlyEdges only

18. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 18 Occlusions Colour, texture and edge tracking

19. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 19 Occlusions Texture onlyEdges only

20. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 20 Occlusions Tracking precision

21. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 21 Symmetrical Objects Colour, texture and edge tracking

22. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 22 Symmetrical Objects Object orientation

23. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 23 Conclusions Fusion of multimodal visual features overcomes weaknesses in individual cues, and provides robust tracking where single cue tracking fails. The proposed framework is extensible; additional modalities can be fused provided a suitable measurement model is devised.

24. Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 24 Open Issues Include additional modalities: –optical flow (motion) –depth from stereo Calculate measurement errors as part of feature extraction for measurement covariance matrix. Modulate size of ROI to reflect current state covariance, so ROI automatically increases as visual conditions degrade, and decreases under good conditions to increase processing speed.