System Integration and Experimental Results Intelligent Robotics Research Centre (IRRC) Department of Electrical and Computer Systems Engineering Monash University, Australia Visual Perception and Robotic Manipulation Springer Tracts in Advanced Robotics Chapter 7 Geoffrey Taylor Lindsay Kleeman

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 2 Overview Stereoscopic light stripe scanning Object Modelling and Classification Multicue tracking (edges, texture, colour) Visual servoing Real-world experimental manipulation tasks with an upper-torso humanoid robot

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 3 Motivation To enable a humanoid robot to perform manipulation tasks in a domestic environment: –A domestic helper for the elderly and disabled Key challenges: –Ad hoc tasks with unknown objects –Robustness to measurement noise/interference –Robustness to calibration errors –Interaction to resolve ambiguities –Real-time operation

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 4 Architecture

5 Light Stripe Scanning Triangulation-based depth measurement. Stripe generator Camera Scanned object B D

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 6 Stereo Stripe Scanner Three independent measurements provide redundancy for validation. Left camera L Scanned object 2b Right camera R Laser diode X xLxL xRxR Left image plane Right image plane θ

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 7 Reflections/Cross Talk

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 8 Single Camera Result Single camera scannerRobust stereoscopic scanner

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 9 3D Object Modelling Want to find objects with minimal prior knowledge. –Use geometric primitives to represent objects Segment 3D scan based on local surface shape. Surface type classification

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 10 Segmentation Fit plane, sphere, cylinder and cone to segments. Merge segments to improve fit of primitives. Raw scan Final segmentation Surface type classification Geometric models

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 11 Object Classification Scene described by adjacency graph of primitives. Objects described by known sub-graphs.

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 12 Modeling Results Box, ball and cup: Raw colour/range scanTextured polygonal models

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 13 Multi-Cue Tracking Individual cues are only robust under limited conditions: –Edges fail in low contrast, distracted by texture –Textures not always available, distracted by reflections –Colour gives only partial pose Fusion of multiple cues provides robust tracking in unpredictable conditions.

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 14 Tracking Framework 3D Model-based tracking: models modelled from light stripe range data. Colour (selector), edges and texture (trackers) are measured simultaneously in every frame. Measurements fused in Extended Kalman filter: –Cues interact with state through measurement models –Individual cues need not recover the complete pose –Extensible to any cues/cameras for which a measurement model exists.

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 15 Colour Cues Filter created from colour histogram in ROI: –Foreground colours promoted in histogram –Background colours supressed in histogram Captured image used to generate filter Output of resulting filter

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 16 Edge Cues Combine with colour to get silhouette edges Sobel mask directional edges Fitted edges Predicted projected edges

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 17 Texture Cues Rendered predictionFeature detectorMatched templates Outlier rejectionFinal matched features

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 18 Tracking Result

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 19 Visual Servoing Position-based 3D visual servoing (IROS 2004). Fusion of visual and kinematic measurements.

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 20 Visual Servoing 6D pose of hand estimated using extended Kalman filter with visual and kinematic measurements. State vector also includes hand-eye transformation and camera model parameters for calibration.

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 21 Grasping Task Grasp a yellow box without prior knowledge of objects in the scene.

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 22 Grasping Task

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 23 Pouring Task Pour the contents of a cup into a bowl.

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 24 Pouring Task

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 25 Smell Experiment Fusion of vision, smell and airflow sensing to locate and grasp a cup containing ethanol.

Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics 26 Summary Integration of stereoscopic light stripe sensing, geometric object modelling, multi-cue tracking and visual servoing allows robot to perform ad hoc tasks with unknown objects. Suggested directions for future research: –Integrate tactile and force sensing –Cooperative visual servoing of both arms –Interact with objects to learn and refine models –Verbal and gestural human-machine interaction