1. Vision Based Control Motion Matt Baker Kevin VanDyke

2. Robots  Today’s robots perform complex tasks with amazing precision and speed  Why then have they not moved from the structure of the factory floor into the “real” world? What is the limiting factor? Vision

3. A “Seeing Robot”  A robot that can perceive and react in complex and unpredictable surroundings  This is not possible with the marker-based systems in use in most laboratory vision-based control systems

4. Common reasons for failure of vision systems  Small changes in the environment can result in significant variations in image data  Changes in contrast  Unexpected occlusion of features

5. Robustness  Stable measurements of local feature attributes, despite significant changes in the image data, that result from small changes in the 3D environment [1].

6. Enhanced Techniques  The Hough-Transform  Robust color classification  Occlusion prediction  Multisensory visual servoing

7. Hough Transform  Used to extract geometrical object features from digital images

8. Hough Transform (con’t)  Features are extracted by detecting maximums in the image  Example geometric features encountered: Lines: Circles: Ellipses:

9. Hough Transform (cont’d)  Advantages  Noise and background clutter do not impair detection of local maxima  Partial occlusion and varying contrast are minimized  Negatives  Requires time and space storage that increases exponentially with the dimensions of the parameter space

10. Hough Transform (con’t)  a real-time application of HT requires both a fast image preprocessing step and an efficient implementation Implementation of a circle tracking algorithm based on HT

11. Robust color classification  Color has high disambiguity power  Real-time is required  Supervised color segmentation  The color distribution of the current scene is analyzed and colors that do not appear in the scene are used as marker colors  These markers are then used as the input to the visual servoing system  Colors represented by their hue-saturation value (H&S relate to color, V relates to brightness)

12. Robust color classification (con’t)  Color segmentation  Choose four colors as marker colors  Color markers brought onto object we wish to track  markers outlined  Color distribution computed  Initial segmentation

13. Model-based handling of occlusion  The previous two techniques take care of bad illumination and partial occlusion  What about aspect changes (complete occlusion)?  Build and maintain a 3D model of the observed objects so they can be tracked despite occlusion  Then use prediction

14. Tracking system model Sensor data Feature extraction 3D pose estimation Robot control Pose prediction Visibility determination Feature selection Geometric model Designed to handle aspect changes online

15. Prediction  Extract measurements of object features based on raw sensor data  Estimate the spatial position and orientation of the target object  Based on history of estimated poses and assumptions about the object motion you can predict an object pose expected in next sampling interval  With predicted pose and 3D model we are able to determine feature visibility in advance  Guide the feature extraction process for the next frame without the risk of searching for occluded features

16. Model-based handling of occlusion (con’t)  Efficient Hidden Line Removal  Explicit modeling of curved object structures allows us to remove virtual lines – or lines that do not have a physical correspondence in the camera image

17. Object tracking with visibility determination

18. Multisensory Servoing  Redundant information is used to increase the performance of the servoing system as well as the robustness against failing sensors

19. Vision Controlled Robot Model

20. Conclusions  We explored a variety of image processing techniques that can significantly improve the robustness of visual servoing systems  These techniques can be implemented in modern robot vision control systems  Techniques such as these will make machine vision in robots a reality in the near future