1. Overview of Our Sensors For Robotics

2. Machine vision Computer vision
To recover useful information about a scene from its 2-D projections.
To take images as inputs and produce other types of outputs (object shape, object contour, etc.)
Geometry + Measurement + Interpretation
To create a model of the real world from images.

3. Topics • Computer vision system • Image enhancement • Image analysis
• Pattern Classification

4. Related fields Image processing Computer graphics Pattern recognition
Transformation of images into other images
Image compression, image enhancement
Useful in early stages of a machine vision system
Computer graphics
Pattern recognition
Artificial intelligence
Psychophysics

5. Vision system hardware

6. Image Processing System

8. Image Representation

9. Image Image : a two-dimensional array of pixels
The indices [i, j] of pixels : integer values that specify the rows and columns in pixel values

11. Sampling, pixeling and quantization
The real image is sampled at a finite number of points.
Sampling rate : image resolution
how many pixels the digital image will have
e.g.) 640 x 480, 320 x 240, etc.
Pixel
Each image sample
At the sample point, an integer value of the image intensity

12. Quantization
Each sample is represented with the finite word size of the computer.
How many intensity levels can be used to represent the intensity value at each sample point.
e.g.) 28 = 256, 25 = 32, etc.

13. Color models Color models for images, RGB, CMY Color models for video,
YIQ, YUV (YCbCr)
Relationship between color models :

15. 6.7. Digital Cameras

16. Digital Cameras Technology Resolution • CCD (charge coupled devices)
• CMOS (complementary metal oxide semiconductor)
Resolution
• 60x80 black/white up to
• several Mega-Pixels in 32bit color
However: Embedded system has to have computing power to deal with this large amount of data!

17. Vision (camera + framegrabber)

18. Digital Cameras
Performance of embedded system: 10% - 50% of standard PC

19. Interfacing Digital Cameras to CPU
Interfacing to CPU:
• Completely depends on sensor chip specs
• Many sensors provide several different interfacing protocols
versatile in hardware design
software gets very complicated
• Typically: 8 bit parallel (or 4, 16, serial)
• Numerous control signals required

20. Interfacing Digital Cameras to CPU
Digital camera sensors are very complex units.
In many respects they are themselves similar to an embedded controller chip.
Some sensors buffer camera data and allow slow reading via handshake (ideal for slow microprocessors)
Most sensors send full image as a stream after start signal
(CPU must be fast enough to read or use hardware buffer or DMA)
We will not go into further details in this course. However, we consider camera access routines

21. Simplified diagram of camera to CPU interface

22. Problem with Digital Cameras
• Every pixel from the camera causes an interrupt
• Interrupt service routines take long, since they need to store register contents on the stack
• Everything is slowed down
Solution
• Use RAM buffer for image and read full image with single interrupt

23. Idea • Use FIFO as image data buffer
• FIFO is similar to dual-ported RAM, it is required since there is no synchronization between camera and CPU
• When FIFO is half full, interrupt is generated
• Interrupt service routine then reads FIFO until empty
(Assume delay is small enough to avoid FIFO overrun)

26. Bayer Pattern

27. De-Mosaic

29. Conversion in Digital Cameras
Bayer Pattern
• Output format of most digital cameras
• Note:
2x2 pattern is not spatially located in a single point!
• Can be simply converted to RGB (drop one green byte)
160x120 Bayer → 80x60 RGB
• Can be better converted using “demosaicing” technique
160x120 Bayer → 160x120 RGB

31. CMUCAM2+ CAMERA www.seattlerobotics.com
The camera can track user defined color blobs at up to 50 fps (frames per second)
Track motion using frame differencing at 26 fps
Find the centroid of any tracking data
Gather mean color and variance data
Gather a 28 bin histogram of each color channel
Manipulate horizontal pixel differenced images
Arbitrary image windowing
Adjust the camera’s image properties
This camera can
do a lot of processing

32. This camera can do a lot of processing Dump a raw image
Up to 160 X 255 resolution
Support multiple baud rates
Control 5 servos outputs
Slave parallel image processing mode off of single camera bus
Automatically use servos to do two axis color tracking
B/W analog video output (Pal or NTSC)
Flexible output packet customization
Multiple pass image processing on a buffered image

34. Vision Guided Robotics
and Applications in Industry and Medicine

35. Contents Robotics in General Industrial Robotics Medical Robotics
What can Computer Vision do for Robotics?
Vision Sensors
Issues / Problems
Visual Servoing
Application Examples
Summary

36. Industrial Robot vs Human
Robot Advantages:
Strength
Accuracy
Speed
Does not tire
Does repetitive tasks
Can Measure
Human advantages:
Intelligence
Flexibility
Adaptability
Skill
Can Learn
Can Estimate
Robot needs vision

37. Industrial Robot Requirements: Accuracy Tool Quality Robustness
Strength
Speed
Price Production Cost
Maintenance
Production Quality

38. Medical (Surgical) Robot
Requirements
Safety
Accuracy
Reliability
Tool Quality
Price
Maintenance
Man-Machine Interface

39. What can Computer Vision do for (industrial and medical) Robotics?
Accurate Robot-Object Positioning
Keeping Relative Position under Movement
Visualization / Teaching / Telerobotics
Performing measurements
Object Recognition
Registration
Visual Servoing

40. Vision Sensors Single Perspective Camera
Multiple Perspective Cameras (e.g. Stereo Camera Pair)
Laser Scanner
Omnidirectional Camera
Structured Light Sensor

41. Vision Sensors
Single Perspective Camera
Single projection

42. Vision Sensors
Multiple Perspective Cameras (e.g. Stereo Camera Pair)

43. Vision Sensors
Multiple Perspective Cameras (e.g. Stereo Camera Pair)

44. Vision Sensors
Multiple Perspective Cameras (e.g. Stereo Camera Pair)

45. Vision Sensors
Laser Scanner

46. Vision Sensors
Laser Scanner

47. Vision Sensors
Omnidirectional Camera

48. Vision Sensors
Omnidirectional Camera

49. Vision Sensors
Structured Light Sensor
                                                                             
                                                             
                                                         
Figures from PRIP, TU Vienna

50. Issues/Problems of Vision Guided Robotics
Measurement Frequency
Measurement Uncertainty
Occlusion, Camera Positioning
Sensor dimensions

51. Visual Servoing Vision System operates in a closed control loop.
Better Accuracy than „Look and Move“ systems
Figures from S.Hutchinson: A Tutorial on Visual Servo Control

52. Visual Servoing Example: Maintaining relative Object Position
Figures from P. Wunsch and G. Hirzinger. Real-Time Visual Tracking of 3-D Objects with Dynamic Handling of Occlusion

53. Camera Configurations for Visual Servoing
End-Effector Mounted
Fixed
Figures from S.Hutchinson: A Tutorial on Visual Servo Control

54. Visual Servoing Architectures
Figures from S.Hutchinson: A Tutorial on Visual Servo Control

55. Position-based vs Image Based control in Visual Servoing
Alignment in target coordinate system
The 3D structure of the target is rconstructed
The end-effector is tracked
Sensitive to calibration errors
Sensitive to reconstruction errors
Image based:
Alignment in image coordinates
No explicit reconstruction necessary
Insensitive to calibration errors
Only special problems solvable
Depends on initial pose
Depends on selected features
End-effector
target
Image of end effector
Image of target

56. EOL and ECL control in Visual Servoing
EOL: endpoint open-loop; only the target is observed by the camera
ECL: endpoint closed-loop; target as well as end-effector are observed by the camera
EOL
ECL

57. Visual Servoing Position Based Algorithm: Example: point alignment
Estimation of relative pose
Computation of error between current pose and target pose
Movement of robot
Example: point alignment p1 p2

58. Visual Servoing Position based point alignment p1m p2m
Goal: bring e to 0 by moving p1
e = |p2m – p1m|
u = k*(p2m – p1m)
pxm is subject to the following measurement errors: sensor position, sensor calibration, sensor measurement error
pxm is independent of the following errors: end effector position, target position
p1m
p2m d

59. Visual Servoing Image based point alignment p1 p2
Goal: bring e to 0 by moving p1
e = |u1m – v1m| + |u2m – v2m|
uxm, vxm is subject only to sensor measurement error
uxm, vxm is independent of the following measurement errors: sensor position, end effector position, sensor calibration, target position p1 p2 u1 v1 v2 u2 d1 d2 c1 c2

60. Visual Servoing Example Laparoscopy
Figures from A.Krupa: Autonomous 3-D Positioning of Surgical Instruments in Robotized Laparoscopic Surgery Using Visual Servoing

61. Visual Servoing Example Laparoscopy
Figures from A.Krupa: Autonomous 3-D Positioning of Surgical Instruments in Robotized Laparoscopic Surgery Using Visual Servoing

62. Registration Registration of CAD models to scene features:
Figures from P.Wunsch: Registration of CAD-Models to Images by Iterative Inverse Perspective Matching

63. Registration Registration of CAD models to scene features:
Figures from P.Wunsch: Registration of CAD-Models to Images by Iterative Inverse Perspective Matching

64. Summary on tracking and servoing
Computer Vision provides accurate and versatile measurements for robotic manipulators
With current general purpose hardware, depth and pose measurements can be performed in real time
In industrial robotics, vision systems are deployed in a fully automated way.
In medicine, computer vision can make more intelligent „surgical assistants“ possible.

65. Omnidirectional Vision Systems
CABOTO Robot’s task:
Building a topological map of an unknown environment;
Sensor:
Omnidirectional vision system;
Work’s aim:
Prove effectiveness of omnidirectional sensors for Spatial Semantic Hierarchy; SSH

66. Spatial Semantic Hierarchy...
... A model of the human knowledge of large spaces
Layers:
Sensory Level
Control Level
Causal Level
Topological Level
Metrical Level
Distance, Direction, Shape
Useful, but seldom essential
Minimal set of
Places, Paths and Regions
Interface with the robot’s sensory system
Control Laws, Transition of State, Distinctiveness Measure
View, Action, Distinct Place
Abstracts Discrete from Continous

67. Tracking Instrument tracking in laparoscopy
Figures from Wei: A Real-time Visual Servoing System for Laparoscopic Surgery

68. Omnidirectional Camera
Composed of:
Standard Color Camera
Convex Mirror
Perspex Cylinder

69. Pros e Cons Advantages Wide vision field High speed Vertical Lines
Rotational Invariance
Disadvantages
Low Resolution
Distortions
Low readability

70. Omnidirectional Vision and SSH
View Omnidirectional image
Exploring around the block P2 P4 P3 P5 P1
Robot should discriminate between “turns” and “travels”
We need an Effective Distinctiveness measure

71. Assumptions for vision system
Man-made environment
Floor flat and horizontal
Wall and objects surfaces are vertical
Static objects
Constant Lighting
Robot translates or rotates
No encoders

72. Features and Events Feature: Events: Vertical Edges A new edge
An edge disappears
Two edges 180° apart
Two pairs of edges 180° apart

73. Experiments Tasks of Caboto robot: Navigation; Map building;
Techniques:
Edge detection;
Colour marking;

74. Caboto’s Images

76. Results Correct tracking of edges Recognition of actions
Calculation of the turn angle
The path segmentation

77. Mirror shape should depend on robot task!
Mirror Design
Mirror shape should depend on robot task!
Design custom mirror profile
Maximise resolution in ROIs
Mirror Profile

78. The new mirror

79. Conclusion on Omnivision camera
Omnidirectional vision sensor is a good sensor for map building with SSH
Motion of the robot was estimated without active vision
The use of a mirror designed for this application will improve the system

80. Omnidirectional Cameras
Compound-eye camera
(from Univ. of Maryland, College Park. )
Panoramic cameras (from Apple)
Omnidirectional cameras
(from University of Picardie - France)

81. Student info.
% of lab marks can be deducted if rules and regulation are not followed
ex: by not cleaning up your bench or sliding your chairs back
underneath bench top.
For more technical information on boards, devices and sensors check out my web page at :
Students are responsible for their own extra parts ex: if you want to add a sensor or device that the dept. doesn’t have you are responsible for the purchase and delivery of that part, on rare occasion did the school purchase those parts.
Back packs off bench tops
TA’s will have student # based on station #
Important issue regarding the design of a new project is to do a current analysis before the start of your design
Setup a leader among your team so that you are better organized
Do not wait, before starting your project start now !
Prepare yourself before coming to the lab
It doesn’t work ! Ask yourself is it software or hardware, use the scope to trouble shoot
Fuses keeps on blowing, stop and do some investigation.
Do not cut any servo, battery and other device wire connectors. If you must please come and see me
No design must exceed 50 volts, ex: do not work with 120 volts AC
I can give you what I have regarding metal, wood and plastic recycled pieces and do some cuts or holes with my band saw and drill press for you,
PLEASE DO NOT ask me to barrow my tools. If you need to do a task with a special tool that I have then I shall do it for you.

82. Problems for students
Hardware and software components of a vision system for a mobile robot
Image representation for intelligent processing
Sampling, pixeling and Quantization
Color models
Types of digital cameras.
Interfacing digital cameras to CPU.
Problems with cameras.
Bayer Patterns and conversion.
What is good about CMUCAM?
Use of vision in industrial robots.
Use of multiple-perspective cameras.
Use of omnivision cameras.
Types of visual servoing.
Applications of visual servoing
Visual servoing in surgery
Explain tracking applications of vision.

83. References Dr. Gaurav Sukhatme Thomas Braunl Students 2002, class 479
Photo’s ,Text and Schematics Information
Alan Stewart
Dr. Gaurav Sukhatme
Thomas Braunl
Students 2002, class 479
E. Menegatti, M. Wright, E. Pagello