1. Vision-Based Motion Control of Robots
Azad Shademan
Guest Lecturer
CMPUT 412 – Experimental Robotics
Computing Science, University of Alberta
Edmonton, Alberta, CANADA

2. Vision-Based Control current desired Left Image Right Image B B A A A
Overview of VS, 2. What are the current approaches and their problems), 3. Which problem have we addressed? (what is our motivation for global model estimation) B
A. Shademan. CMPUT 412, Vision-based motion control of robots

3. Vision-Based Control Left Image Right Image B B B
A. Shademan. CMPUT 412, Vision-based motion control of robots

4. Vision-Based Control
Feedback from visual sensor (camera) to control a robot
Also called “Visual Servoing”
Is it any difficult?
Images are 2D, the robot workspace is 3D 2D data  3D geometry
A. Shademan. CMPUT 412, Vision-based motion control of robots

5. Where is the camera located?
Eye-to-Hand
e.g.,hand/eye coordination
Eye-in-Hand
A. Shademan. CMPUT 412, Vision-based motion control of robots

6. Visual Servo Control law
Position-Based:
Robust and real-time pose estimation + robot’s world-space (Cartesian) controller
Image-Based:
Desired image features seen from camera
Control law entirely based on image features
A. Shademan. CMPUT 412, Vision-based motion control of robots

7. Position-Based Desired pose Estimated pose
A. Shademan. CMPUT 412, Vision-based motion control of robots

8. Image-Based Desired Image feature Extracted image feature
A. Shademan. CMPUT 412, Vision-based motion control of robots

9. Visual-motor Equationx1 x2 x3 x4
q=[q1 … q6]
Visual-Motor Equation
This Jacobian is important
for motion control.
A. Shademan. CMPUT 412, Vision-based motion control of robots

10. Visual-motor Jacobian
Image space
velocity
Joint space
velocity A B
A. Shademan. CMPUT 412, Vision-based motion control of robots

11. Image-Based Control Law
Measure the error in image space
Calculate/Estimate the inverse Jacobian
Update new joint values
A. Shademan. CMPUT 412, Vision-based motion control of robots

12. Image-Based Control Law
Desired
Image feature
Extracted
image feature
A. Shademan. CMPUT 412, Vision-based motion control of robots

13. Jacobian calculation
Analytic form available if model is known. Known model  Calibrated
Must be estimated if model is not known
Unknown model  Uncalibrated
A. Shademan. CMPUT 412, Vision-based motion control of robots

14. Image Jacobian (calibrated)
Analytic form depends on depth estimates.
Camera
Velocity
Camera/Robot transform required.
No flexibility.
A. Shademan. CMPUT 412, Vision-based motion control of robots

15. Image Jacobian (uncalibrated)
A popular local estimator:
Recursive secant method (Broyden update):
A. Shademan. CMPUT 412, Vision-based motion control of robots

16. Calibrated vs. Uncalibrated
Model derived analytically
Global asymptotic stability 
Optimal planning is
possible 
A lot of prior knowledge on the model 
Relaxed model assumptions
Traditionally:
Local methods
No global planning 
Difficult to show asymptotic stability condition is ensured 
The problem of traditional methods is the locality.
Global Model Estimation (Research result)
Optimal trajectory planning 
Global stability guarantee 
Kinematics
A. Shademan. CMPUT 412, Vision-based motion control of robots

17. Synopsis of Global Visual Servoing
Model Estimation (Uncalibrated)
Visual-Motor Kinematics Model
Global Model
Extending Linear Estimation (Visual-Motor Jacobian) to Nonlinear Estimation
Our contributions:
K-NN Regression-Based Estimation
Locally Least Squares Estimation
A. Shademan. CMPUT 412, Vision-based motion control of robots

18. Local vs. Global 1st Rank Broyden update: Jägersand et al. ’97
Key idea: using only the previous estimation to estimate the Jacobian
RLS with forgetting factor Hosoda and Asada ’94
1st Rank Broyden update: Jägersand et al. ’97
Exploratory motion: Sutanto et al. ‘98
Quasi-Newton Jacobian estimation of moving object: Piepmeier et al. ‘04
Key idea: using all of the interaction history to estimate the Jacobian
Globally-Stable controller design
Optimal path planning
Local methods don’t!
RLS with forgetting factor, 1st rank broyden, exploratory motion, quasi-Newton Jacobian estimation of a moving object
A. Shademan. CMPUT 412, Vision-based motion control of robots

19. K-NN Regression-based Method? q1 q2 x1 q2
3 NN q1
A. Shademan. CMPUT 412, Vision-based motion control of robots

20. Locally Least Squares Method? q1 q2 x1
(X,q)
KNN(q)
A. Shademan. CMPUT 412, Vision-based motion control of robots

21. Experimental Setup Puma 560 Eye-to-hand configuration Stereo vision
Features: projection of the end-effector’s position on image planes (4-dim)
3 DOF for control
A. Shademan. CMPUT 412, Vision-based motion control of robots

22. Measuring the Estimation Error
Hidden?
A. Shademan. CMPUT 412, Vision-based motion control of robots

23. Global Estimation Error
Local estimation and KNN have the same order, but the variance of KNN method is much less.
A. Shademan. CMPUT 412, Vision-based motion control of robots

24. Noise on Estimation Quality
KNN
LLS
With increasing noise level, the error decreases
A. Shademan. CMPUT 412, Vision-based motion control of robots

25. Effect of Number of Neighbors
A. Shademan. CMPUT 412, Vision-based motion control of robots

26. Conclusions
Presented two global methods to learn the visual-motor function
LLS (global) works better than the KNN (global) and local updates.
KNN suffers from the bias in local estimations
Noise helps system identification
A. Shademan. CMPUT 412, Vision-based motion control of robots

27. Eye-in-Hand Simulator
A. Shademan. CMPUT 412, Vision-based motion control of robots

28. Eye-in-Hand Simulator
A. Shademan. CMPUT 412, Vision-based motion control of robots

29. Eye-in-Hand Simulator
A. Shademan. CMPUT 412, Vision-based motion control of robots

30. Eye-in-Hand Simulator
A. Shademan. CMPUT 412, Vision-based motion control of robots

31. Mean-Squared-Error
A. Shademan. CMPUT 412, Vision-based motion control of robots

32. Task Errors
A. Shademan. CMPUT 412, Vision-based motion control of robots

33. Questions?
A. Shademan. CMPUT 412, Vision-based motion control of robots

34. Position-Based Robust and real-time relative pose estimation
Extended Kalman Filter to solve the nonlinear relative pose equations.
Cons:
EKF is not the optimal estimator.
Performance and the convergence of pose estimates are highly sensitive to EKF parameters.
A. Shademan. CMPUT 412, Vision-based motion control of robots

35. 2D-3D nonlinear point correspondences
Overview of PBVS
2D-3D nonlinear point correspondences
What kind of nonlinearity?
IEKF
T. Lefebvre et al. “Kalman Filters for Nonlinear Systems: A Comparison of Performance,” Intl. J. of Control, vol. 77, no. 7, pp , May 2004.
A. Shademan. CMPUT 412, Vision-based motion control of robots

36. Measurement equation is nonlinear and must be linearized.
EKF Pose Estimation
yaw
pitch
roll
State variable
Process noise
Measurement noise
Measurement equation is nonlinear and must be linearized.
A. Shademan. CMPUT 412, Vision-based motion control of robots

37. Visual-Servoing Based on the Estimated Global Model
A. Shademan. CMPUT 412, Vision-based motion control of robots

38. Control Based on Local Models
See Spong etc book.
See if this should be left out as “hidden”
A. Shademan. CMPUT 412, Vision-based motion control of robots

39. Estimation for Local Methods
We need to estimate the Jacobian which is equal to minimizing the following problem. In fact we want to fit a plan to the local neighbourhood of the current point q. In discrete form, … This is not how we estimate the Jacobian in practice, rather we estimate the Jacobian locally, that is, at each point we fit a plane to the nonlinear model. For example, by Broyden’s first order method, we estimate the Jacobian as, or using the RLS with forgetting factor (see the paper)
In practice: Broyden 1st-rank estimation, RLS with forgetting factor, etc.
A. Shademan. CMPUT 412, Vision-based motion control of robots

40. A. Shademan. CMPUT 412, Vision-based motion control of robots