1 Computer Vision and Robotics Research Group Dept. of Computing Science, University of Alberta Page 1 Towards Practical Visual Servoing in Robotics R. Tatsambon Fomena

2 Manus ARM, iARM  Exact Dynamics Joysticks and keypads common user interfaces Image from Example of a fully integrated system

3 Click-grasp application with the intelligent Manus Arm Examples of a fully integrated system

4 1. User selects object The position of the object is computed using stereo vision from the shoulder of the camera 2. Robot arm moves to that position expressed in its base frame 3. Having the object in view the robot arm computes a more precise target and adjust orientation 4. Using left gripper camera, robot searches database for best object match. 5. Using the object template the robot arm moves to align the feature points. 6. Once aligned gripper moves forward and closes its gripper. 7. Robot returns object to user. (Tsui et al., JABB, 2011) Visual servoing: Example of a fully integrated system

5 Joystick control.3 Control Modes: 1. Move robot's hand in three dimensional space, while maintain the orientation of the hand. 2. User can modify orientation of the hand, but keeping hand centered at the same point in space. 3. User can grasp and release of the hand using either two or three fingers. Kinova Robotics aims also for a similar system Image from

6 Page 6 Visual servoing: The control concept HRI Specification of Goal S* World ACTION PERCEPTION Robot + Camera(s) S* S + - perception for action (Espiau et al., TRA, 92) (Hutchinson et al., TRA, 96)

7 Page 7 Visual servoing: Why visual sensing?  How to control the position of the end-effector of a robot with respect to an object of unknown location in the robot base frame?  How to track a moving target? A visual sensor provides relative position information

8 Page 8 Visual servoing: How can you use visual data in control?  Look then move  Visual feedback control loop ACTION PERCEPTION Robot + Camera(s) S* S - + ACTION PERCEPTION Robot + Camera(s) S-S*

9 Page 9 Quiz What are the advantages of closed-loop control over open loop control approach?

10 Page 10 Visual servoing: Ingredients for a fully integrated system  HRI  Visual tracking method  Motion control algorithm HRI Specification of Goal S* ACTION PERCEPTION Robot + Camera(s) S* S

11 Page 11 Visual servoing: Visual tracking  Crucial as it provides the necessary visual feedback coordinates of image points or lines  Should give reliable and accurate target position in the image Camshift color tracker provides 2D (x,y) coordinates of the tracked objects PERCEPTION Current image Tracker searches for the end-effector S S=(x,y) Selection of the set of Measurements to use for control

12 Visual Tracking Applications: Watching a moving target  Camera + computer can determine how things move in an scene over time. Uses:  Security: e.g. monitoring people moving in a subway station or store  Measurement: Speed, alert on colliding trajectories etc. Page 12

13 Visual Tracking Applications: Human-Computer Interfaces  Camera + computer tracks motions of human user and interprets this in an on-line interaction.  Can interact with menus, buttons and e.g. drawing programs using hand movements as mouse movements, and gestures as clicking  Furthermore, can interpret physical interactions Page 13

14 Visual Tracking Applications: Human-Machine Interfaces  Camera + computer tracks motions of human user, interprets this and machine/robot carries out task.  Remote manipulation  Service robotics for the handicapped and elderly Page 14

15 Page 15 Visual servoing: Example of visual tracking Registration based Tracking Nearest Neighbor tracker N vs Efficient Second Order Minimization

16 Page 16 Visual servoing: Motion control algorithm  3 possible control methods depending on the selection of S: 2D, 3D, and 2 ½ D (Corke, PhD, 94) High bandwidth requires precise calibration: camera and robot-camera 3D VS 2D VS 2 ½ D VS

17 Page 17 Visual servoing: Motion control algorithm  Key element is the model of the system 3D VS 2D VS 2 ½ D VS Robustness to image noise, calibration errors Suitable for unstructured environments (Corke, PhD, 94) 2D VS2 ½ D VS3D VS Abstraction for control

18 3D Visual servoing Page 18 How to sense position and orientation of an object? (Wilson et al., TRA, 1996)

19 2-1/2 D = Homography-based Visual servoing Page 19 Euclidean Homography? (Malis et al., TRA, 1999)

20 2D Visual servoing Page 20 Example of 2D features? (Espiau et al., TRA, 1992) (Jagersand et al., ICRA, 1997)

21 Page 21 Quiz What are the pro and cons of each approach? 1-a) 2D 1-b) 3D 1-c) 2 ½ D

22 Page 22 Visual servoing: Motion control algorithm  Key element is the model of the system: how does the image measurements S change with respect to changes in robot configuration q? can be seen as a sensitivity matrix

23 Page 23 Visual servoing: Motion control algorithm  How to obtain ? 1) Machine learning technique Estimation using numerical methods, for example Broyen 2) Model-based approach Analytical expression using the robot and the camera projection model Example S=(x,y) How to derive the Jacobian or interaction matrix L? (Jagersand et al., ICRA, 1997)

24 Page 24 Visual servoing: Motion stability  How to move the robot knowing e = S-S* and ?  Classical approach: the control law imposes an exponential decay of the error Classical control

25 Page 25 Visual servoing: Motion control algorithm :=VisualTracker(InitImage) Init = Init While ( > T ) { CurrentImage := GrabImage(camera) := VisualTracker(CurrentImage) Compute = Estimate Compute Change robot configuration with }

26 Page 26 Visual servoing: HRI  Important for task specification point to point alignment for gross motions points to line alignment for fine motions  Should be easy and intuitive  Is user dependent

27 Page 27 (Hager, TRA, 1997)

28 Page 28 (Kragic and Christensen, 2002) How does the error function looks like?

29 Page 29 (Kragic and Christensen, 2002) How does the error function looks like?

30 Page 30 (Kragic and Christensen, 2002) How does the error function looks like?

31 Visual Servoing: HRI-point to point task error  Point to Point task “error”: Why 16 elements? Page 31

32 Visual Servoing: HRI-point to line task error  Point to Line Line: Note: y homogeneous coord. Page 32

33 Visual Servoing: HRI-parallel composition example E ( y ) = wrench y - y y (y  y ) (plus e.p. checks) Page 33

34 Page 34 (Li, PhD thesis, 2013) Maintaining visibility

35 Visual Servoing: HRI with virtual visual fixtures Motivation  Virtual fixtures can be used for motion constraints Potential Applications  Improvements on vision-based power lines or pipelines inspection Flying over a power line or pipeline by keeping a constant yaw angle relative to the line (line tracking from the top) Hovering over a power pole and moving towards the top of a power pole for a closer inspection Page 35

36 Where does virtual fixtures can be useful?  Robot Assistant method for microsurgery (steady hand eye robot) “Here the extreme challenge of physical scale accentuate the need for dexterity enhancement, but the unstructured nature of the task dictates that the human be directly “in the loop”” Page 36 - EyeRobot1

37 Where does virtual fixtures can be useful?  How to assist the surgeon? Cooperative control with the robot Incorporate virtual fixture to help protect the patient, and eliminate hand’s tremors of the surgeon during surgery Page 37 -

38 Where does virtual fixtures can be useful?  Central retinal vein occlusion, solution= retinal vein cannulation  Free hand vein cannulation  Robot assisted vein cannulation  What to prove? “robot can increase success rate of cannulation and increase the time the micropipette is maintained in the retinal vein during infusion” Page 38

39 Virtual fixture: example  JHU for VRCM (Virtual Remote Center of Motion) Page 39

40 What is a virtual fixture?  “Like a real fixture, provides surface that confines and/or guides a motion” (Hager, IROS, 2002) Its role is typically to enhance physical dexterity Page 40 (Bettini et al., TRO, 2004)

41 What is a virtual fixture?  “Software helper control routine” (A. Hernandez Herdocia, Master thesis, 2012) Line constraint, plane constraint Page 41

42 What is virtual visual fixture? Vision-based motion constrains  Geometric virtual linkage between the sensor and the target, for 1 camera visual servoing (Chaumette et al., WS, 1994) Extension of the basic kinematic of contacts  Image-based task specification from 2 cameras visual servoing (Dodds et al., ICRA, 1999) Task geometric constraint defines a virtual fixture Page 42

43 What is virtual visual fixture? Vision-based motion constraints  Geometric virtual linkage (Chaumette et al., WS, 1994) Page 43

44 What is virtual visual fixture? Vision-based motion constraints  Image-based task specification (Dodds et al., ICRA, 1999) Page 44

45 Mathematical insight of virtual fixture  As a control law – filtered motion in a preferred direction  As a geometric constraints – virtual linkage  As condition for observer design - persistency of excitation Page 45 Prove it? (Hager, IROS, 1997)

46 Mathematical insight of virtual visual fixture Take home message: Kernel of the Jacobian or interaction matrix in visual servoing Page 46 (Tatsambon, PhD, 2008)

47 Summary: where are the next steps to move forward?  The conclusion is clear So far only a handful existing fully integrated and tested visual servoing system –Mechatronics & theoretical developments than actual practical software development –Our natural environment is complex: Hard to design an adequate representation for robots navigation  It is time to free visual servoing from its restrictions to solving real world problems Tracking issue: reliability and robustness (light variation, occlusions, …) HRI problem: Image-based task specification, new sensing modalities should be exploited Page 47

48 Short term research goal: new virtual visual fixture  Virtual fixtures Line constraint To keep tool on the line Can be done with point-to-line alignment Ellipse constraint To keep the tool on the mapping of a circle Has never been done Page 48

49 Short term research goal: grasping using visual servoing Page 49 Instead of having predefined grasping points in a database of objects Where to grasp?