1. Enchaînement de tâches robotiques Tasks sequencing for sensor-based control Nicolas Mansard Supervised by Francois Chaumette Équipe Lagadic IRISA / INRIA Rennes

2. 2 Context Sensor-based control Motion planning Control architecture initial desired Execution controller Path deform. Traj. planning ? ? Path deformation Dynamic planning Motors Sensors Motion planning Execution level Symbolic high-level controller Data extraction Motion prediction Complete solution Global convergence Required knowledge Lack of reactivity Accuracy Reactivity Robustness Local convergence Realistic solution Complex software Inherent problems due to the path planning

3. Improve the expressivity of the sensor-based control methods Less planning – More freedom

4. 4 Task sequencing for sensor-based control GLOBAL TASK CONSTRAINTS Centering Zoom Perspective Z-rotation Stack of Tasks TASK-LEVEL CONTROLLER add remove swap

5. 5 Task sequencing for sensor-based control Example CenteringZoomPerspectiveZ-rotation Stack of Tasks CONSTRAINTS

6. 6 Task sequencing for sensor-based control CONSTRAINTS Centering Zoom Perspective Z-rotation Stack of Tasks TASK-LEVEL CONTROLLER add remove swap

7. 7 Outline o Credo o Low Level n Stack of tasks n Improvements o High Level n Task-level controller n Applications

8. 8 Sensor 4 1 2 3 4 1 2 3 What is a task? o Robot position: Robot control input: o An error between current and desired sensor values o A reference behavior of the error o The associated Jacobian matrix o Classical control law [Samson91],[Espiau91] Sensor - +

9. 9 Stack of task Principles o Compute the control law from several (sub-) tasks o Ensure a hierarchy n For avoiding any conflicts n is perfectly realized n is realized under the condition is realized n is realized under the condition, … and are realized o Take into account additional constraints n As a lowest-priority task n Using the potential field formalism o Ensure the continuity at task changes

10. 10 z can be used to vary the trajectory (obstacle avoidance) Stack of task Redundancy formalism P We will use z to realize at best the second elementary task [Rosen60],[Liegeois77]

11. 11 o Two recursive equations to stack several tasks o o o Continuity at stack change o e1e1 e i-1 eiei STACK enen Stack of task Priority and continuity [Chiaverini97] [Robea-Egocentre04]

12. 12 o Gradient Projection Method n Potential function : 0 far from the obstacle Maximal when the robot reaches the obstacle n Gradient as a repulsive force n Projection onto the remaining DOFs o Application to joint limits avoidance Stack of task Considering the constraints [Liegeois77],[Marchand98] [Khatib87]

13. 13 Outline o Credo o Low Level n Stack of tasks n Improvements o Directional redundancy o Varying-feature-set tasks o High Level n Task-level controller n Applications

14. 14 Directional redundancy

15. 15 Directional redundancy

16. 16 Directional redundancy Comparison o Classical redundancy o Directional redundancy Additional DOF

17. 17 Varying-feature-set task o Final goal: n Introduce the constraint IN the stack o Joint limits JLmax JLmin

18. 18 Varying-feature-set task + = Positioning Joint limits Control law CONTINU Positioning Joint limits STACK

19. 19 Varying-feature-set task + = Joint limits Positioning Control law DISCONTINUOUS Positioning Joint limits STACK

20. 20 Varying-feature-set task + = Joint limits Positioning Control law Positioning Joint limits STACK OSCILLATIONS

21. 21 Varying-feature-set task o Definition of a task: n Error vector n Activation matrix o Definition of a new inverse operator n Inverse of J activated by H: n Associated projector: Secondary task alone Joint limits + secondary task Joint limits alone

22. 22 Low level o Stack of tasks n Control from several tasks + constraints n Continuity at stack change n Define some functionality for high-level control o Directional redundancy n Enlargement of the main-task free space o Varying-feature-set task n Enlargement of the expressivity LOCAL CONVERGENCE

23. 23 Task sequencing for sensor-based control CONSTRAINTS Centering Zoom Perspective Z-rotation Stack of Tasks TASK-LEVEL CONTROLLER add remove swap

24. 24 Outline o Credo o Low Level n Stack of tasks n Improvements o High Level n Task-level controller n Applications

25. 25 Higher Level Controllers Stack controller Joint-limit controller Obstacle controller Occlusion controller … COLLISION PREDICTED? Remove a task e1e1 e i-1 eiei STACK enen constraints

26. 26 Higher Level Controllers Examples REMOVE JOINT-LIMIT COLLISION? 

27. 27 Higher Level Controllers Push-back controller Joint-limit controller Obstacle controller Occlusion controller … COLLISION PREDICTED? Push-back controller OBSTACLE AVOIDED ? Add a removed task Remove a task e1e1 e i-1 eiei STACK enen constraints

28. 28 Higher Level Controllers Examples PUSH-BACK JOINT-LIMIT AVOIDED?  initial final

29. 29 Higher Level Controllers Look-ahead controller Joint-limit controller Obstacle controller Occlusion controller … COLLISION PREDICTED? Push-back controller OBSTACLE AVOIDED? Look-ahead controller LOCAL MINMA? DEAD LOCK? Add a specific task Add a removed task Remove a task e1e1 e i-1 eiei STACK enen constraints

30. 30 Higher Level Controllers Examples Desired position REMOVE PUSHBACK RECONFIGURE PUSHBACK

31. 31 o Global task n Positioning (6 DOF) o Define a set of tasks n Tasks to be realized o Centering o Zoom o Rotation o Perspective n Constraints to be respected o Joint limits o Occlusion o Obstacles o High level controller n Constraint controller n Push-back controller n Reconfiguration controller Applications Afma6 manipulator robot

32. 32 CenteringZoomPerspectiveZ-rotation Stack of Tasks CONSTRAINTS

33. 33 CenteringZoomPerspectiveZ-rotation Stack of Tasks CONSTRAINTS

34. 34 Applications Non-holonomic robot

35. 35 Applications Non-holonomic robot Depending from controller state +1,-1 - 1,+1 +1,+1 -1,-1 [Promete94]

36. 36 o Global task n Positioning (no obstacles) o Define a set of tasks n Tasks to be realized o Positioning: second order approximation o Positioning toward a virtual position o High level controller n Virtual goal controller Applications Non holonomic robot

37. 37 Applications Non-holonomic robot

38. 38 Applications Humanoid robot o Application of the previous work n Catching a ball while walking o Define a set of tasks

39. 39 Applications Humanoid robot o Global task n Catching a ball while walking o Define a set of tasks n Tasks to be realized o Centering o Grasping o Walking n Constraints to be respected o Joint limits o Arm manipulability o Chest (immobile)

40. 40 Applications Humanoid robot Task centering: Input dim=2 Output dim = 10 Rank = 2 o Application of the previous work n Catching a ball while walking o Define a set of tasks n Tasks to be realized o Centering o Grasping o Walking n Constraints to be respected o Joint limits o Arm manipulability o Chest (immobile)

41. 41 Applications Humanoid robot Task grasping: Input dim=3 Output dim = 14 Rank = 3 o Application of the previous work n Catching a ball while walking o Define a set of tasks n Tasks to be realized o Centering o Grasping o Walking n Constraints to be respected o Joint limits o Arm manipulability o Chest (immobile)

42. 42 Applications Humanoid robot Subtask distance: Input dim=1 Output dim = 14 Rank = 1 Subtask orientation: Input dim=2 Output dim = 14 Rank = 2 divide Task grasping: Input dim=3 Output dim = 14 Rank = 3 o Application of the previous work n Catching a ball while walking o Define a set of tasks n Tasks to be realized o Centering o Grasping o Walking n Constraints to be respected o Joint limits o Arm manipulability o Chest (immobile)

43. 43 Applications Humanoid robot Task walking: Input dim=12 Output dim = 12 Rank = 12 o Application of the previous work n Catching a ball while walking o Define a set of tasks n Tasks to be realized o Centering o Grasping o Walking n Constraints to be respected o Joint limits o Arm manipulability o Chest (immobile)

44. 44 Applications Humanoid robot Constraint Joint Limits: Input dim=28 → 14 Output dim = 14 Rank = 0 to 14 o Application of the previous work n Catching a ball while walking o Define a set of tasks n Tasks to be realized o Centering o Grasping o Walking n Constraints to be respected o Joint limits o Arm manipulability o Chest (immobile)

45. 45 Applications Humanoid robot Constraint Manipulability: Input dim = 6 Output dim = 6 Rank = 1 o Application of the previous work n Catching a ball while walking o Define a set of tasks n Tasks to be realized o Centering o Grasping o Walking n Constraints to be respected o Joint limits o Arm manipulability o Chest (immobile)

46. 46 Applications Humanoid robot Constraint Chest: Input dim = 2 Output dim = 2 Rank = 2 o Application of the previous work n Catching a ball while walking o Define a set of tasks n Tasks to be realized o Centering o Grasping o Walking n Constraints to be respected o Joint limits o Arm manipulability o Chest (immobile)

47. 47 Applications Humanoid robot o Application of the previous work n Catching a ball while walking o Define a set of tasks n Tasks to be realized o Centering o Grasping o Walking n Constraints to be respected o Joint limits o Arm manipulability o Chest (immobile) o High level controller n Joint-limits control n Reachability control

48. 48

49. 49 Conclusion o Low level n Stack of task: a new tool for task-level control n Directional redundancy n Varying-feature-set task o High level n Task-level controllers to ensure multiple constraints during the displacement n Application for humanoid robotic n An alternative to path planning?

50. 50 Perspectives o Multiple short-term perspective n Application for multi sensors n Integration of the varying-feature-set control laws o Non-holonomic robots control o Generalization of the method for humanoid robot o Integration of the path planning solutions o Learning by imitation

51. Merci à … Many thanks to …

52. Appendix 1 State of the Art

53. 53 Appendix 1 State of art o Sensor-based control + constraints n Singularity [Nelson95], joint limits [Chan95] n Redundancy formalism [Liegeois77],[Hanafusa84], [Siciliano91], [Samson91] o Switching control law n Visual servoing 2D / 3D [Gans03] n Positioning / Visibility [Chesi03] n Positioning / Obstacle avoidance [Soueres03, Folio06] o Ad-hoc sequencing n [Peterson01] n [Chiaverini05] o High-level control n Behavior control [Brooks87],[Mataharic01] n Humanoid task-level control

54. Appendix 2 Calibration-robust Stack of Tasks

55. 55 1. Centering 2. Rotation Appendix 2 Calibration-robust Stack of Tasks

56. 56 robust not robust Centering not respected 1. Centering 2. Rotation Appendix 2 Calibration-robust Stack of Tasks

57. 57 o The redundancy formalism is not robust to jacobian misestimation o We have an initial estimation of J … n Analytical solution o … and then we estimate J on-line n Learning n Correction of the perturbation [Hosoda94],[Jagersand96] [Piepmeier99] [Lapreste04] Appendix 2 Calibration-robust Stack of Tasks

58. 58 o Two cameras on mobile Pan-Tilt n Fixed head for the experiments o Eye-To-Hand servoing n Markers on the hand o Flexible robot n No real zero-position n Difficult to calibrate n Approximation of the Jacobian Appendix 2 Calibration-robust Stack of Tasks

59. 59 Analytical matrices On-line estimated matrices Appendix 2 Calibration-robust Stack of Tasks

60. Appendix 3 Pseudo-linear Control for Non-holonomic Robots

61. 61 + Appendix 3 Pseudo-linear Control for Non-holonomic Robots o o BUT ? =

62. 62 o o where o o In our case + = [Hettlich98] Appendix 3 Pseudo-linear Control for Non-holonomic Robots