1. Project Hands.DVI - KickOff Meeting Italian Institute of Technology, IIT, Genova, 24.01.2011 HANDS.DVI Centro di Ricerca “E. Piaggio” – University of Pisa

2.  The project HANDS.DVI will contribute to the development of a unified structure for programming and controlling robotic hands based on a number of fundamental primitives, and abstracting, to the extent possible, from the specifics of their kinematics, mechanical construction, sensor equipment  The project will propose a new DeVice-Independent programming and control framework for robotic HANDS  This project builds upon well-founded results on sensorimotor synergies developed in neuroscience and preliminary used in some special robotic applications. The project HANDS.DVI will exploit initial results of the research activity developed in the European Project The Hand Embodied (THE) Introduction to project HANDS.DVI

3.  The project THE is focused on the study of how the embodied characteristics of the human hand and its sensors, the sensorimotor transformations, affect and determine the learning and control strategies we use for such fundamental cognitive functions as exploring, grasping and manipulating  The project hinges about the conceptual structure and the geometry of such enabling constraints, or synergies: correlations in redundant hand mobility (motor synergies), correlations in redundant cutaneous and kinaesthetic receptors readings (multi-cue integration), and overall sensorimotor system synergies  These key ideas aim at advancing the state of the art in artificial systems for robotic manipulation, haptic, and neuroprosthetic interfaces Motivation and objective of project THE

4.  The mapping between the paradigmatic hand and the robotic hands can be obtained as the result of an optimal control strategy. A possible solution is to organize this function as two hierarchical levels:  in the high level stage only a few elements, the synergies, are controlled  in the low level the synergies are mapped to the actual kinematic variables of the robotic hand we want to control Motivation and objective of project HANDS.DVI  The device independent mapping will be mediated by a model of an anthropomorphic robotic hand (paradigmatic hand) able to capture the idea of synergies in human hands. This model will be used to design the new control strategies for both anthropomorphic and non– anthropomorphic robotic hands

5.  Force control of robotic grasping: the role of hand synergies in the optimal choice of grasping forces  Extending the paradigmatic hand model: modelling and control of biomorphic fingers using tendon actuation and rolling contacts  Mapping synergies between the paradigmatic hand and robotic hands with different kinematics: “DLR/HIT Hand II” (DLR) and “THE First Hand” (UNIPI) - direct kinematic mapping of the contact points or definition of an optimal function based on grasping forces  Synergies definition: the Motion Capture system and statistical methods for data analysis Contents of the presentation

6. On the Role of Hand Synergies in the Optimal Choice of Grasping Forces

7. Objective of the present study Investigate analytically and numerically the role that limiting the DoFs of an hand via synergistic underactuation have on:  making possible precision/whole-hand grasps;  grasping force distribution;  grasp optimality evaluated with respect to grasp quality indices;  robustness with respect to physical characteristics (e.g., compliance).

8. Fundamental Equations (object) Equilibrium Equations where: (hand) Congruence Equations (Grasp matrix: maps contact forces to object wrench ) (maps contact forces to joint torques ) (Hand Jacobian: maps joint velocities to fingertip twists ) (maps the EE twist to twists of the object contact points ) (object) (hand)

9. Contact compliance model Penetration of the fingertip point into the object surface j-th limb object Contact force (penalty formulation) By properly juxtaposing the expression for all the contact forces (contact stiffness matrix) (contact compliance matrix)

10. Joint compliance model Joint torque costitutive equation By properly juxtaposing the expression for all the joints (joint stiffness matrix) joint reference positions joint actual positions (joint compliance matrix)

11. Soft Synergy Model  We introduce a synergy vector whose components can be interpreted as “high level knobs” controlling joint coordination patterns;  In the classical synergistic model: the hand configuration is rigidly constrained to a dimensional sub-manifold of the joint configuration manifold. Therefore the virtual variations allowed are Synergy matrix found through Principal Component Analysis (PCA) of recorded grasping experiments [Santello et al. 1998];  In the soft synergy model (proposed) synergistic displacements command joint reference positions The actual hand configuration depends on hand-object interaction (hand adaptability to different grasping conditions). (synergy matrix)

12. Soft-Synergy Concept synergistic displacements (actual DoF) coupling defines the synergy vector on references actual joint position

13. Solution of the Grasping Problem with Synergies For a general grasping system applying a wrench to the object, i.e., the contact force distribution is given by where:  = particular solution (minimizes the strain energy) ( weighted right inverse of ) = active internal forces that can be commanded via synergistic displacements If no underactuation takes place (independent joint control): = passive (preload) internal contact forces

14. Internal Controllable Forces Internal controllable forces directly dependent on synergistic variations Defining it is possible to write also (coefficient of the linear combination) The above form turns out to be very convenient in optimizing the grasping forces. Direct employment of is not efficient since it entails the explicit calculation of. A more efficient algorithm can be employed based on the intersection of subspaces.

15. Qualitative analysis of Hand Embodiment (increases) (a plateau is reached) 1) Assume each has non-null projection onto, then 3) For the underactuated case, increase the no. of synergies engaged one by one (increase the number of columns in ):  For Therefore if is “small” (2, 3), embodiment with an equal number of synergies does not endanger ability to exert the same grasping forces with great control simplification. 2) Consider a grasp for which indep. joint control bears

16. Grasping Force Optimization Problem Given a grasp characterized by  weighted right inverse of the Grasp Matrix ;  basis for the subspace of internal controllable forces by employing a given set of synergies;  basis for the subspace on internal passive (preload) forces by employing a given set of synergies;  External wrench to be resisted Grasping Force Optimization Problem (GFOP) In the tests performed:, [N] Problem feasibility force-closure (normal components)

17. Paradigmatic Hand Model Paradigmatic Hand Model features:  model of the Cyberglove employed in the work of Santello et al. (1998) for which high quality experimental data were recorder and made available to us;  the only “plant” for which postural synergies are rigorously defined;  15 DoFs 4 DoFs [T]: Rot., Abd., Metac., Interph.; 3 DoFs [I, R, L]: Abd., Metac., Prox. Interph.; 2 DoFs [M]: Metac., Prox. Interph.  limbs represented as cylinders;  anthropometric data relative to one subject employed in the experimental tests. [T] [I] [M] [R] [L]

18. Numerical tests – Precision grasp As a representative of precision grasp layout we consider the grasp of a cherry. Grasp analysis (independent joint control):  3 contact points: [T] Interph., [I] & [M] Prox. Interph.;  contact constraints: PCWF;  ;  ; (internal)  ; (internal, controllable)  ; (internal, passive)  force-closure: yes! Questions: What is the effect of hand embodiment for this grasp? Is the grasp still force-closure? For which synergy/synergies? What is the effect of synergies on grasp optimality criteria? active limbs cherry.avi

19. Precision grasp – Contact Stiffness Var.ns  Variation of the cost function (2-norm of contact force vector) as more synergies are engaged into the grasp.  The grasp is force-closure already for the 1-st synergy.  Limited effect of Contact Stiffness variation, units [N/mm].

20.  Variation of the cost function (2-norm of contact force vector) as more synergies are engaged into the grasp.  The grasp is force-closure already for the 1-st synergy.  Limited effect of Joint Stiffness variation, units [N/mm]. Precision grasp – Joint Stiffness Var.ns

21. Numerical tests – Power grasp As a representative of power grasp layout we consider the grasp of an ashtray. Grasp analysis (independent joint control):  11 contact points: [T] 3 cps; [I], [M], [R], [L] 2cps each;  contact constraints: PCWF;  ;  ; (internal)  ; (internal, controllable)  ; (internal, passive)  force-closure: yes! Questions: What is the effect of hand embodiment for this grasp? Is the grasp still force-closure? For which synergy/synergies? What is the effect of synergies on grasp optimality criteria? ashtray.avi

22. Power grasp – Contact Stiffness Var.ns  Variation of the cost function (2-norm of contact force vector) as more synergies are engaged into the grasp.  The grasp is not always force-closure for the 1-st synergy.  Some effects of Contact Stiffness variation, units [N/mm].

23. Power grasp – Joint Stiffness Var.ns  Variation of the cost function (2-norm of contact force vector) as more synergies are engaged into the grasp.  The grasp is not always force-closure for the 1-st synergy.  Some effects of Joint Stiffness variation, units [N/mm].

24. Conclusions Theoretical results:  a way has been proposed to solve the force decomposition and optimization problem for hands with synergies;  a way to map synergies from the kinematic domain (where have been observed) to the force domain (soft-synergies). Numerical results:  force-closure properties of grasps strongly depends on which synergies are used to control the hand;  if the first few synergies (PCs) are not actively controlled, force-closure can be obtained only in many more DoFs;  no improvement beyond the first three synergies for precision grasp, continuous but small improvements in the whole-hand grasp case;  results are consistently robust with respect to different values of the stiffness parameters (uncertainty in their knowledge/control).

25. Modelling and Control of a Tendon Actuated Biomorphic Hand

26. Biomorphic reference scheme

27. Contact points move over the surfaces [M. Gabiccini, A. Bicchi, GRSSP Workshop, 2008] Object Palm Contact frames Convective coords Finger Object/finger contact

28. Total wrench on each limb Resultant wrench on the -th body: where : is the wrench applied by the tendons expressed in body frame is the wrench applied by the contact forces expressed in body frame

29. The vector must be chosen by means of an optimization routine to fulfill all the constraints. This can be formulated as a convex optimization problem. Solution of the problem We search a vector such that the system is in equilibrium and all the friction/tendon constraints are satisfied (tendon tension must be positive): If we consider the complete system, we obtain: the system must be in equilibrium under an external wrench, therefore: where is the particular solution; is the homogeneous solution that not affect the overall system eq.

30. Control scheme TRAJECTORY GENERATION INITIAL CONDITIONS COMPUTED TORQUE SYSTEM DYNAMICS OPTIMIZATION ROUTINE CONTACT KINEMATICS SOLUTION (TENDON TENSIONS)

31. System layout The hand/object system consists of: Three-fingered hand grasping a ball; Each finger is a 4-d.o.f. serial manipulator with unilateral joints; The first joint (MCP) is a 2-d.o.f. hyperboloidic (saddle joint); The 2-nd and 3-rd joints are hinge joints with 1-d.o.f. There are 8 tendons which guaranteed the integrity of the system

32. Task Task for the hand/object system: the object has to track a prescribed trajectory (e.g., G follows ellipse in vert. plane); integrity of the hand/object contacts and integrity of the finger limbs have to be guaranteed by proper co-contraction forces in the tendons; integrity meaning: contact forces within friction cone boundaries; max/min contact force below/above prescribed limits; max/min tendon tensions below/above prescribed limits (e.g., tendons cannot pull)

33. Videos of simulations  Move the Center of Gravity of the ball along an ellipse in the vertical plane Move the Center of Gravity of the ball along an ellipse in the vertical plane  Rotate the ball around a vertical axis Rotate the ball around a vertical axis  Move the Center of Gravity of the ball up and down Move the Center of Gravity of the ball up and down

34. 16 DoFs Robotic Hand Prototype

35. Hand Features  16 DoFs  Five fingered hand  Tendon driven  One rotoidal spring for each joint (only one tendon is needed)  Low cost  Human sized  Human-like movements  Servo controlled Kinematic model of the “First Hand” is very similar to the paradigmatic hand model

36. Hand Schematics PHALANGES Index, Middle, Ring and Little ABDUCTIONS PALM  Proximal phalanges are independently controlled w.r.t intermediate and distal phalanges  Intermediate and distal phalanges are dependently controlled THUMB  Phalanges are independently controlled  Middle finger doesn’t have the abduction DOF  Thumb and Little abductions are independently controlled  Two rotoidal joints to close palm on itself

37. Grasping Trials - 1

38. Grasping Trials - 2

39. Hand Synergies First SynergySecond Synergy first.mp4second.mp4

40. Hand Posture Tracking: Motion Capture System

41.  Motion capture system: active leds, impulse cameras, server computer, led controllers, led base station  Easy to reconfigure and calibrate  Fast and reliable data transmission (up to 480 Hz)  Possibility to connect several clients and integrate measurements of grasping forces  Problems to overcome:  Optimal marker positioning (phalanges or joints)  Kinematic hand model that best fits experimental data  Observability, model parameters estimation and posture reconstruction using nonlinear filters (EKF)  Statistical methods for data analysis Experimental Setup

42. Statistical Methods  Modelling grasping movements using statistical methods: Hidden Markov Models and Gaussian Mixture Models  Two Objectives:  Task replication using regression techniques: Gaussian Mixture Regression  Tasks classification from continuous movements while grasping different objects  Problem of data dimension reduction (15 dofs for the paradigmatic hand) using linear and nonlinear PCA  Hand posture reconstruction with inaccurate data using a-priori information and its application to a low cost glove (sensory synergies).

43. Thanks for your attention