1. 1 Shared Control for Dexterous Telemanipulation with Haptic Feedback Weston B. Griffin Dissertation Defense Presentation May 1, 2003

2. 2 Telemanipulation First systems developed ~ 1940’s –handling radioactive materials Can provide access to dangerous environments Benefit from natural human abilities slavemasteroperator [The E1 developed by Goertz at Argonne National Lab] environment

3. 3 Telemanipulation Applications include: –underwater salvage –nuclear waste handling –space station repair –minimally invasive surgery [Intuitive Surgical, Canadian Space Agency, Oceaneering International]

4. 4 Telemanipulation Frameworks computer controlled electro-mechanical systems remote controlled robot <> feeding back information OperatorMaster SystemSlave ControllerSlave Manipulator position force extends a person’s sensing and/or manipulation ability to a remote location several different architectures

5. 5 Manipulation Desire to leverage human manipulation skills –immersive hand/finger based system Slave Manipulator position force Master System

6. 6 Movie Remote Control by Andy Shocken –filmed 2002 in our lab –narrated by Mark Cutkosky

7. 7 Issues in Telemanipulation operator may feel remotely present BUT is not getting normal manipulation cues Current telemanipulation limitations –force feedback (limited accuracy and fidelity) –limited tactile display Master system design –difficult task considering complexity of human hands –active area of research Enhance slave controller –by sharing control between operator and slave system - shared control

8. 8 Contributions Development a human-to-robot mapping method –map glove-based hand motions to a planar robot hand Development and implementation of a shared control framework for dexterous telemanipulation –combining operator commands with a semi-autonomous controller Investigation of an experimental telemanipulation system –results demonstrate benefits of shared control and need to choose carefully types of feedback to achieve a real improvement

9. 9 Outline System overview Human-to-robot mapping Shared control framework Experimental investigation Development of dexterous telemanipulation system

10. 10 Improving Telemanipulation Take advantage of the slave controller and local sensor information for improved dexterity –add “low-level” intelligence Why? –can feedback sensor information by other means –robot can intervene in certain situations (fast response) –human and robot can share control for improved performance

11. 11 Shared Control semi-autonomous dexterous manipulation bilateral telemanipulation high level commands & feedback force position commands sensor feedback

12. 12 Shared Control combining operator high level and low level commands with a remote controller for improved manipulation

13. 13 Master System CyberGlove™ instrumented glove –22 bend sensors –calibrated for dexterous manipulation [Turner 2001] CyberGrasp Hand tracker CyberGlove CyberGrasp™ fingertip force feedback –lightweight exo-skeleton –uni-directional force feedback Logitech hand tracker –ultrasonic transducers and sensors –6 d.o.f. position and orientation [CyberGlove and CyberGrasp are products of Immersion Corporation]

14. 14 Slave System Custom built robot hand –two fingers, two d.o.f. per finger –low inertia DC motors –cable capstan drive Fingertip sensors –two-axis force sensors –contact location sensors Robot arm –Adept industrial arm, five d.o.f. –enlarges task workspace

15. 15 Slave Master System Architecture CyberGlove GUI Adept Control Slave Control CyberGrasp QNX Node-to-Node QNX-Node 2 Wrist Tracker Indirect Feedback 1000 Hz200 Hz 50 Hz 7 Hz 63 Hz 1000 Hz / 200 Hz 200 Hz 1000 Hz QNX-Node 1

16. 16 Outline System overview Human-to-robot mapping Shared control framework Experimental investigation Development of dexterous telemanipulation system

17. 17 Human-to-Robot Mapping Robot is non-anthropomorphic, symmetric, and planar –joint-to-joint mapping not possible –very different workspace

18. 18 Human-to-robot Mapping How do you control a non-anthropomorphic robot hand using a human hand and glove? ?

19. 19 Virtual Object Mapping Interpret human fingertip motions to be imparting motions to a virtual object held between the fingers Virtual object parameters are mapped to robot –to produce fingertip positions OR motions of a grasped object Parameters independently modified –to account for kinematic and workspace differences

20. 20 Index Mapped Positions Thumb Mapped Positions Mapped Pinch Point Position Left Finger Boundary Right Finger Boundary -0.25-0.2-0.15-0.1-0.0500.050.10.150.20.25 -0.25 -0.2 -0.15 -0.1 -0.05 0 Virtual Object Mapping Match natural human manipulation motions to corresponding robot hand motions good mapping? –operator can intuitively control robot and utilize robots workspace

21. 21 Outline System overview Human-to-robot mapping Shared control framework Experimental investigation Development of dexterous telemanipulation system

22. 22 Shared Control Michelman and Allen [1994] –sequencing primitives for dexterous hand control joystick control, no provisions for haptic feedback Williams et al. [2002] –NASA’s Robonaut project - robot arm and dexterous hand –force feedback joystick for control reduced task peak forces Hannford et al. [1991] –force feedback joystick controlling robot arm/gripper –improved task completion time and resulted in lower forces

23. 23 Shared Control Next step: using shared control in a dexterous telemanipulation system with fingertip force feedback How? –implement a semi-autonomous controller capable of dexterous manipulation robot has force and tactile sensors and specialized control laws for manipulation

24. 24 Dexterous Manipulation What does it mean to autonomously manipulate an object? –with sensors robot can detect the object and determine proper fingertip forces for: manipulation

25. 25 Dexterous Manipulation What does it mean to autonomously manipulate an object? –with sensors robot can detect the object and determine proper fingertip forces for: manipulation

26. 26 Dexterous Manipulation What does it mean to autonomously manipulate an object? –with sensors robot can detect the object and determine proper fingertip forces for: manipulationgrasp force regulation

27. 27 Object Manipulation Control Utilize the Grasp Transform to determine robot fingertip forces [Mason & Salisbury 1985] Object Impedance Controller Internal Force Decomposition - + + Internal Force Controller Finger Controller Robot Finger Velocity Grasp Transform Ts ZOH Tactile Sensing + + + - + Tactile Based Object Tracking Forward Grasp Transform

28. 28 Object Manipulation Control Controlling internal force Object Impedance Controller Internal Force Decomposition - + + Internal Force Controller Finger Controller Robot Finger Velocity Grasp Transform Ts ZOH Tactile Sensing + + + - + Tactile Based Object Tracking Forward Grasp Transform

29. 29 Object Manipulation Control Controlling object position Object Impedance Controller Internal Force Decomposition - + + Internal Force Controller Finger Controller Robot Finger Velocity Grasp Transform Ts ZOH Tactile Sensing + + + - + Tactile Based Object Tracking Forward Grasp Transform

30. 30 Shared Control Telemanipulation What are the advantages to programming robot for dexterous manipulation? –robot can monitor operator’s object manipulation –if necessary, robot can intervene (take over control of object manipulation) impedance modification, limit motion, prevent release –robot can warn/inform operator of manipulation status through indirect methods using other feedback modalities (visual indicators, audio, or augmented haptic feedback)

31. 31 Shared Control Telemanipulation What are the advantages to letting robot take control over force regulation and/or object manipulation? –operator can focus on behavior of grasped object or tool –master commands are no longer essential to prevent unwanted slip or damaged objects –operator can still override to release or grasp more tightly

32. 32 Shared Control Telemanipulation Shared control implementation issues –as the robot assumes more control concern the operator’s sense of presence will be reduced –we want to keep the operator “in the loop” –preserve operator’s intent –what type of indirect feedback is most effective? –does sharing control improve performance in an immersive fingertip force feedback system? To answer these questions we perform a set a controlled experiments

33. 33 Outline System overview Human-to-robot mapping Shared control framework Experimental investigation Development of dexterous telemanipulation system

34. 34 Previous Experimental Studies force feedback evaluation –Turner et al. 2000: block stacking and knob turning force feedback with CyberGrasp not always a benefit –Howe & Kontarinis 1992: fragile peg insertion task audio buzzer sounded if grasp force excessive operators were not able to reduce force shared control evaluation –Hannaford et al. 1991: peg insertion task operator’s controlled position, shared orientation control reduction in task completion time and insertion forces

35. 35 Experimental Hypothesis Addition of a dexterous shared control framework will increase an operator’s ability to handle objects delicately and securely compared to direct telemanipulation

36. 36 Experiment Description Motivating scenario: recovering an ancient Greek vase on the sea floor “fragile object handling” - user’s asked to carry an object with minimal force but without dropping the object

37. 37 Experimental Task

38. 38 Experiment Description To assist operator in fragile object handling task the robot computes the minimum grasp force required robot can monitor and warn the operator OR robot can intervene and regulate grasp force If operator’s desired (commanded) force is too low to prevent object dropping

39. 39 Shared Controlled Task Operator maintains manipulation control

40. 40 Shared Controlled Task Operator maintains manipulation control Robot and operator share control over internal force –robot monitors excessive force

41. 41 Shared Controlled Task Operator maintains manipulation control Robot and operator share control over internal force –robot monitors excessive force –robot can apply minimum internal force required to prevent slip

42. 42 Sharing Control in Fragile Task Target window with intervention can be wider: desired force can drop below f int,min without adverse effects In theory, it is possible to always do better without intervention

43. 43 Question that arise... Does warning the operator of a possible failure help? Does task performance improve with robot intervention? If robot intervenes, is it necessary to inform operator? Is it helpful to feed back information of impending state changes (such as object release)? With haptic feedback in a force control task, what forces should be fed back?

44. 44 Case Effects Audio Alarms - when operator’s desired force is too high or too low Force Feedback: actual vs. commanded - during robot intervention, forces to operator’s fingertip are reduced (reduced force feedback) Robot Intervention - robot assumes control when operator’s desired force falls below a threshold (safe minimum internal force) Visual Indicator (fingertip LEDs) - to inform the operator of robot intervention

45. 45 Experiment Cases

46. 46 Case Effects

47. 47 Experimental Procedure Diverse set of subjects –11 subjects total –8 males and 3 females Two sessions –first - calibration and training –second - four trials for each case Case order randomized –reduce possible learning and fatigue effects

48. 48 Evaluating Performance Objective data analysis –measured internal force applied to object fragile object task - lower is better –task failures (number of drops) –task completion time Subjective data analysis –operator’s expressed preference –operator’s perceived difficulty

49. 49 Typical Subject Data Case 2 Excessive Force Warning (Low Tone) Object Slip Warning (High Tone) Force [N] 0510152025 0 2 4 Measured Internal Force Desired Internal Force 101% of Minimum Internal Force Case 6 Force [N] Time [sec] 0510152025 0 2 4 A B C D E Measured Internal Force Desired Internal Force 110% of Minimum Internal Force Excessive Force Warning (Low Tone) Object Release Warning (High Tone) Robot Intervention Case 1 Force [N] 0510152025 0 2 4 Measured Internal Force Desired Internal Force Minimum Internal Force Time [sec]

50. 50 Data Analysis Measured internal force applied to the object –averages of each subject for each case (trial failures excluded) Boxplot –medians and quartiles –observe trends Is there a significant effect?

51. 51 Statistical Analysis ANOVA - determines the probability that these results (differences in averages) are really due to random variation in data Apply to averaged measured internal force –p = 0.003 (<< 0.05), indicating that there is a difference between the means –but which ones are different Can’t use a simple t-test for multiple comparisons increase probability of false-positive –Dunnett’s method - comparison to a control (Case 1) Cases 4, 6, 7 have statistically different mean than Case 1 a reduction of approximately 15%

52. 52 Task Failures Number of failures that occurred for each case (dropped object) 1234567 0 2 4 6 8 Number of Failures in Each Case - All Subjects Case Number Number of Failures 1234567 0 2 4 6 8 Case Number Number of Failures Number of Failures in Each Case sub1 sub2 sub3 sub4 sub5 sub6 sub7 sub8 sub9 sub10 sub11 N/A Case 5 and 6 had least number of failures Case failures not dominated by one subject

53. 53 Objective Data Analysis Results Robot intervention improves performance –presence and type of direct and indirect feedback had an effect Cases 4, 6, and 7 had lower internal force Case 3 and 5 did not

54. 54 Analysis Results Robot intervention improved performance –presence and type of direct and indirect feedback had an effect Cases 4, 6, and 7 had lower internal force Case 3 and 5 did not –only informing of intervention not adequate Case 7 had most failures indicating alarms were helpful 1234567 0 2 4 6 8 Case Number Number of Failures

55. 55 Analysis Results Robot intervention improved performance –presence and type of direct and indirect feedback had an effect Cases 4, 6, and 7 had lower internal force Case 3 and 5 did not –only informing of intervention not adequate Case 7 had most failures indicating alarms were helpful Reduced force feedback –compare Case 3 to 5 –slight improvement in measured internal force (6%) –fewer failures in Case 5 Cases 4 and 6 show similar results 1234567 0 2 4 6 8 Case Number Number of Failures

56. 56 Task Time May reveal any physical or mental difficulties associated with the various conditions 1234567 10 15 20 25 30 35 Time [Sec] Case Number Average of all subjects for each case shared control did not improve task completion time BUT did not make it worse no obvious trends p = 0.82 (i.e., no difference in means)

57. 57 Results Given objective data analysis performance criteria minimizing internal force but preventing failures: provided best overall performance compared to bilateral case Case 6 - shared control with multi-modal feedback In post experiment surveys, subjects also generally ranked Case 6 highest in preference and ease-of-use

58. 58 Conclusions Answering our hypothesis –Can the addition of a dexterous shared control framework increase an operator’s ability to handle objects delicately and securely compared to direct telemanipulation? YES, shared control gives better performance but you need to: a) let the operator know when the intervention is active b) let the operator know of impending state changes c) feed back force based on commanded force and not actual forces (during intervention)

59. 59 Summary of Contributions Development a human-to-robot mapping method –map glove-based hand motions to a planar robot hand that allows for intuitive hand control Development and implementation of a shared control framework for dexterous telemanipulation –combining operator commands with a semi-autonomous controller Investigation of an experimental telemanipulation system –results demonstrate benefits of shared control and need to choose carefully types of feedback to achieve a real improvement

60. 60 Future Work Do the benefits of shared control extend to other situations and applications? –assembly tasks –e.g., steer-by-wire vehicles Do the same requirements for shared control improvement hold? –informing the operator of intervention –notifying of impending state changes –modifying the forces fed back

61. 61 Acknowledgements Mark Cutkosky Defense Committee Will Provancher The DML Eric (setting the pace in the final days)

62. 62 Shared Control for Dexterous Telemanipulation with Haptic Feedback Weston B. Griffin Dissertation Defense Presentation May 1, 2003

63. 63 Backup Slides

64. 64 One Slide Statistics Review statistical analysis –two competing hypotheses null: cases have no real effect (all the means are the same) alternate: at least one case is different (all means are NOT the same)

65. 65 One Slide Statistics Review if ratio large: at least one mean is different between within statistical analysis –two competing hypotheses null: cases have no real effect (all the means are the same) alternate: at least one case is different (all means are NOT the same) ANOVA - analysis of variance –tests if difference in means of several samples is significant based on variances if ratio small then: all means are the same –how likely is it to have a t.s. as extreme as observed (p-Value) –compare to a significance level (95%) (e.g., reject null if p < 0.05) Case performance quantity

66. 66 Manipulation Desire to leverage human manipulation skills –immersive hand/finger based system Operator Master System Human-to-robot Mapping Slave Manipulator position force

67. 67 Telemanipulation Glove based –Brunner et al. 1994, DLR dexterous robot hand –Li et al. 1996 - NASA DART project –Ambrose et al. 2000, NASA Robonaut project Teleoperation / telemanipulation –Lawn and Hannaford 1993 –Lawrence et al. 1993 –Daniel and McAree 1998 –Sherman et al. 2000 –Speich and Goldfarb 2002

68. 68 FhFh Control Architectures general four-channel one d.o.f. framework C1 C4 C3 C2 Z m -1 CmCm Z s -1 CsCs ZeZe ZhZh Fh*Fh* - - - - - - Human Operator Master System Comm. Link Slave System Environ- ment Fe*Fe* FeFe VeVe VhVh FhFh VmVm + ++ + + + [Lawrence 1993] VeVe FeFe

69. 69 Mapping Background anthropomorphic –linear joint-to-joint [Kyriakopoulos et. al 1997] –fingertip position mapping scaling [Fisher et a. 1998] semi-anthropomorphic –pose matching [Pao and Speeter 1989] joint angle transformation matrix –fingertip position mapping [Speeter 1992, Rholing et al. 1993] forward kinematics, inverse kinematics non-anthropomorphic –greater dissimilarities grammar based functional mapping [Speeter 1992] DLR hand Utah/MIT hand JPL/Salisbury hand Dexter hand

70. 70 Point-to-Point Mapping initial approach –planar projection of fingertip positions –standard planar frame transformation

71. 71 Mapping Implementation computing robot positions –based on planar virtual object compute virtual object parameters –3D size to capture thumb motion –planar reduction

72. 72 Transformation to Robot Frame kinematics v.o. orientation –angular offset v.o. midpoint –frame transformation must modify and scale parameters for desired correspondence workspace v.o. midpoint & size –scaled

73. 73 Parameter Determination based on individual’s recorded hand motion –three simple poses/motions defining –orientation offset –midpoint transformation variables –midpoint scaling –size scaling

74. 74 Mapping Results Virtual Object Mapping improved achievable positions pinch-point can be mapped to any point fundamentally analytical –continuous, smooth, and predictable fingertip-to-fingertip correspondence

75. 75 Modeling Model: averaged percent difference in measured internal force compared to Case 1 1234567 -30 -25 -20 -15 -10 -5 0 5 10 15 Means with Error Bars of Two Standard Deviations Percent Difference (from Case 1 for each subject) in Mean Internal Force Percent Difference, [%] Case Number

76. 76 Model Analysis Look at residuals: 1234567 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 Means with Error Bars of Two Standard Deviations Residuals due to Task Order (Learning and Fatigue Effects) Force, [N] Order