1. MURI Low-Level Control Fabrication High-Level Control How is Compliance used in Locomotion? Berkeley & Stanford: Measurements of Cockroach Locomotion What Compliance Strategies in Human-level Tasks? Harvard & Johns Hopkins: Compliance Learning and Strategies for Unstructured Environments

2. Fabrication MURI Low-Level Control High-Level Control What strategies are used in insect locomotion and what are their implications? Insect locomotion studies (Berkeley Bio) New measurement capabilities (Stanford) What motor control adaptation strategies do people use and how can they be applied to robots? Compliance Learning and Strategies for Unstructured Environments (Harvard & Johns Hopkins) Implications for biomimetic robots (Harvard, Stanford) Guiding questions

3. Yoky Matsuoka and Rob Howe Harvard University MURI High-Level Control Impedance Adaptation in Unstructured Environments

4. MURI High-Level Control An example of manipulation with impedance Why is biology superior to current robots in an unstructured environment? Manipulation with Impedance

5. MURI High-Level Control Identify Impedance Learning Strategy in Human Two big questions: –What is the initial strategy used to cope with unknown/unstructured environments? –After learning, what does the biology pick as the good solution for impedance for a given environment? How can these solutions help robot’s control strategy?

6. MURI High-Level Control Comparison Between Analytical and Biological Solutions We can mathematically derive optimal impedance for a linear world. –Biological system converges to the analytical solution. --- great! –Biological system converges to a different solution. --- what and why: put the biological solution back in the equations and reverse engineer. What about a nonlinear varying world where it is difficult to derive the optimal impedance? –What does the biological system do? Can it be modeled as a solution for robots?

7. Goal: Find the “best” impedance. –For this case, find best K hand. Uncertainty in the world –m ball, k ball, ball (0), and k hand m hand k hand m ball k ball x ball x hand MURI High-Level Control Example: Linear World --- Catching a Ball ball hand m ball m hand k ball k hand

8. Cases: 1. Hand stiffnes (k hand) is too high hand < 0 bounces up 2. Hand stiffness (k hand) is too low x hand > Thresholdbottoms out 3. Hand stiffness (k hand) is just right x ball x hand until switch is pressed k hand 0 infinite 2 3 1 MURI High-Level Control Example: Linear World --- Catching a Ball m hand k hand m ball k ball x ball x hand

9. Solve for x hand (t) and x ball (t) –initial condition ball (0) > 0 x ball (0) = 0 hand (0)= 0 x hand (0) = 0 x hand MURI High-Level Control Example: Linear World --- Catching a Ball m hand k hand m ball k ball x ball

10. MURI High-Level Control Analytical Linear World to Biological Motor Control The example relates task performance to limb impedance and optimal solution. –Other examples: leg impedance, etc. Now measure human strategy…. –“System identification” –Need a new technique

11. MURI High-Level Control Existing System Identification Techniques Time invariant systems --- easy –assume constant m, b, and k over time. –apply external impulse perturbation force. –repeat the same condition and average.

12. MURI High-Level Control Existing System Identification Techniques Time varying systems Cannot apply impulses close to each other. Need multiple impulses to solve for multiple unknowns. –PRBS (Lacquaniti, et al. 1993)

13. Setup Handle Accelerometer Data Acquisition System Processor Human Subject MURI High-Level Control New System Identification Technique to Observe Learning Robot Force Sensor Monitor

14. MURI High-Level Control New System Identification Technique to Observe Learning Very short duration Very clean data from force and acc. sensors F m*a b*v k*x m=F/ab= (F-ma)/vk= (F-ma-bv)/x

15. MURI High-Level Control Testing the New Technique

16. MURI High-Level Control Video of the task here

17. Phantom robot is used as the perturbation/measurement tool. Task: balance the moving ball on paddle. –ball moves at constant speed –dies when the ball falls off the paddle –perturbation applied every second MURI High-Level Control Testing the New Technique

18. MURI High-Level Control Impedance Change with Learning k change over time b change over time m change over time

19. Observe the impedance change within one catch Observe the impedance change between catches ** under development --- pilot studies underway MURI High-Level Control Contact Interaction Task -- Impedance Dependent Task kbkb

20. MURI High-Level Control Current Understanding of the Structure of the Biological Controller From Shadmehr

21. MURI High-Level Control Developed a new impedance identification technique –Based on virtual environment --- extremely versatile –Confirmed ability to measure instantaneous impedance, characterized changes with learning. Impedance Adaptation Conclusions and Future Work

22. MURI High-Level Control Current experiments –Determine human interaction strategies initial impedance learning characteristics final impedance Next experiments –Determine human interaction strategies for nonlinear varying tasks e.g. plastic deformation (running in sand) Impedance Adaptation Conclusions and Future Work

23. Control of Locomotion MURI High- Level Control Local controller (single limb): Control of the limb based on local information: - position and velocity of the limb Is the limb far enough to the back so that I can start the return stroke? - forces acting on the limb Is the supporting load small enough for me to lift the limb? Task controller (all limbs): Coordination with other limbs: - position of the other legs Across species, control of limb based on local information appears similar (Cruse, TINS, 90). Coordination with other limbs appears highly species dependent.

24. MURI High- Level Control Step cycle: generation of power (stance) and return (swing) strokes. Return phase: Move the limb from posterior to anterior position along a desired velocity profile. - Maintain proper impedance to remain stable in case of perturbations After hitting an obstacle, the limb should converge back to the desired path - Adapt impedance to allow for generation of desired behavior in the face of a persistent environment limb is moving in highly viscous fluid, it must adapt its impedance to the characteristics of the environment. Impedance control and adaptation in a position control task Power phase: Maintain contact, maintain height of load, move limb from front to rear. Impedance control and adaptation in a contact/force control task Control of a Limb Based on Local Information

25. MURI High- Level Control Current Limb Local Control Models for Locomotion in Insects Cruse et al (Neural Networks 1998): Stick insect model - limb has little or no inertia - no muscle like actuators - controller output is velocity, feedback sensing via position and linear feedback - no ability to adapt Essentially a kinematic model of a limb only, with little or no dynamics This kind of model tells us little about how to design good controllers

26. MURI High- Level Control General Goals: 1. To understand what impedance strategies a biomechanical controller uses when it moves the limb in a position control task. Apply the results to control of the return phase. 2. To understand impedance strategies of the biomechanical controller in a force control task. Apply the results to the control of the stance phase. Approach: Study the human arm’s impedance adaptive control strategies in both position and force control tasks. Test validity of the strategies on a robotic system. Designing a Single Limb Impedance Controller

27. MURI High- Level Control Designing a Single Limb Impedance Controller Task Division: 1. Impedance control at very short time intervals (<10 msec, preflexes) Yoky Matsuoka and Rob Howe 2. Impedance control at intermediate and long time intervals (<300 msec) Tie Wang and Reza Shadmehr 3. Test and implementation on a robotic system Jay Dev Desai and Rob Howe

28. MURI High- Level Control Challenges: The biomechanics of the human arm are dominated by multiple feedback loops, with various time delays. Impedance measurements are done through imposition of perturbations and measurement of responses. How do time delays affect measures of arm impedance? Humans learn internal models when they learn control. How does a change in the internal model affect measures of arm impedance? Impedance measures require an estimation of where the system would have been if it had not been perturbed. How well can we do this with a non-stationary system like the human arm? Quantifying Impedance Control Strategies of a Biomechanical Controller

29. MURI High-Level Control Current Understanding of the Structure of the Biological Controller Through modulation of input u to the muscles, impedance of the system is changed. The impedance depends on 3 feedback pathways: 1. Near zero-delay mechanical stiffness/viscosity of the muscles (Yoky). 2. Short delay sensory feedback through spinal structures. 3. Long delay sensory feedback through cortical structures (forward model).

30. Are muscle “preflexes” enough? Intact control system High-level sensory feedback loop disrupted MURI High-Level Control

31. MURI High-Level Control Impedance of a biological arm: A definition

32. MURI High- Level Control Estimating Impedance: Theory

33. MURI High- Level Control Estimating Impedance: Requirements

34. MURI High-Level Control 1.0 Estimating Inertial Dynamics of the Arm (Theory)

35. Give a force pulse, use data for up to 14 ms after the pulse to estimate inertial parameters. MURI High- Level Control 1.1 Estimating Inertial Dynamics of the Arm (Methods)

36. MURI High- Level Control 1.2 Estimating Inertial Dynamics of the Arm (Results) 010203040506070 -4 -2 0 2 4 6 Shoulder Torque (Nm) Experiment Model Error 010203040506070 -4 -3 -2 0 1 2 3 4 Elbow Torque (Nm) Index of Array

37. MURI High- Level Control 2.0 Predicting the Un-perturbed Trajectory (Theory) Legend

38. MURI High- Level Control 2.1 Trajectory Prediction (Methods)

39. MURI High- Level Control Prediction error (%) 2.2 Trajectory Prediction (velocity)

40. MURI High- Level Control 2.3 Trajectory Prediction (force) 0204060 -2 0 1 0204060 -2 0 1 0204060 -2 0 1 0204060 -2 0 1 0204060 -2 0 1 0204060 -2 0 1 0204060 -2 0 1 0204060 -2 0 1 Conclusion: Position, velocity, and force can be reasonably well predicted for up to 300 msec after the last sampled data point. Time (10 msec) Force (N)

41. MURI High- Level Control 3.0 Estimating the Effect of  u(t) on Arm Impedance  u is the change in the input to the muscles as a result of our perturbation. While  u cannot be measured directly, we know that it depends on a number of time-delayed, possibly adaptive error feedback systems. Time-delayed error feedback from the spinal reflexes Time-delayed error feedback from the forward model based cortical pathways Input from inverse model based “open-loop” controller

42. MURI High- Level Control In general, a time delay d in a feedback loop reduces apparent viscosity and adds apparent mass to a system. Example: 3.1 Time-delayed effect of  u(t) on Arm Impedance

43. MURI High- Level Control 3.2 Estimating  u(t) in Terms of Measurable Quantities 1. Effect of Spinal Reflexes

44. MURI High- Level Control 3.2 Estimating  u(t) in Terms of Measurable Quantities 2. Effect of the Inverse model

45. MURI High- Level Control 3.2 Estimating  u(t) in Terms of Measurable Quantities 3. Effect of the Forward Model

46. MURI High- Level Control 3.2 Estimating  u(t) in Terms of Measurable Quantities 4. Effect of Adaptation of the Forward Model Predictable changes in impedance should occur as a function of the kind of model that the system learns as it practices movements in an unstructured environment. If learning is via a forward model, the apparent viscosity must decrease as compared to values obtained before the controller had adapted.

47. MURI High- Level Control -0.1-0.0500.050.10.15 -0.05 0 0.05 0.1 0.15 X (m) Y (m) Perturb the movement in different directions Measuring Impedance of the Moving System

48. MURI High- Level Control 00.050.10.150.20.250.30.350.40.450.5 -8 -6 -4 -2 0 2 4 6 8 10 12 Time into the movement (s) Interaction force Inertial dynamics Impedance Controlled force acceleration (m/s 2 ) velocityx25 (m/sec) positionx50 (m) Force (N) Perturb Impedance Early into the Movement

49. MURI High- Level Control 00.050.10.150.20.250.30.350.40.450.5 -8 -6 -4 -2 0 2 4 6 8 10 12 Dynamic force Interaction force Impedance controller’s force Positionx50 (m) Velocity x 25 (m/s) Acceleration (m/s 2 ) Time into the movement (s) Force (N) Impedance in the Middle of the Movement Perturb

50. MURI High- Level Control 00.050.10.150.20.250.30.350.40.450.5 -8 -6 -4 -2 0 2 4 6 8 10 12 Time into the movement (s) velocityx25 (m/sec) acceleration (m/s 2 ) Inertial dynamics Interaction force Perturb Impedance Controlled force positionx50 (m) Force (N) Impedance Near End of Movement

51. MURI High- Level Control 0 -200 Joint Stiffness (N.m/rad) 100 msec Stiffness of the System: Results System Characteristics: - Initially a very stiff system (likely due to intrinsic muscle stiffness). - The system yields as the perturbation persists, with stiffness dropping as the short- and long-loop reflex systems take control.

52. MURI High- Level Control Immediate Plans Position control task: - Impose a force field, measure impedance changes as the system adapts - Do we see evidence for formation of a forward model as indicated by reductions in the system’s viscosity? Force control task: - Measure impedance changes in a task that requires maintaining contact (pushing in order to roll a virtual conveyer belt). - Push too hard, the belt breaks. Push too soft, arm slips on the belt. - Measure impedance changes as the virtual belt’s dynamics changes, requiring adaptation in the controller. - Virtual belt is produced by the robot manipulandum.