National University of Singapore IEEE International Conference on  Robotics and Automation (ICRA 2017) Singapore, May 29 - June 3, 2017 Learning and Control of Robots in Interacting with Unknown Environments Yanan Li PhD BSc, Imperial College London Shuzhi Sam Ge FIEEE, FIFAC, FIET, FSAE, PhD, DIC, BSc National University of Singapore http://robotics.nus.edu.sg

Vision of Social Robots Social robots: Robots that are able to interact and communicate among themselves, with humans, and with the environment, within the social and cultural structure attached to their roles. -International Journal of Social Robotics More robots out of the laboratory into schools, homes and clinic places… Interacting with people in their own space Changing everyday lifestyle

International Journal of Social Robotics Editor in Chief: Shuzhi Sam Ge, NUS Co-Editor in Chief: Oussama Khatib, Stanford University, USA Aims and scope: Provide a platform for presenting findings and latest developments in social robotics, covering relevant advances in engineering, computing, arts and social sciences. Provide an overview of the current state of the social robotics scene. 1 Volume(-s) with 5 issue(-s) per annual subscription IF: 1.407

Outline Introduction Impedance Learning Intention Estimation Zero Force Regulation Conclusion and Future Work

Control: Industry vs Social Accuracy Fast response Fixed environment Interaction Safety Compliant behavior Unknown environment

Intelligent Control Human adjusts their limb impedance and trajectory when catching a ball. It is possible to apply impedance and trajectory learning skills to robot control. http://cnx.org/content/m42183/latest/?collection=col11406/latest http://www.rocketsticks.com/science Human Limb Robot Arm

Intelligent Control Robot will act as a load to the human partner if it plays a follower role. Robot’s trajectory is adapted according to human’s intention. Human-Robot Collaboration Mass-Damping-Stiffness System Central Nervous System: Motion Intention

Intelligent Control Hybrid Position/Force Control: controlling force and position in a non-conflicting way Impedance Control [N. Hogan, 1985]: developing a relationship between force and position Passive Robot Environment Stable [J. E. Colgate, 1988] Passive systems include arbitrary combinations of masses, springs and dampers, linear or nonlinear. require the decomposition of two directions

Problems and Solutions How to impose a target impedance model on a robot arm, in the presence of uncertainties and unknown robot dynamics? Impedance control design (available in the literature) How to find a target impedance model? How to determine the impedance parameters, subject to unknown environment dynamics? Impedance learning Impedance adaptation How to determine the rest position/desired trajectory in the impedance model? Intention Estimation Zero Force Regulation

Outline Introduction Impedance Learning Intention Estimation Zero Force Regulation Conclusion and Future Work

Related Works Damping control Known environment dynamics Passivity assumption Modest performance [S. P. Buerger] Known environment dynamics Stability Optimal performance [R. Johansson; M. Matinfar] Unknown environment dynamics Environment identification and modeling Impedance learning and adaptation Earlier works: simple tasks State-of-the-art: reinforcement learning – complex computation [J. Buchli] Still an open problem

Human Learning Skill A new door/ball is an unknown environment. Human beings learn to adjust their limb impedance iteratively when opening a door/catching a ball. It is possible to apply impedance learning skill to robot control. Source: http://www.chumpysclipart.com/illustration/3943/picture_of_an_angry_man_trying_to_open_a_locked_door

Problem Formulation The dynamics of a robot arm follow an impedance model: Consider that the environment is unknown and dynamically changing, and it is described by Problem: How to determine so that a desired interaction performance is guaranteed? Define a cost function to measure the interaction performance:

Impedance Learning 1. According to gradient-following scheme: 2. The target impedance model: 3. How to obtain ?

Impedance Learning Consider the environment dynamics: Choose states The environment is described as Define The environment dynamics become which is a time-varying system.

Betterment Scheme Theorem [S. Arimoto, 1984]: Consider the linear time-varying system described by The control input is iteratively updated in the following manner The betterment process is convergent in the sense that as Convergence guarantee:

Impedance Learning According to the betterment scheme, we have By employing gradient-following and betterment schemes, impedance parameters are updated as:

Experiment Study ATI 6 axis force/torque sensor Scenario: In each iteration with a period of 18s, the interaction starts at t = 5s and ends at t = 16s. Unknown environment: human hand Initial impedance parameters: Learning rate:

The First Case Cost function: Cost Function Stiffness

The First Case Cost function: Tracking Error Interaction Force

The Second Case Cost function: Cost Function Stiffness

The Second Case Cost function: Tracking Error Interaction Force

Discussion By choosing different cost functions, it is determined that the control objective can be trajectory tracking, integral force tracking or the combination/compromise of these two. The proposed impedance learning guarantees that the defined cost function becomes smaller iteratively, subject to unknown dynamic environments, and the expected interaction performance has been achieved.

Discussion The advantage of the proposed impedance learning over impedance control with fixed impedance parameters lies in: a modest performance can be obtained if a good set of fixed impedance parameters is predefined (when k = 0), and a better performance can be obtained only with variant impedance parameters because the environments are dynamically changing. While impedance parameters have been obtained through iterative learning, how to determine the rest position/desired trajectory in a target impedance model?

Outline Introduction Impedance Learning Intention Estimation Zero Force Regulation Conclusion and Future Work

Scenario: Human-Robot Collaboration Most tasks that are either too complex to automate or too heavy to manipulate manually are impractical and even impossible to be solely taken by either fully automated robots or human beings. Robots and human beings may share the same workspace and they have complementary advantages.

Scenario: Human-Robot Collaboration The robot will act as a load to the human partner if x0 is far away from x. x0 is supposed to be adapted according to human partner’s motion intention. Trajectory adaptation is another human beings’ learning skill which can be applied to robot control.

Related Works Intention estimation Force regulation Motion characteristics of the human limb: minimum jerk model [B. Corteville, 2007] Motion intention state: hidden Markov model [Z. Wang, 2009] Intentional walking direction: Kalman filter [K. Wakita, 2011] Robot and human partner: separately analyzed Force regulation Force regulation under impedance control [K. Lee, 2008] Existing works: constant rest position and only stiffness in the environment (human limb)

Problem Formulation Human limb model [M. M. Rahman 2002]: motion intention planned in the central nervous system Assumption: In a typical collaborative task, the motion intention of the human partner (in each direction) is determined by the interaction force, position and velocity at the interaction point (in the corresponding direction) of the human limb and robot arm. intention estimation

Intention Estimation NN is employed to estimate human partner’s motion intention, as below: Define a cost function: Updating law:

Intention Estimation

Experiment Study Scenario: The left wrist of Nancy is moved by human being’s hand. Note: The actual trajectory of the robot arm cannot be compared with the motion intention directly. Impedance parameters: NN setting:

Point-to-Point Movement Joint angle

Point-to-Point Movement External torque

Time-Varying Trajectory Joint angle

Time-Varying Trajectory External torque

Discussion The motion intention of the human partner has been observed by employing the human limb model. With the proposed trajectory adaptation, the robot arm is able to “actively” follow its human partner. Human partner and robot are considered to be two separated subsystems, and the performance of the whole coupled interaction system is yet to be rigorously analyzed.

Outline Introduction Impedance Learning Intention Estimation Zero Force Regulation Conclusion and Future Work

Problem Formulation Human limb model [M. M. Rahman 2002]: Control objective: design x0 in so that . Three cases: Point-to-point movement Periodic trajectory Arbitrary continuous trajectory

Zero Force Regulation Trajectory adaptation: Point-to-point movement: Periodic trajectory: Arbitrary continuous trajectory: Similar to the intention estimation method!

Main Result Theorem: With the above trajectory adaptations, the following performance of the closed-loop interaction system can be guaranteed: The robot is able to “actively” follow its human partner. Lemma: The above result will not be affected by the transient performance of the inner position control loop, if it is asymptotically stable.

Experiment Study Scenario: The left wrist of Nancy is moved by human being’s hand. Note: The actual trajectory of the robot arm cannot be compared with the motion intention directly. Impedance parameters:

Point-to-Point Movement Joint angle

Point-to-Point Movement External torque

Periodic Trajectory Joint angle

Periodic Trajectory External torque

Arbitrary Continuous Trajectory Joint angle

Arbitrary Continuous Trajectory External torque

Discussion Zero force regulation has been investigated for human-robot collaboration, so that the robot is able to “actively” follow its human partner. Adaptive control has been proposed to deal with the point-to-point movement, and learning control and NN control have been developed to generate periodic and non-periodic trajectories, respectively.

Outline Introduction Impedance Learning Intention Estimation Zero Force Regulation Conclusion and Future Work

Conclusion Impedance learning is developed to obtain desired impedance parameters when robots interact with unknown and dynamically changing environments. Trajectory adaptation is investigated for human-robot collaboration, so that robot is able to “actively” follow its human partner.

Future Work Simultaneous learning/adaptation of impedance and trajectory will be studied. Interaction control with other sensory information (e.g., image) will be investigated. The applications of the proposed methods in specific areas (e.g., rehabilitation, tele-operation, and human-robot collaboration) will be considered. Critical issues in practical implementations, such as time-delay and human factor, will be investigated.

References C. Wang, Y. Li, S. S. Ge and T. H. Lee, “Reference Adaptation for Robots in Physical Interactions with Unknown Environments,” IEEE Transactions on Cybernetics, 2016 Y. Li and S. S. Ge, “Human-Robot Collaboration Based on Motion Intention　Estimation,” IEEE Transactions on Mechatronics, 2014 Y. Li and S. S. Ge, “Impedance Learning for Robot Interacting with Unknown Environments,” IEEE Transactions on Control Systems Technology,　2013 S. S. Ge and Y. Li, “Force Tracking Control for Motion Synchronization in Human-Robot Collaboration,” Robotica, 2013 S. S. Ge, Y. Li and C. Wang, “Impedance Adaptation for Optimal Robot-　Environment Interaction,” International Journal of Control, 2013

Published Books Published Books Intelligent Control:

Published Books Switched Dynamical Systems: Robotic Systems:

International Journal of Social Robotics Editor in Chief: Shuzhi Sam Ge, NUS Co-Editor in Chief: Oussama Khatib, Stanford University, USA Aims and scope: Provide a platform for presenting findings and latest developments in social robotics, covering relevant advances in engineering, computing, arts and social sciences. Provide an overview of the current state of the social robotics scene. 1 Volume(-s) with 5 issue(-s) per annual subscription IF: 1.407

Welcome 2017 International Conference on Social Robotics (ICSR 2017) will be held in Tsukuba City, November 22-24, 2017—first time in Japan http://www.icsr2017.org/committees.html

Acknowledgement Machine Technology Intelligent Control Robot Sensing Machine Learning Cooperation Mechanisms Affective Sciences Behavioral Studies Psychological Impact Ethical Implications Interface Design