1. Sam Burden currently: Postdoc in EECS at UC Berkeley Sep 2015: Asst Prof in EE at UW Seattle Dynamic Inverse Models in Human-Cyber-Physical Systems S. Shankar Sastry Dean of Engineering at UC Berkeley Embedded Humans: Provably Correct Decision Making for Networks of Humans and Unmanned Systems (N00014-13-1-0341)

2. Increasingly mediated by automation – Augmented by hardware and software – Machines adapt to collaborate and assist Human interaction with the physical world Human-Cyber-Physical Systems (HCPS)

3. Embedding humans amid automation Can lead to performance degradation – pilot-induced oscillations in rotory/fixed-wing aircraft McRuer, Krendal 1974; Hess J. Guid. Cont. Dyn. 1997 Pavel et al. Prog. Aero. Sci. 2013 – overreliance on adaptive cruise control in cars Rudin-Brown, Parker Trans. Rch. F: Traffic Psych. and Behav. 2004 Requires predictive models for human behavior

4. Predictable behavior from internal models Popular paradigm posits pairs of internal models – forward model predicts sensory effect of motor action Sutton, Barto Psych. Rev. 1981; Jordan, Rumehlart Cog. Sci. 1992; Wolpert, et al. Science 1995 – inverse model computes motor command expected to yield desired behavior Kawato Curr. Opin. Neurobio. 1999; Thoroughman, Shadmehr Nature 2000; Conditt, Mussa-Ivaldi PNAS 1999 – Theoretical and empirical evidence for paired forward + inverse models Bhushan, Shadmehr Bio. Cybern. 1999; Sanner, Kosha Bio. Cybern. 1999 Hanuschkin, Ganguli, Hahnloser Front. Neural Circ. 2013; Giret, Kornfeld, Ganguli, Hahnloser PNAS 2014 Parallels in control theory, robotics, AI – Internal models, adaptive control, learning Francis, Wonham Automatica 1976; Sastry, Bodson Prentice Hall 1989; Sutton, Barto, Williams IEEE CSM 1992 Crawford, Sastry UCB EECS 1996; Atkeson, Schaal ICML 1997; Papavassiliou, Russell IJCAI 1999

5. Instantiating internal models Forward model ( M : U  Y ): static vs. dynamic – static map (linear or nonlinear) Hanuschkin, Ganguli, Hahnloser Front. Neural Circ. 2013 Giret, Kornfeld, Ganguli, Hahnloser PNAS 2014 – dynamic map depends on intermediate state Thoroughman, Shadmehr Science 2000 Wolpert, Diedrichsen, Flanagan Nature Neurosci. 2011 Inverse model ( M -1 : Y  U ): hard to define – static map may fail to be one-to-one or onto – dynamic map may be acausal or need state estimate

6. Today’s talk: dynamic inverse models from the perspective of mathematical control theory 1.derivation of dynamic inverse model 2.properties and implications for design of HCPS Dynamic inverse models in HCPS

7. Single input/single output forward model Consider forward model in control-affine form: – x in R n, u in R, y in R – f, g in C r (R n, R n ), h in C r (R n, R) Suppose model has strict relative degree  in N : – Expressed in terms of Lie derivatives L f h(x), L g h(x) : – intuitively, input affects  -th derivative of output – e.g.  =2 for Lagrangian mechanical systems applicable to interaction with physical world

8. Transformation of forward model Forward model: Suppose model has strict relative degree  in N : – e.g.  =2 for Lagrangian mechanical systems Then model is linear in new coordinates: – There exists such that in coordinates forward model has the form – Choosing yields simpler forward model

9. Dynamic inverse model Forward model: Given desired output y d, we seek desired input u d – Since y =  1, there exists unique v d such that – States  rendered unobservable by input v d ! Note that exact tracking is too stringent – need initial cond. But it’s easy to achieve exponential tracking – applying input yields  How does tracking affect unobservable states  ?

10. Tracking with stable model pair Forward model: Dynamic inverse model: Theorem: If forward and inverse models are exponentially stable, then feedforward input from dynamic inverse of internal model achieves exponential tracking for physical system. – Trajectories converge for stable model pairs (M, M -1 ) – Feedforward input “asymptotically inverts” dynamics

11. Tracking with stable model pair (M, M -1 ) Theorem implies: – For stable model pair, trajectories x, x’ converge to – Feedforward input “asymptotically inverts” dynamics (M, M -1 )

12. Suppose human ( H ) implements inverse model: – can infer desired task y H from observed input u H – nominal forward model becomes: – automation can intervene to improve performance by minimizing cost function J:R n  R using input  – guarantees performance improvement following intervention in human-cyber-physical system Application to provably-correct interventions

13. Today: predictive models for interaction Future: enhance human ability to interact with and control the built world – Human-Cyber-Physical – Human Intranet – Cybathlon Dynamics of humans embedded w/ machines Humans are the enabling technology

14. - Extensions and Generalizations - Properties of dynamic inverse model - Behavioral repertoire of humans Appendix

15. Extensions and generalizations Forward model: Dynamic inverse model: Results easily extend to accommodate: – multiple inputs / multiple outputs – (small) perturbations in dynamics Sastry Springer 1999 – approximate input-output linearization Hauser PhD Thesis 1989; Hauser, Sastry, Kokotovic IEEE TAC 1992; Banaszuk, Hauser SIAM JCO 1996 – learning / adaptation / estimation of dynamics Sutton, Barto, Williams IEEE CSM 1992; Papavassiliou, Russell ICJAI 1999 Sastry, Bodson Prentice Hall 1989; Vrabie, Vamvoudakis, Lewis IET 2013

16. Properties of dynamic inverse model Forward model: Dynamic inverse model: Property: dynamic inverse model is unique – Exact tracking input determined by y d – Independent of how internal model is represented or obtained (e.g. reinforcement learning, adaptive ctrl.) Sutton, Barto, Williams IEEE CSM 1992; Papavassiliou, Russell ICJAI 1999 Sastry, Bodson Prentice Hall 1989; Vrabie, Vamvoudakis, Lewis IET 2013 – Impossible to learn if inverse model is unstable

17. Behavioral repertoire of humans Too rich to model from first principles – Spans computational, algorithmic, & physical “levels of analysis” Marr, Poggio MIT AI MEMO 1976 – Influenced by neurophysiological state (cognitive load, hunger) LaPointe, Stierwalt, Maitland Int. J. Speech-Lang. Pathology 2010; Danziger, Levav, Avnaim-Pesso PNAS 2011 Can reduce dramatically during particular tasks – Bernstein posed the “problem of motor redundancy” Bernstein Pergamon Press 1967. – Perhaps instead we “exploit the bliss of motor abundance” e.g. using synergies, uncontrolled manifolds, optimality Latash Exp. Brain Rch. 2012; Ting, Macpherson J. Neurophys. 2005; Scholz, Schoner, Exp. Brain Rch. 1999 Todorov, Jordan Nature Neurosci. 2002; Diedrichsen, Shadmehr, Ivry Trends Cog. Sci. 2010 – For instance, locomotion naturally reduces dimensionality Burden, Revzen, Sastry IEEE TAC 2015