Silvia Ferrari and Mark Jensenius Department of Mechanical Engineering Duke University Crystal City, VA, September 28, 2005 Robust and Reconfigurable Flight Control by Neural Networks

A Multiphase Learning Approach for Automated Reasoning  On-line Control Identification Planning Routing Scheduling... Supervised Learning: Reinforcement Learning: The same performance metric is optimized during both phases!

Introduction Stringent operational requirements introduce Complexity Nonlinearity Uncertainty Classical/neural synthesis of control systems A-priori control knowledge Adaptive neural networks Action network takes immediate control action Critic network evaluates the action network performance Dual heuristic programming adaptive critic architecture:

Sigmoidal neural networks for control: coping with complexity Applicability to nonlinear systems Applicability to multivariable systems Batch and incremental training Closed-loop stability and robustness by IQCs Constrained training for robust adaptation on line Motivation

Full Envelope Control! Modeling On-line Training Design Approach Initialization Linear Control Linearizations

Nonlinear Dynamical System Full-scale Aircraft Simulation: In particular: State vector: Control vector: Vector of parameters: Output vector: YBYB XBXB ZBZB Thrust Drag Lift V mg p q r  

Classical Control Design Linearizations: Altitude (m) Velocity (m/s) Classical linear designs: Multivariable control (PI) Multi-objective synthesis (LMI) Flight envelope and design points: (  =  =  = 0) k  ( )

Input-to-node variable One-hidden Layer Sigmoidal Neural Network s - Hidden nodes Output: z = NN(p) Input: p Adjustable parameters: W, d, v w 11 p1p2...pqp1p2...pq d1d1 1 d2d2 1 dsds w sq n1n1 n2n2 nsns v1v1 v2v2 vsvs 1 b z 11 22 ss Output equations: z = v T  [Wp + d] Gradient equations: v i  '(n i )w ij, j = 1,..., q

Training set: Requirements: Output and Gradient Initialization Equations: General Algebraic Training Approach Known neural network.. Gradient Output Input (c k ) T = W T {v  [Wx k + d]} u k = v T  [Wx k + d] u = Sv c k = B k W

Gradient-based Algebraic Training n: vector of all input-to-node constants, n i k c: vector of feedback gains b: output bias vector Assume each input-to-node variable, is a known constant Then, n is known and the initialization equations can be written as: Linear algebraic initialization equations: where: Vec operator ; to be solved for w a ; to be solved for w x ; to be solved for v ~

A: (p 2 3s) sparse matrix of scheduling variables S: (p s) matrix of sigmoidal functions of n X: (np ns) sparse matrix evaluated from v and n where: k = 1, 2,..., p Initialization Matrices

Linear Control Comparison of Initialized PI NN and Linear Controllers Time (sec) Velocity (m/s) Climb Angle (deg) Large-Angle Maneuver Small-Angle Maneuver Initialized Neural Network Control Aircraft Response to Climb-Angle Command Input, at Interpolating Conditions (H 0, V 0 ) = (2Km, 95 m/s)

Stability Analysis via Integral Quadratic Constraints (IQCs) Standard feedback interconnection between a transfer matrix G(s) or LTI system, and a causal bounded operator  : IQC Stability Theorem: G(s)G(s) w v  Equivalent LMI feasibility problem with positive, real parameters p i and symmetric matrix P:  , then the interconnection is stable. If there exists  > 0, such that,

Closed-loop Stability of Neural Network Controller Closed-loop system comprised of NN controller and LTI model, Lure-type System Applying the IQC Stability Theorem:  is a bdd, causal diagonal operator with repeated nonlinearities that are monotonically non-decreasing, slope-restricted, and belong to [0, 0.5]. Thus, the stability of the NN controlled system is guaranteed if there exists constant symmetric matrices M, P   s  s that satisfy the following LMIs: i = 1, …, s B N = BV, C N = WC a

Adaptive Critic On-line Adaptation ycyc x(t)x(t) _ u(t)u(t) ucuc + _ xcxc ys(t)ys(t) e a NN F SVG CSG  V/  x a (t)(t) NN A NN C

Dynamic Programming Approach By The Principle of Optimality, Time J*J* terminal cost t0t0 ttftf the minimization of J can be imbedded in the minimization of V(t): V*V* t0t0 ttftf terminal cost a b c V * abc = V ab + V * bc

Critic network criterion: = NN C Target at t NN A Target at t Action network criterion (optimality condition): Recurrence relation [Howard, 1960] : Dual Heuristic Programming

Action/Critic Network On-line Learning, at Time t The (action/critic) network must meet its target, NN +  NN Target Generation E  Network performance   Network error w  Network weights w l+1 = w l +  w l w(t) = w 0 w(t + 1) wlwl w l+1 RProp Modified Resilient Backpropagation (RProp) minimizes E w.r.t. w:

Adaptive vs. Fixed NN Controllers During a Coupled Maneuver Velocity (m/s) Climb Angle (deg) Roll Angle (deg) Sideslip Angle (deg) Time (sec) Aircraft response, (H 0, V 0 ) = (2 Km, 95 m/s) Adaptive Critic Neural Control: Fixed Neural Control: Command Input:

Adaptive vs. Fixed NN Controllers During a Large-Angle Maneuver Velocity (m/s) Climb Angle (deg) Roll Angle (deg) Sideslip Angle (deg) Time (sec) Aircraft response, (H 0, V 0 ) = (7 Km, 160 m/s) Adaptive Critic Neural Control: Command Input: Fixed Neural Control:

Adaptive vs. Fixed NN Controllers During a Large-Angle Maneuver Adaptive Critic Neural Control Fixed Neural Control Time (sec)  T (%) Control history, (H 0, V 0 ) = (7 Km, 160 m/s)  S (deg)  A (deg)  R (deg) Trajectory Altitude (m) North (m) East (m)

Fixed Neural Controller Performance in the Presence of Control Failures Time (sec) Aircraft response, (H 0, V 0 ) = (3 Km, 100 m/s) Fixed Neural Control Command Input Control history Time (sec)  T (%)  S (deg)  A (deg)  R (deg) Control Failures:  T = 0, 0  t  10 sec  S = 0, 5  t  10 sec  R = –34 o, t  5  R = 0, 5  t  10 sec V (m/s)  (deg)  (deg)  (deg)  (deg)

Adaptive vs. Fixed NN Controllers in the Presence of Control Failures Time (sec) Adaptive Critic Neural Control Fixed Neural Control  T (%) Control history, (H 0, V 0 ) = (7 Km, 160 m/s)  S (deg)  A (deg)  R (deg) Control Failures: (10  t  15 sec)  T max = 50%  R = – 15 o

Adaptive vs. Fixed NN Controllers in the Presence of Control Failures Velocity (m/s) Climb Angle (deg) Roll Angle (deg) Sideslip Angle (deg) Time (sec) Aircraft response after t = 10 sec, (H 0, V 0 ) = (3 Km, 100 m/s) Adaptive Critic Neural Control: Command Input: Fixed Neural Control: Yaw Angle (deg) Angle of Attack (deg)

M1M1 M2M2 a1a1 ~ a2a2 ~ 1 a WAWA WRWR V xaxa ~ u ~ or b Robust Adaptation: Constrained Algebraic Training

, b, A, W A constrained weights unconstrained weights Zero Randomized Design points Hyperspherical initialization construction functions Neural Network Weights Partitioning

Interpolation Point

Linear Non-adapting Neural Adapting Neural Controller Performance at Interpolation Point

Linear Non-adapting Neural Adapting Neural Controller Performance at Interpolation Point

Linear Non-adapting Neural Adapting Neural On-line Cost Optimization through Adaptation

Action Neural Networkt = 0 sect = 5 sect = 10 sec Constrained Output MSE x x x10 -7 Unconstrained Output MSE x x x10 11 Constrained Gradient MSE x x x Unconstrained Gradient MSE x x x10 -4 Adaptive NN Controller Performance at Design Points

Extrapolation Point

Linear Non-adapting Neural Adapting Neural Controller Performance at Extrapolation Point

Linear Non-adapting Neural Adapting Neural Controller Performance at Extrapolation Point

Summary of Results Properties of learning control system:  Improves global performance  Lends itself to stability and robustness analysis via IQCs  Preserves prior knowledge through constrained training  Suspends and resumes adaptation, as appropriate Future work:  Computational complexity  Aircraft system identification by neural networks  Stochastic effects  Optimal estimation Acknowledgment: This research is funded by the National Science Foundation.

Silvia Ferrari Department of Mechanical Engineering Duke University Many Thanks to: Mark Jensenius Robust and Reconfigurable Flight Control by Neural Networks

Backup Slides

CBCB P CICI C F, f[] = 0 : Algebraic Initialization, Proportional-Integral Neural Network Controller yc(t)yc(t) x(t)x(t) u(t)u(t) uc(t)uc(t) + - uB(t)uB(t) uI(t)uI(t) xc(t)xc(t) ys(t)ys(t) e(t)e(t) a(t)a(t) NN F SVG CSG (t)(t) : On-line Training. NN I NN C NN B

Feedback Neural Network Initialization zB(t)zB(t) NN B a Linear optimal control law: Initialization Requirements: At each design point (k), (R1) (R2)

Development of Feedback Initialization Equations Feedback Neural Network Initialization Equations: Network output: j = 1, 2,..., q Network gradient: l = 1, 2,..., m where is the l th -row of the matrix where and