1. POLI di MI tecnicolano ADAPTIVE AUGMENTED CONTROL OF UNMANNED ROTORCRAFT VEHICLES C.L. Bottasso, R. Nicastro, L. Riviello, B. Savini Politecnico di Milano AHS International Specialists' Meeting on Unmanned Rotorcraft Chandler, AZ, January 23-25, 2007

2. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Rotorcraft UAVs at PoliMI navigation control Low-cost platform for development and testing of navigation and control strategies (including vision, flight envelope protection, etc.); Vehicles: off-the-shelf hobby helicopters; On-board control hardware based on PC-104 standard; everything is in-house developed Bottom-up approach: everything is in-house developed (Inertial Navigation System, Guidance and Control algorithms, Linux-based real-time OS, flight simulators, etc. etc.)

3. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Outline Non-linear model predictive control; Reference Augmented Predictive Control (RAPC): motivations; Reference Augmented Model Identification; Reference Augmented Neural Control; Results; Conclusions and outlook.

4. Reference Augmented Predictive Control POLITECNICO di MILANO DIA UAV Control Architecture Target Obstacles Hierarchical three-layer control architecture Hierarchical three-layer control architecture (Gat 1998): Vision/sensor range Strategic layer: assign mission objectives (typically relegated to a human operator); Tactical layer: generate vehicle guidance information, based on input from strategic layer and sensor information; Reflexive layer: track trajectory generated by tactical layer, control, stabilize and regulate vehicle. Adaptive Non-linear Model Predictive Control In this paper: Adaptive Non-linear Model Predictive Control.

5. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Non-Linear Model Predictive Control Non-linear Model Predictive Control Non-linear Model Predictive Control (NMPC): non-linear reduced model Find the control action which minimizes an index of performance, by predicting the future behavior of the plant using a non-linear reduced model. - Reduced model: - Initial conditions: - Output definition: Cost: with desired goal outputs and controls. Stability results Stability results: Findeisen et al. 2003, Grimm et al. 2005. L ( y ; u ) = ( y ¡ y ¤ ) T Q ( y ¡ y ¤ ) + ( u ¡ u ¤ ) T R ( u ¡ u ¤ ) m i n u ; x ; y J = Z t 0 + T p t 0 L ( y ; u ) d t s. t. : f ( _ x ; x ; u ) = 0 t 2 [ t 0 ; t 0 + T p ] x ( t 0 ) = x 0 y = g ( x ) t 2 [ t 0 ; t 0 + T p ] ( ¢ ) ¤

6. Reference Augmented Predictive Control POLITECNICO di MILANO DIA F u t ure P as t S t eer i ngw i n d ow P re d i c t i onw i n d ow x 0 P re d i c t i onerror S t a t e t rac k i ngerror C on t ro l t rac k i ngerror t 0 t 0 t 0 + T p t 0 + T s C ompu t e d con t ro l u ( t ) P l an t response e x ( t ) P re d i c t e d response x ( t ) G oa l response x ¤ ( t ) G oa l con t ro l u ¤ ( t ) Non-Linear Model Predictive Control

7. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Model-Adaptive Predictive Control 1. Tracking problem Plant response 3. Reduced model update Predictive solutions 2. Steering problem Prediction window Steering window Tracking cost Prediction error Prediction window Tracking cost Steering window Prediction error Tracking cost Prediction window Steering window Prediction error Goal trajectory Receding horizon control Receding horizon control:

8. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Motivation knowledgelegacy For any given problem: wealth of knowledge and legacy methods which perform reasonably well; undesirable wasteful Quest for better performance/improved capabilities: undesirable and wasteful to neglect valuable existing knowledge; Reference Augmented Predictive ControlRAPC Reference Augmented Predictive Control (RAPC): exploit available legacy methods, embedding them in a non-linear model predictive control framework. Specifically: Model Model: augment flight mechanics rotorcraft models (BEM+inflow theories) to account for unresolved or unmodeled physics; Control Control: design a non-linear controller augmenting linear ones (LQR) which are known to provide a minimum level of performance about certain linearized operating conditions.

9. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Reference Augmented Predictive Control Approach: reference model / reference control law; - Choose a reference model / reference control law; adaptiveparametric function; - Augment the reference using an adaptive parametric function; Adjustgood approximation of the actual system / optimal control law - Adjust the function parameters to ensure good approximation of the actual system / optimal control law (parameter identification). Reasons for using a reference model / control Reasons for using a reference model / control: even before any learning - Reasonable predictions / controls even before any learning has taken place (otherwise would need extensive pre-training); -small quantity - Easier and faster adaption: the defect is typically a small quantity, if the reference solution is well chosen.

10. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Reference model Reference model: Euler’s eqs.+BEM+inflow model Plant

11. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Outline Non-linear model predictive control; Reference Augmented Predictive Control (RAPC): motivations; Reference Augmented Model Identification; Reference Augmented Neural Control; Results; Conclusions and outlook.

12. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Goal Goal: reduced modelpredicting the behavior of the plant Develop reduced model capable of predicting the behavior of the plant with minimum error (same outputs when subjected to same inputs); self-adaptive Reduced model must be self-adaptive (capable of learning) to adjust to varying operating conditions. Predictive solutions Prediction (tracking) window Steering window Prediction error to be minimized Reference Augmented Model Identification

13. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Neural augmented reference model Neural augmented reference model: reference (problem dependent) analytical model, Remarknotadequate Remark: reference model will not, in general, ensure adequate predictions, i.e. when = system states/controls, = model states/controls. Augmented reference model: unknowndefect where is the unknown reference model defect that ensures when if we knewperfect prediction Hence, if we knew, we would have perfect prediction capabilities. d d e u = u. e u = u ; f re f ( _ x ; x ; u ) = 0 : e x 6 = x x ; u e x ; e u f re f ( _ x ; x ; u ) = d ( x ; u ) ; e x = x Reference Augmented Model Identification

14. Reference Augmented Predictive Control POLITECNICO di MILANO DIA single-hidden-layer neural networks Approximate with single-hidden-layer neural networks: where and = functional reconstruction error; = matrices of synaptic weights and biases; = sigmoid activation functions; = network input. reduced model parameters The reduced model parameters Kalman filtering are identified on-line using Kalman filtering. d ¾ ( Á ) = ( ¾ ( Á 1 ) ;:::; ¾ ( Á N n )) T d ( y ; u ) = d p ( x ; u ; p m ) + " ; d p ( x ; u ; p m ) = W m T ¾ ( V m T i + a m ) + b m ; " W m ; V m ; a m ; b m i = ( x T ; u T ) T p m = ( :::; W m i k ; V m i k ; a m i ; b m i ;::: ) T Reference Augmented Model Identification

15. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Model Augmentation Results Pitch rate for plant, reference, and neural-augmented reference with same prescribed inputs. Short transient = fast adaption Black: plant Red: reference model Blue: reference model +neural network

16. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Outline Non-linear model predictive control; Reference Augmented Predictive Control (RAPC): motivations; Reference Augmented Model Identification; Reference Augmented Neural Control; Results; Conclusions and outlook.

17. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Prediction problem: Enforcing optimality Enforcing optimality, we get: Non-Linear Model Predictive Control m i n u ; x ; y J = Z t 0 + T p t 0 L ( y ; u ) d t s. t. : f ( _ x ; x ; u ) = 0 t 2 [ t 0 ; t 0 + T p ] x ( t 0 ) = x 0 y = g ( x ) t 2 [ t 0 ; t 0 + T p ] Model equations: Adjoint equations: Transversality conditions: State initial conditions: Co-state final conditions:

18. Reference Augmented Predictive Control POLITECNICO di MILANO DIA F u t ure P as t P re d i c t i onw i n d ow x 0 t 0 t 0 t 0 + T p G oa l response x ¤ ( t ) G oa l con t ro l u ¤ ( t ) Non-Linear Model Predictive Control u ( t ) O p t i ma l con t ro l u ( t ) x ( t ) ; t < t 0 u ( t ) ; t < t 0 Â ( ¢ ; ¢ ; ¢ ; ¢ ) minimizing control It can be shown that minimizing control is See paper for details. u ( t ) = Â ¡ x 0 ; y ¤ ( t ) ; u ¤ ( t ) ; t ¢

19. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Reference augmented form: where is the unknown control defect. Remark Remark: if one knew, the optimal control would be available without having to solve the open-loop optimal control problem. Idea Idea: -Approximate - Approximate using an adaptive parametric element: -Identify - Identify on-line, i.e. find the parameters which minimize the reconstruction error. p c " Reference Augmented Predictive Control u ( t ) = u re f ( t ) + À ¡ x 0 ; y ¤ ( t ) ; u ¤ ( t ) ; t ¢ À ( ¢ ; ¢ ; ¢ ; ¢ ) À ( ¢ ; ¢ ; ¢ ; ¢ ) À ( ¢ ; ¢ ; ¢ ; ¢ ) À ¡ x 0 ; y ¤ ( t ) ; u ¤ ( t ) ; t ¢ = À p ¡ x 0 ; y ¤ ( t ) ; u ¤ ( t ) ; t ; p c ¢ + " c À p ( ¢ ; ¢ ; ¢ ; ¢ )

20. POLITECNICO di MILANO DIA Iterative procedure Iterative procedure to solve the problem in real-time: forward (state prediction): Integrate reduced model equations forward in time over the prediction window, using and the latest available parameters (state prediction): backward (co-state prediction): Integrate adjoint equations backward in time (co-state prediction): Correct Correct control law parameters, e.g. using steepest descent: p c u re f p c _ p c = ¡ ´ ^ J ; p c ! p new c = p o ld c ¡ ´ ^ J ; p c ¡ d ( f T ; _ x ¸ ) d t + ( f ; x + u T ; x f ; u ) T ¸ + y T ; x L ; y + u T ; x L ; u = 0 t 2 [ t 0 ; t 0 + T p ] ¸ ( t 0 + T p ) = 0 f ( _ x ; x ; u ; p m ) = 0 t 2 [ t 0 ; t 0 + T p ] x ( t 0 ) = x 0 On-line Identification of Control Parameters

21. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Remark Remark: the parameter correction step transversality condition seeks to enforce the transversality condition optimal satisfied Once this is satisfied, the control is optimal, since the state and co- state equations and the boundary conditions are satisfied. _ p c = ¡ ´ ^ J ; p c Z t 0 + T p t 0 À T ; p c ( L ; u + f T ; u ¸ ) d t = 0 On-line Identification of Control Parameters

22. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Tracking cost Future Target Predict Predict state forward Predict Predict co-state backwards Predict Predict control action Update Update estimate of control action, based on transversality violation _ p c = ¡ ´ ^ J ; p c Advance Advance plant Update Update model, based on prediction error Past Optimal control Prediction error Repeat Repeat Future Past Prediction horizon Steering window State Control On-line Identification of Control Parameters x ( t ) ¸ ( t ) u ( t )

23. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Neural-Network-Based Implementation - - Drop dependence on time history of goal quantities: - - Approximate temporal dependence using shape functions: -single-hidden-layer neural network - Associate each nodal value with the output of a single-hidden-layer feed-forward neural network, one for each component: where Output: Input: Control parameters: À p ¡ x 0 ; y ¤ ( t ) ; u ¤ ( t ) ; t ; p c ¢ ¼ À p ¡ x 0 ; y ¤ ( t 0 ) ; u ¤ ( t 0 ) ; t ; p c ¢ À p ¡ x 0 ; y ¤ ( t 0 ) ; u ¤ ( t 0 ) ; ¿ ; p c ¢ ¼ ( 1 ¡ » ) À p k ¡ x 0 ; y ¤ ( t 0 ) ; u ¤ ( t 0 ) ; p c ¢ + » À p k + 1 ¡ x 0 ; y ¤ ( t 0 ) ; u ¤ ( t 0 ) ; p c ¢ o c = W T c ¾ ( V T c i c + a c ) + b c o c = ( À T p 0 ; À T p 1 ;:::; À T p M ¡ 1 ) T i c = ¡ x T 0 ; x ¤ T ( t 0 ) ; u ¤ T ( t 0 ) ¢ T p c = ( :::; W c ij ;:::; V c ij ;:::; a c i ;:::; b c i ;::: ) T

24. Reference Augmented Predictive Control POLITECNICO di MILANO DIA F u t ure P as t P re d i c t i onw i n d ow x 0 t 0 t 0 t 0 + T p x ( t ) ; t < t 0 u ( t ) ; t < t 0 u ¤ ( t 0 ) x ¤ ( t 0 ) NN À p k Neural-Network-Based Implementation x ¤ ( t ) u ¤ ( t )

25. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Outline Non-linear model predictive control; Reference Augmented Predictive Control (RAPC): motivations; Reference Augmented Model Identification; Reference Augmented Neural Control; Results; Conclusions and outlook.

26. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Vehicle Model and Simulation Environment Vehicle model Vehicle model: Blade element and inflow theory (Prouty, Peters); Quasi-steady flapping dynamics, aerodynamic damping correction; Look-up tables for aerodynamic coefficients of lifting surfaces; Effects of compressibility and downwash at the tail due to main rotor; Process and measurement noise, delays. Reflexive controller Reflexive controller: State reconstruction by Extended Kalman Filtering; Reference controller: output- feedback LQR at 50 Hz; Goal trajectory planned as in Bottasso et al. 2007.

27. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Results

28. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Results Integral tracking error vs. length of prediction window: Significant improvement over LQR

29. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Results Turn rate vs. time: RAPC LQR

30. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Results Integral tracking error vs. model mismatch parameter: RAPC without model adaption RAPC with model adaption Significant improvement over LQR

31. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Results Main rotor collective & norm of control network parameters: Initial transient Adapted

32. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Conclusions reduced model identification Non-linear reduced model identification for capturing unmodeled or unresolved physics; promoted Linear controller promoted to non-linear; real-time Hard real-time capable (fixed number of ops, no iterations); independently Adaption of control action can be performed independently from adaption of reduced model; before adaptionsimplify adaption Reference model and reference control ensure good predictions even before adaption, avoid need for pre-training, simplify adaption since defect is small; adaption diagnostics Conceptually possible (but not investigated here) to do adaption diagnostics by monitoring defects; non-linearly stable Theoretically non-linearly stable (if identification of, successful); Basic concept demonstrated in a high-fidelity virtual environment. p c p m

33. Reference Augmented Predictive Control POLITECNICO di MILANO DIA Outlook Real-time implementation and integration in a rotorcraft UAV (in progress) at the Autonomous Flight Lab at PoliMI; Testing and extensive experimentation; Integration with vision for fully autonomous navigation in complex environments.