1. Extremum seeking control Dragan Nešić The University of Melbourne Acknowledgements: Y. Tan, I. Mareels, A. Astolfi, G. Bastin, C. Manzie; A. Mohammadi; W. Moas Australian Research Council. TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AAA A A

2. Outline Motivating examples Background Ad hoc designs Black box: - Problem formulation - Systematic design Gray box: - Problem formulation - Systematic design Conclusions & future directions

3. A Prelude This is an approach for online optimisation of the steady-state system behaviour. A standing assumption is that the plant model or the cost is not known. The controller finds the extremum in closed- loop fashion.

4. Motivating examples

5. Continuously Stirred Tank (CST) Reactor Substrate Product u=Vol. flow ratePerformance output y: Productivity J P Yield J Y Inflow Outflow Overall J T

6. Single enzymatic reaction Michaelis-Menten Kinetics Productivity and yield Total cost is typically unknown!! In steady-state, we would typically want to operate around G. Bastin, D. Nešić, Y. Tan and I. Mareels, “On extremum seeking in bioprocesses with multivalued cost functions”, Biotechnology Progress, 2009.

7. Raman amplifiers Fibre span Power sensors Pump lasers u=laser power P.M. Dower, P. Farrell and D. Nešić, “Extremum seeking control of cascaded optical Raman amplifiers”, IEEE Trans. Contr. Syst. Tech., 2008. Cost Performance output y: Spectral flatness (equalization) Desired power ¸ p

8. Other engineering examples PlantPerformance output Turbine Generated power Solar cell Generated power Variable cam timing Fuel consumption Tokamak Reflected power during Lower Hybrid (LH) plasma heating experiments Non-holonomic vehicles Distance from a source of a signal Paper machine Retention of fines and fibers in the sheet Ultrasonic/Sonic Driller/Corer Distance from resonance Human Exercise Machine The user’s power output ABS Magnitude of friction force

9. Examples from biology E. Coli bacteria search for food in a similar manner to an extremum seeking algorithm (M. Krstic et al). Some fish search for food in a similar manner to extremum seeking (M. Krstic et al).

10. Background

11. Classification of approaches NLP based ESC [Popović, Teel,…] Adaptive ESC [Krstić, Ariyur, Guay, Tan, Nešić,…] Deterministic Stochastic Adaptive ESC [Krstić, Manzie,…] NLP based ESC [Spall,..] Also continuous-time versus discrete-time.

12. Brief history (deterministic): 19221950 2000 First ESC? 196019702009 Vibrant research area Many new schemes proposed Especially Adaptive First local stability result for adaptive ESC Systematic design discrete-time NLP. Systematic design adaptive Schemes. Beginning Ad-hoc designs Rigorous analysis and design Åström & Wittenmark: “one of the most promising adaptive control techniques”. 1995

13. Ad hoc adaptive designs

14. Adaptive ESC [Krstić & Wang 2000] +x Extremum seeking controller Parameters:

15. Static scalar case (gradient descent) +x Extremum seeking controller Parameters: Y. Tan, D. Nešić and I. Mareels, “On non-local stability properties of extremum seeking control”, Automatica, 2006.

16. Comments Many similar adaptive algorithms proposed. Case-by-case convergence analysis. No clear relationship with optimization. A unifying design approach is unavailable. A unifying convergence analysis is missing. A unifying approach exists for another class of schemes [Teel and Popovic, 2000].

17. Black Box Approach

18. Problem formulation (black box) Extremum Seeking Controller Assumption 1: - Q(.) has an extremum (max) - Q(.) is unknown Dynamic case: Problem: Design ESC so that Assumption:

19. Systematic design (derivatives estimation) D. Nešić, Y. Tan, W. Moas and C. Manzie, “A unifying approach to extremum seeking: adaptive schemes based on derivatives estimation”, IEEE Conf. Dec. Contr. 2010.

20. Continuous optimization (offline) No inputs & outputs Q(.) is known, so all derivatives of Q(.) are known

21. Examples Gradient method Continuous Newton method

22. Extremum seeking (online) Inputs & outputs available Q(.) is unknown Derivatives estimator a, ! L, ² are positive controller parameters

23. Systematic design (use the previous block diagram) Step 1: Choose an optimization scheme. Step 2: Use an estimator for D N Q( ¢ ). Step 3: Adjust the controller parameters.

24. Estimator design

25. Estimating DQ( µ )

26. Analysis where µ is assumed constant. Average the right hand side of the model. Model of the system:

27. Estimating D 2 Q( µ )

28. Higher order derivatives

29. Convergence analysis

30. Model of the overall system ! L, ² and a are controller parameters that need to be tuned to achieve appropriate convergence properties. Slow: Fast:

31. Assumption 1 (global max) There exists a global maximum

32. Assumption 2 (robust optimizer) The solutions of satisfy for sufficiently small w(t).

33. Theorem Suppose Assumptions 1-2 hold. Then Tuning guidelines

34. Geometrical interpretation Fast transient (estimator) Slow transient optimization Exist ! L, ², a For any ¢, º

35. Comments A systematic design approach proposed. Rigorous convergence analysis provided. Controller tuning proposed in general. Dynamic plants treated in the same way. Multi-input case is treated in a similar way. Averaging and singular perturbations used. Tradeoffs between the domain of attraction, accuracy and speed of convergence!

36. Bioreactor example All our assumptions hold – gradient method used.

37. Gray Box Approach

38. Problem formulation (gray box) Extremum Seeking Controller Assumption 1: - Q(., p ) has an extremum (max) - Q(.,.) is known; p is unknown Dynamic case: Problem: Design ESC so that Assumption:

39. Systematic design (parameter estimation) D. Nešić, A. Mohamadi and C. Manzie, “A unifying approach to extremum seeking: adaptive schemes based on parameter estimation”, IEEE Conf. Dec. Contr. 2010.

40. Extremum seeking (online) Inputs & outputs available p is unknown Parameter estimator a, ! L, ² are controller parameters

41. Comments Similar systematic framework in this case. Similar convergence analysis holds. Classical adaptive parameter estimation schemes can be used. Dynamic plants dealt with in the same way. Persistence of excitation is crucial for convergence. Tradeoffs between domain of attraction, accuracy and convergence speed.

42. Example Consider the static plant: We used the continuous Newton method. Classical parameter estimation used. Values p 1 =9 and p 2 =8 used in simulations.

43. Simulations Performance output Control inputsParameters

44. Final remarks Several tradeoffs exist; convergence slow. Many degrees of freedom: dither shape, controller parameters, optimization algorithm, estimators. Some global convergence results available (similar to simulated annealing).

45. Summary A systematic design framework presented for two classes of adaptive control schemes. Precise convergence analysis provided. Controller tuning and various tradeoffs understood well. Applicable to a range of engineering and non- engineering fields.

46. Future directions Tradeoffs: convergence speed, domain of attraction and accuracy. Various extensions: non-compact sets, global results, non-smooth systems, multi-valued cost functions. Schemes robust although no formal proofs. Tailor the tools to specific problems. Exciting research area.

47. Thank you!