1. Approximate Dynamic Programming and Reinforcement Learning for Nonlinear Optimal Control of Power Systems November 4, 2003 Ronald Harley Georgia Institute of Technology ECS-0231632 ECS-0080764 Kumar Venayagamoorthy University of Missouri-Rolla

2. Adaptive Critic Design: Nonlinear Optimal Control Plant Informaton Utility Function ( U ) Optimal cost-to-go function ( J ) Critic Networks : To minimize the value (of derivatives) of J with respect to the states Derivatives via BP Model Network (Identifier) : To learn the dynamics of plant Model Network Action Network : To find optimal control u Plant Control Reinforcement Learning

3. STATCOM Control

4. Simulation Results 100ms SC at PCC, Line Voltage, Generator Terminal Voltage

5. The simplified schematic of the SSSC (160 MVA, 15KV V L-L ) Optimal control for FACTS devices Internal control for static series synchronous compensator (SSSC)

6. Optimal control for FACTS devices Internal control for SSSC (CONVC) PI Based internal controller (CONVC) for the SSSC Publication: N.G. Hingorani and L. Gyugyi, “Understanding FACTS-Concepts and Technology of Flexible AC Transmission Systems”, IEEE Press, New York, 2000.

7. Optimal control for FACTS devices Case study: 100 ms three phase short circuit test at receiving-end (infinite-bus) Rotor angle

8. Schematic single-line diagram showing an SCRC with external controller (160 MVA, 15KV V L-L ) Optimal control for FACTS devices External control for series capacitive reactance compensator (SCRC)

9. Optimal control for FACTS devices DHP based external controller (DHPEC) Schematic single-line diagram showing the DHP based external controller (DHPEC) Synchronous Generator Inf. bus v s i s SCRC r e2 x r e1 x Turbine- Governor AVR - Exciter Internal Control of SCRC Voltage Source Inverter V dc v c + GTO    X C + + Line #1 Line #2 * C X C X v r DHP based external controller (DHPEC)

10. Optimal control for FACTS devices Case study: Step changes X* C [pu] Speed deviation

11. Application in Multi-Machine power system Large-scale multi-machine power system

12. A UPFC in the POWER SYSTEM Infinite Bus Shunt Inverter Series Inverter VdcVdc Series Inverter Control Shunt Inverter Control V1V1 V dcref R 1, L 1 V2V2 V1V1 V 1ref Z1Z1 Synch Generator Governor AVR Exciter + - UPFC  Z1Z1 V 1ref  V dc P ref  P inj Q inj Q ref  1 2 P out, Q out V err V dcerr P err Q err R 2, L 2 VrVr Turbine P ref Neurocontroller Neuroidentifier QQ PP eded eqeq Neurocontroller Neuroidentifier  V dc VV  e pd  e pq

13. Responses of the Generator for a 180 ms 3- phase Short Circuit at bus 2 at P=0.8 p.u & Q=0.15 p.u Load angle Speed response

14. Micro-Machine Research Lab. at the University of Natal, Durban, South Africa

15. Gen. #1: Trans. Line Impedance Increase 1010.51111.51212.51313.51414.515 20 25 30 35 40 Time in seconds Load angle in degrees DHP_CONV CONV_PSS_CONV CON_CONV 1010.51111.51212.51313.51414.5 0.97 0.98 0.99 1 1.01 Time in seconds Terminal voltage in pu DHP_CONV CONV_PSS_CONV CONV_CONV