Approximate Dynamic Programming and Reinforcement Learning for Nonlinear Optimal Control of Power Systems November 4, 2003 Ronald Harley Georgia Institute of Technology ECS ECS Kumar Venayagamoorthy University of Missouri-Rolla

Adaptive Critic Design: Nonlinear Optimal Control Plant Informaton Utility Function ( U ) Optimal cost-to-go function ( J ) Critic Networks : To minimize the value (of derivatives) of J with respect to the states Derivatives via BP Model Network (Identifier) : To learn the dynamics of plant Model Network Action Network : To find optimal control u Plant Control Reinforcement Learning

STATCOM Control

Simulation Results 100ms SC at PCC, Line Voltage, Generator Terminal Voltage

The simplified schematic of the SSSC (160 MVA, 15KV V L-L ) Optimal control for FACTS devices Internal control for static series synchronous compensator (SSSC)

Optimal control for FACTS devices Internal control for SSSC (CONVC) PI Based internal controller (CONVC) for the SSSC Publication: N.G. Hingorani and L. Gyugyi, “Understanding FACTS-Concepts and Technology of Flexible AC Transmission Systems”, IEEE Press, New York, 2000.

Optimal control for FACTS devices Case study: 100 ms three phase short circuit test at receiving-end (infinite-bus) Rotor angle

Schematic single-line diagram showing an SCRC with external controller (160 MVA, 15KV V L-L ) Optimal control for FACTS devices External control for series capacitive reactance compensator (SCRC)

Optimal control for FACTS devices DHP based external controller (DHPEC) Schematic single-line diagram showing the DHP based external controller (DHPEC) Synchronous Generator Inf. bus v s i s SCRC r e2 x r e1 x Turbine- Governor AVR - Exciter Internal Control of SCRC Voltage Source Inverter V dc v c + GTO    X C + + Line #1 Line #2 * C X C X v r DHP based external controller (DHPEC)

Optimal control for FACTS devices Case study: Step changes X* C [pu] Speed deviation

Application in Multi-Machine power system Large-scale multi-machine power system

A UPFC in the POWER SYSTEM Infinite Bus Shunt Inverter Series Inverter VdcVdc Series Inverter Control Shunt Inverter Control V1V1 V dcref R 1, L 1 V2V2 V1V1 V 1ref Z1Z1 Synch Generator Governor AVR Exciter + - UPFC  Z1Z1 V 1ref  V dc P ref  P inj Q inj Q ref  1 2 P out, Q out V err V dcerr P err Q err R 2, L 2 VrVr Turbine P ref Neurocontroller Neuroidentifier QQ PP eded eqeq Neurocontroller Neuroidentifier  V dc VV  e pd  e pq

Responses of the Generator for a 180 ms 3- phase Short Circuit at bus 2 at P=0.8 p.u & Q=0.15 p.u Load angle Speed response

Micro-Machine Research Lab. at the University of Natal, Durban, South Africa

Gen. #1: Trans. Line Impedance Increase Time in seconds Load angle in degrees DHP_CONV CONV_PSS_CONV CON_CONV Time in seconds Terminal voltage in pu DHP_CONV CONV_PSS_CONV CONV_CONV