Hanoi, January 28 th 2015 Rodolfo Soncini-Sessa DEI – Politecnico di Milano IMRR Project 8 – Design algorithms INTEGRATED AND SUSTAINABLE WATER MANAGEMENT OF RED-THAI BINH RIVER SYSTEM IN A CHANGING CLIMATE

IMRR phases econnaissance odeling the system ndicators identification cenarios definition lternative design valuation R M I S A E Soncini Sessa, 2007 omparison … C

The Design Problem (It)(It) scenario

Design algorithm SDP Stochastic Dynamic Programming

The Design Problem (It)(It) scenario

Assumptions The objectives are separable Compensation is acceptable then

The Design Problem for SDP If  t+1 is a white process (It)(It) scenario If  t+1 is a white process and we do not consider exogenous information … If  t+1 is a white process and we do not consider exogenous information … then Stochastic Dynamic Programming (SDP)

SDP algorithm x t+1 = f t (x t,u t,  t+1 )  t+1 ~  t ( ) utUt (xt) utUt (xt) accordingly p= {m t (); t= 0,1,…,h} step costOptimal expected Cost-to-go

SDP algorithm Pros: 1.It guarantees the best solution (provided assumptions are satisfied) Cons: 1.Only one solution per run!

J1J1 J2J2 Only one solution !

Gestione delle Risorse Naturali, Politecnico di Milano Stochastic Dynamic Programming Stochastic Dynamic Programming (SDP) suffers from a dual curse: 1) computational cost grows exponentially with state, control and disturbance dimension (curse of dimensionality [Bellman, 1967]); Look-up table H-function unknown H-function computations are numerically performed on a discretized variable domain 2) a dynamic model of any variable considered among the operating rule’s arguments has to be embedded in the algorithm (curse of modelling [Bertsekas and Tsitsiklis, 1996]). time t t + 1 models are use in a multiple one-step- ahead-simulation mode

Number of iterations for 1 reservoir: 10 1 x 80 1 x 5 2 x (365) x 3 = 22 x 10 6 x 3 Time per evaluation: 9 x sec. Total time: 3 minutes

Number of iterations for RTBR system: 10 4 x 80 4 x 5 5 x (365) x 3 = 1.4 x x 3 Time per evaluation: 3.7 x sec. Total time: 1,650,000 years!

Gestione delle Risorse Naturali, Politecnico di Milano Stochastic Dynamic Programming Stochastic Dynamic Programming (SDP) suffers from a dual curse: 1) computational cost grows exponentially with state, control and disturbance dimension (curse of dimensionality [Bellman, 1967]); Look-up table H-function unknown H-function computations are numerically performed on a discretized variable domain 2) a dynamic model of any variable considered among the operating rule’s arguments has to be embedded in the algorithm (curse of modelling [Bertsekas and Tsitsiklis, 1996]). time t t + 1 models are use in a multiple one-step- ahead-simulation mode

Design algorithm Genetic Algorithm

4th December 2013 GA are search methods based on two principles inspired by nature: WHAT ARE GENETIC ALGORITHMS? Genetics = recombination of structures Natural Selection = survival of the fittest

The Design Problem (It)(It) scenario (I t, θ) scenario

4th December 2013 Gestione delle Risorse Naturali, Politecnico di Milano Universal function approximators Artificial Neural Networks with some particular features can be used as universal function approximators, i.e. as policies. Multi-layer Perceptron u 1,t u q,t θ = [γ 1 1,1, …., γ 1 m,n, …, β L 1, …, β L q ]

4th December 2013 SOLVING APPROACH:  ANN to describe the control law ;  GA to find the optimal ANN parameterization. ALGORITHM: Gestione delle Risorse Naturali, Politecnico di Milano Run a system simulation for each individual Selection, crossover and mutation new population initial population time series of historical inflow objectives

J1J1 J2J2 Initial (random) population

J1J1 J2J2 selection of the “best” solutions according to the Pareto dominance criterion

J1J1 J2J2 survival of the fittest

J1J1 J2J2 generation of a new population

J1J1 J2J2 selection of the “best” solutions according to the Pareto dominance criterion

J1J1 J2J2 survival of the fittest

J1J1 J2J2 iterating….

J1J1 J2J2

J1J1 J2J2

J1J1 J2J2 final approximation of the Pareto front

GA algorithm Pros: 1.The whole Pareto boundary is generated in one run Cons: 1.It does not guarantees the best solution, neither an asymptotic convergence

Time per policy evaluation over 39 years for the RTBR system: 0.53 sec. Dim θ = (2 x N input + N output ) x N neur N neur ≥ N input + N output N input = 4+2 N output = 4 N neur = 10 Dim θ = 160 Num policies = reservoirs N input = 3+2 N output = 3 N neur = 9 Dim θ = 117 Num policies = reservoirs N input = 1+2 N output = 1 N neur = 5 Dim θ = 35 Num policies = reservoir Too large! Might be feasible Running time: 29 days Num evaluations about SDP 250 seconds = 470 policy evaluations SDP is surely faster How to reduce the number of reservoirs to 3 only? We will see tomorrow.

GA with extreme events

Design scenario 20 normal years 10 extreme years regular indicators extreme indicators J F, J S, J H ….. J eF, J eS ↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓ ↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓

Extreme events

Pareto boundary (qualitative) Flood I HP1: Hydrop. Production Extreme floods Irrigation

Extreme vs regular floods Regular Flood Extreme floods Irrigation

Trade off between extreme and standard floods

Hoa Binh and Ha Noi flooding t r,ar,a A A It is feasible only when A <C A flood of volume A is coming. How to minimize flooding in Ha Noi? Catch. C a r inflow a C Capacity release r flooding threshould HN HB

Hoa Binh and Ha Noi flooding t r,ar,a C A flood of volume A is coming. How to minimize flooding in Ha Noi? C a r inflow a C Capacity release r flooding threshould HN HB If A> C the spillway starts acting If A> C the spillway starts acting Flooding! What can we do?

Hoa Binh and Ha Noi flooding t r,ar,a A flood of volume A is coming. How to minimize flooding in Ha Noi? C a r inflow a C Capacity release r flooding threshould HN HB Intentionally produce a small flood! What if the big flood doesn’t arrive? C We have flooded for nothing!

Thanks for your attention XIN CẢM ƠN

A51A87A20 H22 20 I F extF F>

A51A87A20 H22 20 I F extF F>

A51A87A20 H22 20 I F extF F>

4th December 2013 Gestione delle Risorse Naturali, Politecnico di Milano The evaluation scheme a (m3/s) r (m3/s) s (m3) q_YB (m3/s) q_HY (m3/s) h_HN (m) q_ST (m3/s) g_hyd (kwh) g_flo (cm) Hydropower plant (conceptual) Flow routing (data-driven) flooding cost deficit cost g_sup (m3/s) 2 2 Reservoirs model (conceptual) hydropower cost P (kwh) u (m3/s)

Gestione delle Risorse Naturali, Politecnico di Milano Universal Approximation Theorem (Cybenko 1989, Funahashi 1989, Hornik et al. 1989) Every continuous function defined on a closed and bounded set can be approximated arbitrarily closely by a Multi-Layer Perceptron, provided that the number n of neurons in the hidden layers is sufficiently high and that their activation function belongs to a restricted class of functions with particular properties. Precisely,  must be differentiable and monotonically increasing;  the input to the j-th neuron (denoted with ) must enjoy the following property: Universal function approximators Sigmoidal functions meet both the requirements. e.g., the hyperbolic tangent is a sigmoidal function:

Gestione delle Risorse Naturali, Politecnico di Milano Universal Approximation Theorem (Cybenko 1989, Funahashi 1989, Hornik et al. 1989) Every continuous function defined on a closed and bounded set can be approximated arbitrarily closely by a Multi-Layer Perceptron, provided that the number n of neurons in the hidden layers is sufficiently high and that their activation function belongs to a restricted class of functions with particular properties. Universal function approximators In practice, a 2-layer perceptron is enough output parameters