1. Hanoi, January 28 th 2015 Rodolfo Soncini-Sessa DEI – Politecnico di Milano IMRR Project 8 – Design algorithms INTEGRATED AND SUSTAINABLE WATER MANAGEMENT OF RED-THAI BINH RIVER SYSTEM IN A CHANGING CLIMATE

2. IMRR phases econnaissance odeling the system ndicators identification cenarios definition lternative design valuation R M I S A E Soncini Sessa, 2007 omparison … C

3. The Design Problem (It)(It) scenario

4. Design algorithm SDP Stochastic Dynamic Programming

5. The Design Problem (It)(It) scenario

6. Assumptions The objectives are separable Compensation is acceptable then

7. The Design Problem for SDP If  t+1 is a white process (It)(It) scenario If  t+1 is a white process and we do not consider exogenous information … If  t+1 is a white process and we do not consider exogenous information … then Stochastic Dynamic Programming (SDP)

8. SDP algorithm x t+1 = f t (x t,u t,  t+1 )  t+1 ~  t ( ) utUt (xt) utUt (xt) accordingly p= {m t (); t= 0,1,…,h} step costOptimal expected Cost-to-go

9. SDP algorithm Pros: 1.It guarantees the best solution (provided assumptions are satisfied) Cons: 1.Only one solution per run!

10. J1J1 J2J2 Only one solution !

11. Gestione delle Risorse Naturali, Politecnico di Milano Stochastic Dynamic Programming Stochastic Dynamic Programming (SDP) suffers from a dual curse: 1) computational cost grows exponentially with state, control and disturbance dimension (curse of dimensionality [Bellman, 1967]); Look-up table H-function unknown H-function computations are numerically performed on a discretized variable domain 2) a dynamic model of any variable considered among the operating rule’s arguments has to be embedded in the algorithm (curse of modelling [Bertsekas and Tsitsiklis, 1996]). time t t + 1 models are use in a multiple one-step- ahead-simulation mode

12. Number of iterations for 1 reservoir: 10 1 x 80 1 x 5 2 x (365) x 3 = 22 x 10 6 x 3 Time per evaluation: 9 x 10 -6 sec. Total time: 3 minutes

13. Number of iterations for RTBR system: 10 4 x 80 4 x 5 5 x (365) x 3 = 1.4 x 10 18 x 3 Time per evaluation: 3.7 x 10 -5 sec. Total time: 1,650,000 years!

14. Gestione delle Risorse Naturali, Politecnico di Milano Stochastic Dynamic Programming Stochastic Dynamic Programming (SDP) suffers from a dual curse: 1) computational cost grows exponentially with state, control and disturbance dimension (curse of dimensionality [Bellman, 1967]); Look-up table H-function unknown H-function computations are numerically performed on a discretized variable domain 2) a dynamic model of any variable considered among the operating rule’s arguments has to be embedded in the algorithm (curse of modelling [Bertsekas and Tsitsiklis, 1996]). time t t + 1 models are use in a multiple one-step- ahead-simulation mode

15. Design algorithm Genetic Algorithm

16. 4th December 2013 GA are search methods based on two principles inspired by nature: WHAT ARE GENETIC ALGORITHMS? Genetics = recombination of structures Natural Selection = survival of the fittest

17. The Design Problem (It)(It) scenario (I t, θ) scenario

18. 4th December 2013 Gestione delle Risorse Naturali, Politecnico di Milano Universal function approximators Artificial Neural Networks with some particular features can be used as universal function approximators, i.e. as policies. Multi-layer Perceptron u 1,t u q,t θ = [γ 1 1,1, …., γ 1 m,n, …, β L 1, …, β L q ]

19. 4th December 2013 SOLVING APPROACH:  ANN to describe the control law ;  GA to find the optimal ANN parameterization. ALGORITHM: Gestione delle Risorse Naturali, Politecnico di Milano Run a system simulation for each individual Selection, crossover and mutation new population initial population time series of historical inflow objectives

20. J1J1 J2J2 Initial (random) population

21. J1J1 J2J2 selection of the “best” solutions according to the Pareto dominance criterion

22. J1J1 J2J2 survival of the fittest

23. J1J1 J2J2 generation of a new population

24. J1J1 J2J2 selection of the “best” solutions according to the Pareto dominance criterion

25. J1J1 J2J2 survival of the fittest

26. J1J1 J2J2 iterating….

27. J1J1 J2J2

28. J1J1 J2J2

29. J1J1 J2J2 final approximation of the Pareto front

30. GA algorithm Pros: 1.The whole Pareto boundary is generated in one run Cons: 1.It does not guarantees the best solution, neither an asymptotic convergence

31. Time per policy evaluation over 39 years for the RTBR system: 0.53 sec. Dim θ = (2 x N input + N output ) x N neur N neur ≥ N input + N output N input = 4+2 N output = 4 N neur = 10 Dim θ = 160 Num policies = 10 160 4 reservoirs N input = 3+2 N output = 3 N neur = 9 Dim θ = 117 Num policies = 10 117 3 reservoirs N input = 1+2 N output = 1 N neur = 5 Dim θ = 35 Num policies = 10 35 1 reservoir Too large! Might be feasible Running time: 29 days Num evaluations about 5.5 10 6 SDP 250 seconds = 470 policy evaluations SDP is surely faster How to reduce the number of reservoirs to 3 only? We will see tomorrow.

32. GA with extreme events

33. Design scenario 20 normal years 10 extreme years regular indicators extreme indicators J F, J S, J H ….. J eF, J eS ↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓ ↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓

34. Extreme events

35. Pareto boundary (qualitative) Flood I HP1: Hydrop. Production Extreme floods Irrigation

36. Extreme vs regular floods Regular Flood Extreme floods Irrigation

37. Trade off between extreme and standard floods

38. Hoa Binh and Ha Noi flooding t r,ar,a A A It is feasible only when A <C A flood of volume A is coming. How to minimize flooding in Ha Noi? Catch. C a r inflow a C Capacity release r flooding threshould HN HB

39. Hoa Binh and Ha Noi flooding t r,ar,a C A flood of volume A is coming. How to minimize flooding in Ha Noi? C a r inflow a C Capacity release r flooding threshould HN HB If A> C the spillway starts acting If A> C the spillway starts acting Flooding! What can we do?

40. Hoa Binh and Ha Noi flooding t r,ar,a A flood of volume A is coming. How to minimize flooding in Ha Noi? C a r inflow a C Capacity release r flooding threshould HN HB Intentionally produce a small flood! What if the big flood doesn’t arrive? C We have flooded for nothing!

41. Thanks for your attention XIN CẢM ƠN

42. A51A87A20 H22 20 I330316124 F9088106 extF397328441490 F>13.48142300

43. A51A87A20 H22 20 I330316124 F9088106 extF397328441490 F>13.48142300

44. A51A87A20 H22 20 I330316124 F9088106 extF397328441490 F>13.48142300

47. 4th December 2013 Gestione delle Risorse Naturali, Politecnico di Milano The evaluation scheme a (m3/s) r (m3/s) s (m3) q_YB (m3/s) q_HY (m3/s) h_HN (m) q_ST (m3/s) g_hyd (kwh) g_flo (cm) Hydropower plant (conceptual) Flow routing (data-driven) flooding cost deficit cost g_sup (m3/s) 2 2 Reservoirs model (conceptual) hydropower cost P (kwh) u (m3/s)

48. Gestione delle Risorse Naturali, Politecnico di Milano Universal Approximation Theorem (Cybenko 1989, Funahashi 1989, Hornik et al. 1989) Every continuous function defined on a closed and bounded set can be approximated arbitrarily closely by a Multi-Layer Perceptron, provided that the number n of neurons in the hidden layers is sufficiently high and that their activation function belongs to a restricted class of functions with particular properties. Precisely,  must be differentiable and monotonically increasing;  the input to the j-th neuron (denoted with ) must enjoy the following property: Universal function approximators Sigmoidal functions meet both the requirements. e.g., the hyperbolic tangent is a sigmoidal function:

49. Gestione delle Risorse Naturali, Politecnico di Milano Universal Approximation Theorem (Cybenko 1989, Funahashi 1989, Hornik et al. 1989) Every continuous function defined on a closed and bounded set can be approximated arbitrarily closely by a Multi-Layer Perceptron, provided that the number n of neurons in the hidden layers is sufficiently high and that their activation function belongs to a restricted class of functions with particular properties. Universal function approximators In practice, a 2-layer perceptron is enough output parameters