1. SWAN 2006, ARRI_ Control of Distributed-Information Nonlinear Stochastic Systems Prof. Thomas Parisini University of Trieste

2. SWAN 2006, ARRI Summary Examples Problem Formulation Approximate NN Solution Two Significant Applications

3. SWAN 2006, ARRI Example: Water Distribution Networks Objective: control the spatio-temporal distribution of drinking water disinfectant throughout the network by the injection of appropriate amount of disinfectant at appropriately chosen actuator locations

4. SWAN 2006, ARRI Example: Robot Soccer Objective: By the year 2050, develop a team of fully autonomous humanoid robots that can win against the human world soccer champions. Robot soccer incorporates various technologies including: design principles of autonomous agents, multi-agent cooperation, strategy acquisition, real-time reasoning, robotics, and sensor-fusion.

5. SWAN 2006, ARRI Typical Applications Traffic control systems Coordination of teams of UAVs Geographically distributed systems Dynamic routing in communications networks Large-scale process-control systems Coordination of teams of UAVs etc.

6. SWAN 2006, ARRI Distributed Decision and Control How did this happen ? Outer-loop control: as we move from low-level control to higher-level control, we face the need to take into consideration other feedback systems that influence the environment. Complexity: due to the presence of more complex systems, we had to break down the controller into smaller controllers. Communications and Networks: data networks and wireless communications have facilitated the design of feedback systems that require distributed decision and control techniques.

7. SWAN 2006, ARRI Distributed Decision and Control What does it imply ? Need for different type of control problem formulations. Need to handle competition from other controllers (agents). Need to handle cooperation with other controllers Need to handle inter-controller communication issues. Need for suitable individual and team evaluation methods

8. SWAN 2006, ARRI Distributed Decision and Control How does Learning come in ? Learning the environment Learning the strategy of adversarial agents Learning is crucial because it is a highly time-varying (evolving) environment. Predicting the actions of collaborating agents

9. SWAN 2006, ARRI Structure of a “Team” of Agents

10. SWAN 2006, ARRI Problem formulation Definitions Unpredictable variables: Random vector representing all uncertainties in the external world with known p.d.f.

11. SWAN 2006, ARRI Problem formulation Definitions Information function:

12. SWAN 2006, ARRI Problem formulation Definitions Decision function:

13. SWAN 2006, ARRI Problem formulation Definitions Cost functional:

14. SWAN 2006, ARRI Problem formulation Problem T Given and Find the optimal strategies that minimize Solving analytically Problem T is in general impossible

15. SWAN 2006, ARRI About Problem T Further Definitions and Concepts Predecessor: The control actions generated by affect the information set of for any possible

16. SWAN 2006, ARRI About Problem T Further Definitions and Concepts Information set inclusion: The information set is a function of the information set ( is “nested” in )

17. SWAN 2006, ARRI About Problem T Further Definitions and Concepts Information network:

18. SWAN 2006, ARRI About Problem T Further Definitions and Concepts Information Structures Static: information of each is not influenced by decisions of other Dynamic: otherwise

19. SWAN 2006, ARRI About Problem T Sufficient conditions for analytic solution LQG Static Teams Linear optimal control strategy

20. SWAN 2006, ARRI About Problem T Sufficient conditions for analytic solution “Partially Nested” LQG Dynamic Teams Any can reconstruct the information of the influencing its own information Linear optimal control strategy

21. SWAN 2006, ARRI About Problem T Sufficient conditions for analytic solution Existence of a sequential partition Optimal control strategy by DP

22. SWAN 2006, ARRI Approximate Solution of Problem T A Simple Methodology Assumption: no loops in the information network Given parametric structure Vector of “free” parameters to be optimized

23. SWAN 2006, ARRI Formulation of the Approximate Problem Problem T’ Substitute and into the cost function

24. SWAN 2006, ARRI Formulation of the Approximate Problem Problem T’ Given and Find the optimal vector that minimizes Given NN structures Functional Optimization Problem T Nonlinear Programming Problem T’

25. SWAN 2006, ARRI NN Learning Algorithm Gradient Method cannot be written in explicit form However:

26. SWAN 2006, ARRI NN Learning Algorithm Stochastic Approximation Compute the “realization”

27. SWAN 2006, ARRI NN Learning Algorithm Stochastic Approximation randomly generated according to the (known) p.d.f. of The step is chosen so as Example:

28. SWAN 2006, ARRI NN Learning Algorithm Important Remark Gradient method + Stochastic Approximation Distributed Learning: each DM is able to compute “locally” its control function by “exchanging messages” with cooperating DMs according to the Information Structure

29. SWAN 2006, ARRI Methodology: Conceptual Steps Problem T: minimize Exact optimal solutions Replace with the NN structure Problem T’: minimize Stoc. Appr. to solve Problem T’ Approximate optimal solutions

30. SWAN 2006, ARRI The Witsenhausen Counterexample Problem W Given: and independent Find the optimal strategies that minimize information functions: cost function:

31. SWAN 2006, ARRI The Witsenhausen Counterexample Problem W

32. SWAN 2006, ARRI The Witsenhausen Counterexample Remarks on Problem W LQG hypotheses hold Information structure not partially nested An optimal solution does exist But: are not affine functions of

33. SWAN 2006, ARRI The Witsenhausen Counterexample Remarks on Problem W Best affine solution: Wit. solution: For and

34. SWAN 2006, ARRI The Witsenhausen Counterexample Remarks on Problem W Optimized Wit. solution: For and given the structures

35. SWAN 2006, ARRI The Witsenhausen Counterexample Remarks on Problem W Opt. Wit. solution outperforms the best linear solutions:

36. SWAN 2006, ARRI Approximate NN Solution of Problem W Given parametric structures Vector of “free” parameters to be optimized Choice of the parametric structures

37. SWAN 2006, ARRI Approximate NN Solution of Problem W Problem W’ Substitute into the cost functional

38. SWAN 2006, ARRI Approximate NN Solution of Problem W Problem W’ Given and independent Find the optimal NN weights that minimize information functions: cost function:

39. SWAN 2006, ARRI Conceptual Steps to Solve Approximately Problem W Problem W: minimize Exact optimal solutions Replace with the NNs Problem W’: minimize Stoc. Appr. to solve Problem W’ Approximate optimal solutions

40. SWAN 2006, ARRI Approximate NN Solution of Problem W Results and Comparisons Best Linear NN Opt. W. Best Linear NN Opt. W.

41. SWAN 2006, ARRI Approximate NN Solution of Problem W Results and Comparisons Best Linear NN Opt. W. Best Linear NN Opt. W.

42. SWAN 2006, ARRI Approximate NN Solution of Problem W Results and Comparisons Best Linear NN Opt. W. Best Linear NN Opt. W.

43. SWAN 2006, ARRI Approximate NN Solution of Problem W Results and Comparisons Best Linear NN Opt. W. Best Linear NN Opt. W.

44. SWAN 2006, ARRI Approximate NN Solution of Problem W Results and Comparisons Best Linear NN Opt. W. Best Linear NN Opt. W.

45. SWAN 2006, ARRI Approximate NN Solution of Problem W Results and Comparisons The “3-step” areaThe “linear” areaThe “5-step” area

46. SWAN 2006, ARRI Approximate NN Solution of Problem W Results and Comparisons Costs of the Neural, Optimized Witsenhausen, and Best Linear Solutions

47. SWAN 2006, ARRI Concluding Remarks Decision makers act as cooperating members of a team General approximate methodology for the solution of distributed-information control problems: Team functional optimization problem reduced to a nonlinear programming one Distributed learning scheme: each DM can compute (or adapt) its “personal” control function “locally” Straightforward extension to the infinite-horizon case (receding-horizon paradigm)

48. SWAN 2006, ARRI Acknowledgments Riccardo Zoppoli, Marios Polycarpou, Marco Baglietto, Angelo Alessandri, Alessandro Astolfi, Daniele Casagrande, Riccardo Ferrari, Elisa Franco, Frank Lewis, R. Selmic, Jason Speyer, Marcel Staroswiecki, Jakob Stoustrup, …