1. Collaborative Reinforcement Learning Presented by Dr. Ying Lu

2. Credits Reinforcement Learning: A User’s Guide. Reinforcement Learning: A User’s Guide. Bill Smart at ICAC 2005 Jim Dowling, Eoin Curran, Raymond Cunningham and Vinny Cahill, "Collaborative Reinforcement Learning of Autonomic Behaviour", 2nd International Workshop on Self-Adaptive and Autonomic Computing Systems, pages 700-704, 2004. [Winner Best Paper Award].Collaborative Reinforcement Learning of Autonomic Behaviour

3. What is RL? “a way of programming agents by reward and punishment without needing to specify how the task is to be achieved” [Kaelbling, Littman, & Moore, 96]

4. Basic RL Model 1.Observe state, s t 2.Decide on an action, a t 3.Perform action 4.Observe new state, s t+1 5.Observe reward, r t+1 6.Learn from experience 7.Repeat Goal: Find a control policy that will maximize the observed rewards over the lifetime of the agent ASR World

5. An Example: Gridworld Canonical RL domain States are grid cells 4 actions: N, S, E, W Reward for entering top right cell -0.01 for every other move Maximizing sum of rewards  Shortest path In this instance +1

6. The Promise of RL Specify what to do, but not how to do it Through the reward function Learning “fills in the details” Better final solutions Based on actual experiences, not programmer assumptions Less (human) time needed for a good solution

7. Mathematics of RL Before we talk about RL, we need to cover some background material Some simple decision theory Markov Decision Processes Value functions

8. Making Single Decisions Single decision to be made Multiple discrete actions Each action has a reward associated with it Goal is to maximize reward Not hard: just pick the action with the largest reward State 0 has a value of 2 Sum of rewards from taking the best action from the state 0 1 2 A B 2 1

9. Markov Decision Processes We can generalize the previous example to multiple sequential decisions Each decision affects subsequent decisions This is formally modeled by a Markov Decision Process (MDP) 0 1 2 A B 2 1 5 3 4 A A -1000 1 A A 10 1 B 1

10. Markov Decision Processes Formally, an MDP is A set of states, S = {s 1, s 2,..., s n } A set of actions, A = {a 1, a 2,..., a m } A reward function, R: S  A  S→  A transition function, We want to learn a policy,  : S →A Maximize sum of rewards we see over our lifetime

11. Policies There are 3 policies for this MDP 1.0 →1 →3 →5 2.0 →1 →4 →5 3.0 →2 →4 →5 Which is the best one? 0 1 2 A B 2 1 5 3 4 A A -1000 1 A A 10 1 B 1

12. Comparing Policies Order policies by how much reward they see 1.0 →1 →3 →5 = 1 + 1 + 1 = 3 2.0 →1 →4 →5 = 1 + 1 + 10 = 12 3.0 →2 →4 →5 = 2 – 1000 + 10 = -988 0 1 2 A B 2 1 5 3 4 A A -1000 1 A A 10 1 B 1

13. Value Functions We can define value without specifying the policy Specify the value of taking action a from state s and then performing optimally This is the state-action value function, Q 0 1 2 A B 2 1 5 3 4 A -1000 1 A 10 1 B 1 Q(0, A) = 12 Q(0, B) = -988 Q(3, A) = 1 Q(4, A) = 10 Q(1, A) = 2 Q(1, B) = 11 Q(2, A) = -990 A A How do you tell which action to take from each state?

14. Value Functions So, we have value function Q(s, a) = R(s, a, s’) + max a’ Q(s’, a’) In the form of Next reward plus the best I can do from the next state These extend to probabilistic actions s’ is the next state

15. Getting the Policy If we have the value function, then finding the best policy is easy  (s) = arg max a Q(s, a) We’re looking for the optimal policy,   (s) No policy generates more reward than   Optimal policy defines optimal value functions The easiest way to learn the optimal policy is to learn the optimal value function first

16. Collaborative Reinforcement Learning to Adaptively Optimize MANET Routing Jim Dowling, Eoin Curran, Raymond Cunningham and Vinny Cahill

17. Overview Building autonomic distributed systems with self* properties Self-Organizing Self-Healing Self-Optimizing Add collaborative learning mechanism to self- adaptive component model Improved ad-hoc routing protocol

18. Introduction Autonomous distributed systems will consist of interacting components free from human interference Existing top-down management and programming solutions require too much global state Bottom up, decentralized collection of components who make their own decisions based on local information System wide self* behavior emerges from interactions

19. Self-* Behavior Self-adaptive components that change structure and/or behavior at run-time, adapt to discovered faults reduced performance Requires active monitoring of component states and external dependencies

20. Self-* Distributed Systems using Distributed (collaborative) Reinforcement Learning For complex systems, programmers cannot be expected to describe all conditions Self-adaptive behavior learnt by components Decentralized co-ordination of components to support system-wide properties Distributed Reinforcement Learning (DRL) is extension to RL and uses neighbor interactions only

21. Model-Based Reinforcement Learning 1.Action Reward 2. State Transition Model 3. Next State Reward Markov Decision Process = {States }, {Actions}, R(States,Actions), P (States, Actions, States)

22. Decentralised System Optimisation Coordinating the solution to a set of Discrete Optimisation Problems (DOPs) Components have a Partial System View Coordination Actions Actions ={delegation} U {DOP actions} U {discovery} Connection Costs

23. Collaborative Reinforcement Learning Advertisement Update Partial Views of Neighbours Decay Negative Feedback on State Values in the Absence of Advertisements Action Reward State Transition Model Cached Neighbour’s V-value Connection Cost

24. Adaptation in CRL System A feedback process to Changes in the optimal policy of any RL agent Changes in the system environment The passing time

25. SAMPLE: Ad-hoc Routing using DRL Probabilistic ad-hoc routing protocol based on DRL Adaptation of network traffic around areas of congestion Exploitation of stable routes Routing decisions based on local information and information obtained from neighbors Outperforms Ad-hoc On Demand Distance Vector Routing (AODV) and Dynamic Source Routing (DSR)

26. SAMPLE: A CRL System (I)

27. SAMPLE: A CRL System (II) Instead of always choosing the neighbor with the best Q value, i.e., taking the delegation action a= arg max a Q i (B, a), a neighbor is chosen probabilistically

28. SAMPLE: A CRL System (III) P i (s’|s, a j ) = E(C S /C A )

29. SAMPLE: A CRL System (IV)

30. Performance Metric: Maximize throughput ratio of delivered packets to undelivered packets Minimize number of transmission required per packet sent Figures 5-10

31. Questions/Discussions