1 Ann Nowé Nature inspired agents to handle interaction in IT systems Ann Nowé Computational modeling Lab Vrije Universiteit Brussel

2 Ann Nowé A challenging multi-agent system Routing in telecom networks Non-stationary state dependent Highly distributed Communication cost Competition - Collaboration

3 Ann Nowé Road map Agent Technology: ‘Agent based computing is probably the most important new paradigm for software development since object orientated programming.’ Agent technology three perspectives. Agents as Design Metaphor. Agents as a Simulation. Agents as a source of Technologies. Computing as Interaction, [M Luck, et.al.]

4 Ann Nowé Multi-agent Learning   I R S A S A

5 Ann Nowé Single agent RL in Markovian environment : convergence to optimal policy is guaranteed MAS RL : convergence to Pareto Optimal Nash Equilibrium not guaranteed But (Narendra and Wheeler, 1989) Players in an n-person non-zero sum game who use independently a reward-inaction update scheme with an arbitrarily small step size will always converge to one of the equilibrium points. Which equilibrium point is reached depends on the initial conditions. Background theory

6 Ann Nowé The Homo Egualis society => Coordinated exploration Payoff i is doing better than j i is doing worse than j

7 Ann Nowé Conflicting Interest games: periodical policies

8 Ann Nowé Non-communication period : agents are selfish RL. Exploration: agents can exclude actions from their private action space  other Nash equilibria can be exploited. Coordinated Exploration: 2 phases Selfish RL agents, with social rules

9 Ann Nowé ● Master-slave software ● Coarse granular hardware ● Heterogeneous nodes ● Improve parallel efficiency using LA ● Communication bottleneck ● Every computing node has LA ● IR = 1/ blocking time ● LA learns the amount of work to request ● Results: ● Computing time -39% ● Blocking time -62% MAS & parallel computing

10 Ann Nowé More on LA LA can solve MDP’s 1 LA per state Information is shared between states in an ant-like way LA games (tree structured Markov games) n LA per state, with n number of agents Monte-Carlo updates LA for general Markov Games is still a research topic.

11 Ann Nowé Multi-type ACO  Multiple ant colonies are used.  Each colony has their own type of pheromone  Ants are attracted by their own type of pheromone but repelled by other types.  Each colony converges to their own path, disjoint with paths of other colonies.  Multiple ant colonies are used.  Each colony has their own type of pheromone  Ants are attracted by their own type of pheromone but repelled by other types.  Each colony converges to their own path, disjoint with paths of other colonies.

12 Ann Nowé Conclusion We focus on simple agents with simple learning rules. Nature inspired : social system & ant’s viewpoint A general framework to study the dynamics of these new RL techniques is needed in order to justify their use and show their robustness. Only then they will find their way to real-world future IT applications.