1. Agent-Organized Networks for Dynamic Team Formation Multi-Agent Planning and Learning Laboratory (MAPLE) Department of Computer Science and EE University of Maryland Baltimore County Matt Gaston mgasto1@cs.umbc.edu Marie desJardins mariedj@cs.umbc.edu

2. Agent-Organized Networks 2 Overview  Introduction and Motivation  Team Formation  Agent-Organized Networks  Experimental Results  Related Projects:  Connections Model AONs  AONs for Production and Exchange  Stable Team Formation  Future Work and Conclusions

3. Agent-Organized Networks 3 Introduction and Motivation

4. Agent-Organized Networks 4 Introduction: Multi-Agent Systems  Agent: Autonomous, intelligent software system.  Physical (robot, autonomous vehicle, mobile sensor) or virtual (search, travel planning, trading / e-commerce, information retrieval)  MAS: “Community” of agents – competitive or cooperative  Connections form a “social network” of agents

5. Agent-Organized Networks 5 Why Adapt?  Multi-agent systems are growing in popularity and size  Technologies like the Semantic Web support the deployment and evolution of large-scale, dynamic multi- agent systems  Agent cognitive capacities are limited, preventing all agents from knowing/interacting with all other agents  Previous findings suggest that network structure plays an essential role in understanding team formation dynamics in multi-agent systems  Identifying the “best” network structure is difficult or impossible to do a priori  Solution: Agent-Organized Networks

6. Agent-Organized Networks 6 Team Formation [AAMAS 2005]

7. Agent-Organized Networks 7 Multi-Agent Team Formation Model  Agents must form teams to complete tasks  Agent states:  Uncommitted  Committed  Active  Tasks are advertised to the network of agents  A valid team:  Connected path in network  Task skill requirements met  Formed within time constraints 11 2 2 2 2 2 3 1, 1, 2, 3 2, 2 1, 2, 2, 2, 4 4

8. Agent-Organized Networks 8 Multi-Agent Team Formation Model  Some model details  Parameters Number of agents: N Skill diversity:  Task introduction interval:  Team/task size: T Advertisement duration:  Task duration:  Network structure  Organizational Efficiency # of tasks successfully completed total # of tasks advertised efficiency = 11 2 2 2 2 2 3 4

9. Agent-Organized Networks 9 Team Joining Strategy With some initiation probability, start a new team if needed: Always join a team if it’s already been started, and it needs your skill. Considering each task in random order...

10. Agent-Organized Networks 10 Agent-Organized Networks

11. 11 Agent-Organized Networks  Definition: An agent-organized network (AON) is an organizational network structure, or agent-to-agent interaction topology, that is the result of local rewiring decisions made by the individual agents in a networked multi-agent system.  Design considerations:  Local perception of global performance  Adaptation triggers  Rewiring strategies  Evaluation metrics:  Learning rate  Stability  Structural properties of resulting networks

12. Agent-Organized Networks 12 Structure-Based Adaptation  Adapt based on preferential attachment  Natural network formation process that leads to scale-free networks  Adaptation trigger (random):  Probability of adaptation for each uncommitted agent: 1/N  Rewiring strategy:  Disconnect from a random neighbor  Connect to some neighbor’s neighbor with probability

13. Agent-Organized Networks 13 Performance-Based Adaptation  Adaptation trigger:  Adapt if performance drops below neighbors’ average performance:  Rewiring strategy:  Drop the lowest-performing neighbor:  Add a connection to the highest-performing neighbor a k of the highest-performing neighbor a l:

14. Agent-Organized Networks 14 Results

15. Agent-Organized Networks 15 Experimental Setup  Initial network structure: Random geometric graph  Randomly place agents in a unit square  Connect agents that are closer than d units apart  Use the minimal d that guarantees all neighbors have at least one edge  Run team formation with no adaptation to establish baseline  Run with each adaptation strategy separately  Results are an average of 50 runs

16. Agent-Organized Networks 16 Results: Summary  Significant performance improvement (over baseline) for both AON methods

17. Agent-Organized Networks 17 Stability of Networks  Structure-based AONs outperform performance-based AONs, but result in substantially more rewirings  Performance-based AONs are more efficient (“better value” if adaptation cost is in similar units to performance measure)

18. Agent-Organized Networks 18 Evolution of the Network: Structure-Based  Converges to a network with hub structure and short average path length

19. Agent-Organized Networks 19 Evolution of the Network: Performance-Based  Convergence to short-average-path-length structure happens more slowly  Qualitatively similar structure to strategy-based (but in this case not by design!)

20. Agent-Organized Networks 20 Connections Model AONs [AAAI 2005 Workshop on Multi-Agent Learning]

21. Agent-Organized Networks 21 The (Symmetric) Connections Model  Symmetric when  ij =  and c ij = c for all i and j  0 <  < 1 is the value of a relationship, discounted by distance  c is the cost of a direct connection (Jackson & Wolinsky 1996; Jackson 2002)

22. Agent-Organized Networks 22 Dynamic Network Formation in SCM Based on pairwise stability (Watts 2001):  At each iteration:  Two agents meet (are selected) at random (synchronous)  If they have a connection, they remove the connection if at least one of them benefits -- unilateral deletion  If the do not have a connection, they add a connection if it is mutually beneficial -- bilateral creation But...

23. Agent-Organized Networks 23 Experiment: Watts Dynamic Network Formation  = 0.9, c = 0.8, optimal = 7878.42

24. Agent-Organized Networks 24 A (Simple) Multi-Agent Learning Approach  Goals:  Eliminate need for “global” knowledge  Eliminate need for “global” computation  Maintain bilateral network formation (agents agree to create link)  Follow dynamic network formation process of Watts  On-line learning  Approach  Stateless Q-Learning (Claus & Boutilier 1998)  A = { add, delete, nothing }  Agents add connection if both have largest Q value for add (bilateral)  Agents remove connection if one has largest Q value for delete (unilateral)  Reinforcement signal comes from omniscient oracle (!) Agent-Organized Networks (AONs)“Distributed Annealing”

25. Agent-Organized Networks 25 Experiment: Learning to Form Networks Adaptive Learning Rate: Win or Lose Fast (WoLF) (Bowling & Veloso 2002)  = 0.9, c = 0.8, optimal = 7878.42

26. Agent-Organized Networks 26 Experiment: Adding an Unselfish Agent  = 0.9, c = 0.8, optimal = 7878.42

27. Agent-Organized Networks 27 AONs for Production and Exchange [AAAI 2005]

28. Agent-Organized Networks 28 A Model of Production and Exchange  n agents in an artificial economy with two goods  Each agent i possesses g 1 i units of good 1 and g 2 i units of good 2  Each agent is a producer of either good 1 or good 2  At each iteration of the model, the agents are selected in random order and choose between initiating trade with another agent or producing their respective good in order to maximize utility  Agent utility: (Wilhite 2001: 2003) fully rational behavior

29. Agent-Organized Networks 29 Push Referral AON Strategies  Random referral: agent selected randomly from N j (i)  Degree referral:  Production referral: Definition: Assuming that agent i is adapting its connection to agent j, a push referral is a local rewiring by i from j to an agent in N j (i)

30. Agent-Organized Networks 30 Results production referral degree referral random selection n = 400, q = 30,  = 0.05  =  = 0.1  =  = 0.1 initialized values to 1

31. Agent-Organized Networks 31 Stable Team Formation [AAAI 2004 Workshop on Team/Coalition Formation]

32. Agent-Organized Networks 32 Economic Model of Team Formation  Share-based scheme for pay-off distribution  Team’s revenue is stored in “team account”  Team members get shares for joining and working  Share value = team account / # outstanding shares  Agents bound to the team by a contract  Joining Shares, S join : sign-on bonus  Commission, S comm : shares given to the agent for every task completed by the team in which the agent actively participates  Dividend, S div : shares given to the agent for every task completed by the team in which the agent does not participate (Dividend < Commission)  Penalty, p : the amount to be paid to the team when leaving the team

33. Agent-Organized Networks 33 Results: Effect of Deadlines

34. Agent-Organized Networks 34 Results: Stable vs. Dynamic Agents

35. Agent-Organized Networks 35 Conclusions and Future Work Summary:  AONs based only on local knowledge can improve team formation in networked MAS  AON ideas can also be applied to other MAS domains and models  Stability can be achieved through a contractual model of team formation Future Work:  Quantitative analysis of post-adaptation network structures  Learning individual agent team selection strategies  [JAAMAS 2006]  Skill placement and replacement for dynamic team formation

36. Agent-Organized Networks 36