1. 1

2. 2 Plan of the course u Introduction u Rules of Encounters u Strategic Negotiation u Auctions s protocols s strategies u Argumentation

3. 3 Machines Controlling and Sharing Resources u Electrical grids (load balancing) u Telecommunications networks (routing) u PDA’s (schedulers) u Shared databases (intelligent access) u Traffic control (coordination)

4. 4 Broad Working Assumption u Designers (from different companies, countries, etc.) come together to agree on standards for how their automated agents will interact (in a given domain) u Discuss various possibilities and their tradeoffs, and agree on protocols, strategies, and social laws to be implemented in their machines

5. 5 Attributes of Standards Efficient:Pareto Optimal Stable:No incentive to deviate Simple:Low computational and communication cost Distributed:No central decision-maker Symmetric:Agents play equivalent roles Designing protocols for specific classes of domains that satisfy some or all of these attributes

6. 6 u Distributed Problem Solving (DPS) — Centrally designed systems, built-in cooperation, have global problem to solve Multi-Agent Systems (MAS) — Group of utility-maximizing heterogeneous agents co-existing in same environment, possibly competitive Distributed Artificial Intelligence (DAI)

7. 7 Phone Call Competition Example u Customer wishes to place long-distance call u Carriers simultaneously bid, sending proposed prices u Phone automatically chooses the carrier (dynamically) AT&T MCI Sprint $0.20 $0.18 $0.23

8. 8 Best Bid Wins u Phone chooses carrier with lowest bid u Carrier gets amount that it bid AT&T MCISprint $0.20 $0.23 $0.18

9. 9 Attributes of the Mechanism Distributed Symmetric  Stable  Simple  Efficient AT&T MCISprint $0.20 $0.18 $0.23 Carriers have an incentive to invest effort in strategic behavior “Maybe I can bid as high as $0.21...”

10. 10 Best Bid Wins, Gets Second Price u Phone chooses carrier with lowest bid u Carrier gets amount of second-best price AT&T MCISprint $0.20 $0.18 $0.23

11. 11 Attributes of the Mechanism Distributed Symmetric Stable Simple Efficient AT&T MCISprint $0.20 $0.18 $0.23 Carriers have no incentive to invest effort in strategic behavior “I have no reason to overbid...”

12. 12 Database Domain Common Database “All female employees with more than three children.” 2 1 TOD “All female employees making over $50,000 a year.”

13. 13 Negotiation “A discussion in which interested parties exchange information and come to an agreement.” — Davis and Smith, 1977 u Two-way exchange of information u Each party evaluates information from its own perspective u Final agreement is reached by mutual selection

14. 14 Game Theory--Short Introduction u Game theory is the study of decision making in multi-person situations where the outcome depends on everyone’s choice. u In Decision Theory and the theory of competitive equilibrium from economics the other participants actions are considered as an environmental parameter. The effect of the of the decision- maker’s actions on the other participants is not taken into consideration.

15. 15 Describing a Game u Essential elements: players, actions, information, strategies, payoffs, outcome, and equilibria. u Ways to present social interactions as a game: u Extensive form:the most complete description. u Strategic form: many details are omitted. u Coalitional form: binding agreements exist.

16. 16 Example of two players game d india D sikh deal blow op 1212 0 2- 3- 0 2- 1-

17. 17 Nash Equilibrium u An action profile is an order set a=(a1,…,aN) of one action for each of the N players in the game. u An action profile a is a Nash Equilibrium (Nash 53) of a strategic game, if each agent j does not have a different action yielding an outcome that it prefers to that generated when chooses aj, given that every other player I chooses ai.

18. 18 c 0.6 0.4 Ind op -3,0- 2,1- dealH sik dealH blow3-,5 2,5 2,1- blow yes 3,4 Ind dealH op 1,4 4-,4

19. 19 Rules of Encounter Jeffrey S. Rosenschein Gilad Zlotkin

20. 20 Domain Theory u Task Oriented Domains  Agents have tasks to achieve  Task redistribution u State Oriented Domains  Goals specify acceptable final states  Side effects  Joint plan and schedules u Worth Oriented Domains  Function rating states’ acceptability  Joint plan, schedules, and goal relaxation

21. 21 Postmen Domain Post Office a c d e  2 1     TOD b f

22. 22 Database Domain Common Database “All female employees with more than three children.” 2 1 TOD “All female employees making over $50,000 a year.”

23. 23 Fax Domain faxes to send a c b d e f Cost is only to establishconnection 2 1 TOD

24. 24 Slotted Blocks World 12 3 12 3 SOD 2 1

25. 25 The Multi-Agent Tileworld 22 2 2 5 5 3 4 A B tile hole obstacle agents WOD

26. 26 Task Oriented Domain (TOD) A tuple where:  T is the set of all possible tasks • A = A 1, …, A n is a list of agents • c is a monotonic function c : [2 T ]   + An encounter is a list T 1,…, T n of finite sets of tasks from T such that agent A k needs to achieve all the tasks in T k (also called agent A k ’s goal ).

27. 27 Building Blocks Domain s A precise definition of what a goal is s Agent operations u Negotiation Protocol s A definition of a deal s A definition of utility s A definition of the conflict deal u Negotiation Strategy s In Equilibrium s Incentive-compatible

28. 28 Deal and Utility in two-agent TOD  Deal  is a pair (D 1, D 2 ): D 1  D 2 = T 1  T 2  Conflict deal:  = (T 1, T 2 )  Utility i (  ) = Cost(T i ) – Cost(D i )

29. 29 Negotiation Protocols u Agents use a product-maximizing negotiation protocol (as in Nash bargaining theory); u It should be a symmetric PMM (product maximizing mechanism); u Examples: 1-step protocol, monotonic concession protocol…

30. 30 Building Blocks Domain s A precise definition of what a goal is s Agent operations Negotiation Protocol s A definition of a deal s A definition of utility s A definition of the conflict deal u Negotiation Strategy s In Equilibrium s Incentive-compatible

31. 31 Negotiation with Incomplete Information a c bh f d g e What if the agents don’t know each other’s letters? Post Office 2   1 1 1 2

32. 32 –1 Phase Game: Broadcast Tasks Agents will flip a coin to decide who delivers all the letters. a c bh f d g e Post Office   1 1 2 2 1 e b, f

33. 33 Hiding Letters They then agree that agent 2 delivers to f and e. (hidden) a c bh f d g e Post Office   (1) 1 2 e b 2 1 f

34. 34 Another Possibility for Deception a c b They will agree to flip a coin to decide who goes to b and who goes to c. Post Office   b, c 2 1 1, 2

35. 35 Phantom Letter b, c, d Post Office 2 1 b, c a c b   1, 2 d  1 (phantom) They agree that agent 1 goes to c.

36. 36 Negotiation over Mixed Deals Theorem : With mixed deals, agents can always agree on the “all-or-nothing” deal Mixed deal (D 1, D 2 ) : p The agents will perform (D 1, D 2 ) with probability p, and the symmetric deal (D 2, D 1 ) with probability 1 – p

37. 37 Hiding Letters with Mixed All-or-Nothing Deals They will agree on the mixed deal where agent 1 has a 3/8 chance of delivering to f and e. (hidden) a c bh f d g e Post Office   (1) 1 2 e b 2 1 f

38. 38 Phantom Letters with Mixed Deals They will agree on the mixed deal where A has 3/4 chance of delivering all letters, lowering his expected utility. a c b b, c, d Post Office 2  1  b, c 1, 2 d  1 (phantom)

39. 39 Sub-Additive TODs TOD is sub-additive if for all finite sets of tasks X, Y in T we have: c(X  Y)  c(X) + c(Y)

40. 40 Sub-Additivity c (X  Y)  c (X) + c (Y) XY

41. 41 Sub-Additive TODs The Postmen Domain, Database Domain, and Fax Domain are sub-additive. The “Delivery Domain” (where postmen don’t have to return to the Post Office) is not sub-additive. 

42. 42 Incentive Compatible Mechanisms Sub-Additive a c b   1, 2 d  (phantom) 1 (hidden) a c bh fd g e   (1) 1 2 Theorem : For all encounters in all sub-additive TODs, when using a PMM over all-or-nothing deals, no agent has an incentive to hide a task. Hidden Pure LL A/N T T/P Mix LT/P Phantom

43. 43 Decoy Tasks Sub-Additive Hidden Pure LL A/N T T/P Mix LT/P Phantom L L L Decoy Decoy tasks, however, can be beneficial even with all-or-nothing deals      1 1 11 2 2 1

44. 44 Concave TODs TOD is concave if for all finite sets of tasks Y and Z in T, and X  Y, we have: c(Y  Z) – c(Y)  c(X  Z) – c(X) Concavity implies sub-additivity.

45. 45 Concavity XY Z The cost Z adds to X is more than the cost it adds to Y. (Z - X is a superset of Z - Y)

46. 46 Concave TODs The Database Domain and Fax Domain are concave (not the Postmen Domain, unless restricted to trees).      1 1 11 2 2 1 X Z This example was not concave; Z adds 0 to X, but adds 2 to its superset Y (all blue nodes).

47. 47 Three-Dimensional Incentive Compatible Mechanism Table Sub-Additive Hidden Pure LL A/N T T/P Mix LT/P Phantom L L L Decoy Concave Hidden Pure LL A/N TT Mix L T Phantom L T T Decoy Theorem : For all encounters in all concave TODs, when using a PMM over all-or- nothing deals, no agent has any incentive to lie.

48. 48 Modular TODs TOD is modular if for all finite sets of tasks X, Y in T we have: c(X  Y) = c(X) + c(Y) – c(X  Y) Modularity implies concavity.

49. 49 Modularity c(X  Y) = c(X) + c(Y) – c(X  Y) XY

50. 50 Modular TODs The Fax Domain is modular (not the Database Domain nor the Postmen Domain, unless restricted to a star topology). Even in modular TODs, hiding tasks can be beneficial in general mixed deals.

51. 51 Three-Dimensional Incentive Compatible Mechanism Table Sub-Additive Pure A/N Mix Concave Pure A/N Mix H LL TT L T P L T T D H LL T T/P L P L L L D Modular Pure A/N Mix H L T TT L T P T T T D

52. 52 Related Work u Coalitions Formations: Shehory, Sandholm u Mechanism design:Ephrati, Kraus, Tennenholtz u Other models of negotiation: Sycara, Durfee, Lesser, Gasser, Gmytrasiewicz, Jennings u Consensus mechanisms, voting techniques, economic models: Ephrati, Wellman, Sandholm

53. 53 Conclusions u By appropriately adjusting the rules of encounter by which agents must interact, we can influence the private strategies that designers build into their machines u The interaction mechanism should ensure the efficiency of multi-agent systems Rules of Encounter Efficiency

54. 54 Conclusions u To maintain efficiency over time of dynamic multi-agent systems, the rules must also be stable u The use of formal tools enables the design of efficient and stable mechanisms, and the precise characterization of their properties Stability Formal Tools

55. 55 Strategic Negotiation Collaborators: Jon Wilkenfeld, Rina Schwartz- Azoulay, Orna Shechter, Esti Freitsis

56. 56 DAI Overview AI DAI DPS MA strategic negotiation

57. 57 Strategic Negotiation Model u Model of alternative offers (Rubinstein) which takes negotiation time into consideration: reduces negotiation time. u During the strategic-negotiations agents communicate their respective desires to reach mutually beneficial agreement. u The model provides a unified to many problems.

58. 58 Structure of the Negotiation  There are N self motivated agents, randomly designated 1,2,...  All the agents negotiate to reach an agreement.  The negotiation process may include several equidistant iterations 0,1,2… ־ Time and can continue forever. In each time period t, agent j(t) =t mod N makes an offer.

59. 59 Structure of the Negotiation - cont.  The other agents respond simultaneously: YES 4 or NO 8 or OPT .  If the offer was accepted 4 by all the agents: the last offer is implemented.  If at least one agent opts out  : a conflict occurs.  Otherwise (the offer was rejected 8 by at least one agent), the negotiation proceeds to period t+1. ֲאֱ

60. 60 Applications u Information servers (large databases). u Resources sharing. u Tasks distribution. u Computer assisted negotiation. u Union/management negotiation.

61. 61 Negotiation on data allocation in multi-server environment

62. 62 Environment Description u There are several information servers. Each server is located at a different geographical area. u Each server receives queries from the clients in its area, and sends documents as responses to queries. These documents can be stored locally, or in another server.

63. 63 Environment Description server i server j a query document/s area i area j distance a client the document/s the query

64. 64 Environment Description - cont. u The information is clustered in datasets (corresponding to file, fragment, etc.) u Each new dataset has to be allocated to one of the servers by mutual agreement among the servers. u Each server wants to store the datasets in a location which reduces its communication and storage costs. u A negotiation session is initiated when a set of new datasets arrive.

65. 65 Motivation u Cooperation among servers with similar areas of interest (e.g., Web servers). u The Data and Information System component of the Earth Observing System (EOSDIS) of NASA: A distributed knowledge system which supports archival and distribution of data at multiple and independent servers.

66. 66 Motivation - cont. u Each data collection, or file, is called a dataset. The datasets are huge, so each dataset has only one copy. u The current policy for data allocation in NASA is static: old datasets are not reallocated; each new dataset is located by the server with the nearest topics (defined according to the topics of the datasets stored by this server).

67. 67 Related Work - File Allocation Problem The original problem: How to distribute files among computers, in order to optimize the system performance. Our problem: How can self-motivated servers decide about distribution of files, when each server has its own objectives.

68. 68 Basic Definitions u SERVERS: the set of the servers. u DATASETS: the set of datasets (files) to be allocated. u Allocation: a mapping of each dataset to one of the servers. The set of all possible allocation is denoted by Allocs. u U: the utility function of each server.

69. 69 The Conflict Allocation  If at least one server opts out  of the negotiation, then the conflict allocation conflict_alloc is implemented.  We consider the conflict allocation to be the static allocation. (each dataset is stored in the server with closest topics).

70. 70 Utility Function  U server (alloc,t) specifies the utility of server from alloc ־ Allocs at time t. u It consists of s The utility from the assignment of each dataset. s The cost of negotiation delay. U server (alloc,0)= S V server (x,alloc(x)). x ־ DATASETS

71. 71 Parameters of utility u query price: payment for retrieved docoments. u usage(ds,s): the expected number of documents of dataset ds from clients in the area of server s. u storage costs, retrieve costs, answer costs.

72. 72 Cost over time u Cost of communication and computation time of the negotiation. u Loss of unused information: new documents can not be used until the negotiation ends. u Datasets usage and storage cost are assumed to decrease over time, with the same discount ratio (p-1).  Thus, there is a constant discount ratio of the utility from an allocation: U server (alloc,t)=  t *U server (alloc,0) - t*C.

73. 73 Assumptions P Each server prefers any agreement over continuation of the negotiation indefinitely. P The utility of each server from the conflict allocation is always greater or equal to 0. P OFFERS - the set of allocations that are preferred by all the agents over opting out.

74. 74 Equilibrium Nash equilibrium: A strategy profile p is a Nash Equilibrium if no player has a different strategy yielding an outcome that he prefers to that generated when it chooses p i. Subgame Perfect Equilibrium: If the strategy profile induced in every subgame is a Nash Equilibrium of this subgame.

75. 75 Negotiation Analysis - Simultaneous Responses  Simultaneous responses: A server, when responding, is not informed of the other responses.  Theorem: For each offer x  ־  OFFERS, there is a subgame- perfect equilibrium of the bargaining game, with the outcome x offered and unanimously accepted in period 0.

76. 76 Choosing the Allocation u The designers of the servers can agree in advance on a joint technique for choosing x: u giving each server its conflict utility. u maximizing a social welfare criterion: s the sum of the servers’ utilities.  or the generalized Nash product of the servers’ utilities:  (Us(x)-Us(conflict)).

77. 77 Choosing the Allocation - cont. u The problem of finding an optimal allocation is NP-complete (a reduction from the multiprocessors scheduling). u When finding x is intractable, we suggest the following protocol: s each server will search for an allocation s the allocation which maximizes the predefined social welfare criterion will be chosen.

78. 78 Search Methods u We have implemented the following algorithms: s A backtracking algorithm: Searching the search space of the allocation problem. s A random restart hill-climbing algorithm: Starts with a random allocation and tries to improve it. s A genetic algorithm: Searching by simulating an evolution process. Each individual represents an allocation. The algorithm involves: reproduction, crossover and mutation of individuals.

79. 79 Experimental Evaluation u How do the parameters influence the results of the negotiation? u vcost(alloc): the variable costs due to an allocation (excludes storage_cost and the gains due to queries). u vcost_ratio: the ratio of vcosts when using negotiation, and vcosts of the static allocation.

80. 80 Effect of Parameters on The Results  As the number of servers grows, vcost_ratio increases ( more complex computations ) .  As the number of datasets grows, vcost_ratio decreases ( negotiation is more beneficial ).  Changing the mean usage did not influence vcost_ratio significantly , but vcost_ratio decreases as the standard deviation of the usage increases.

81. 81 Influence of Parameters - cont.  When the standard deviation of the distances between servers increases, vcost_ratio decreases.  When the distance between servers increases, vcost_ratio decreases. u In the domains tested,  answer_cost ס vcost_ratio ס .  storage_cost ס vcost_ratio ס .  retrieve_cost ס vcost_ratio ע .  query_price ס vcost_ratio ע .

82. 82 Social Criteria u We studied the effect of the choice of the social welfare criterion on the results. u We compare the following criteria: s Sum of agents’ utilities. s Product of agents’ utilities. u Maximizing the sum achieves lower vcost_ratio. u Maximizing the product achieves lower dispersion of the agents’ utilities.

83. 83 Incomplete Information u Each server knows: s The usage frequency of all datasets, by clients from its area. s The usage frequency of datasets stored in it, by all clients.

84. 84 Incomplete Information - cont. u A revelation mechanism: s First, all the servers report simultaneously all their private information: –for each dataset, the past usage of the dataset by this server. –for each server, the past usage of each local dataset by this server. s Then, the negotiation proceeds as in the complete information case.

85. 85 Incomplete Information - cont. u Lemma: There is a Nash equilibrium where each server tells the truth about its past usage of remote datasets, and the other servers usage of its local datasets. u Lies concerning details about local usage of local datasets are intractable.

86. 86 Summary: negotiation on data allocation u We have considered the data allocation problem in a distributed environment. u We have presented the utility function of the servers, which expresses their preferences. u We have proposed using a negotiation protocol for solving the problem. u For incomplete information situations, a revelation process was added to the protocol.

87. 87 Negotiations in the pollution sharing problem Collaborator: Esti Freitsis

88. 88 Environment Description u There are some closely grouped plants in an industrial region. u Each plant can produce several types of products. u Each plant has a utility function (profit). u There are several types of pollution substances.  Each plant has norms, restricting maximal emission of each polluting substance that it emits. The pollution always has to be below these norms. We refer to the situation when only these norms have to be carried out as usual circumstances.

89. 89 Special circumstances  Sometimes there is a need to reduce pollution for some period because of external factors such as weather (high humidity, wind towards residential area). In this case plants receive new norms. We refer to this situation as special circumstances.

90. 90 Current solution u Current solution: each plant reduce pollution according to the new norms. u Disadvantage: for one plant it is less costly to reduce one substance while for another it is less costly to reduce another substance.

91. 91 Negotiations u Our solution: plants negotiate to reach beneficial agreements about the emission of what substances and by which percent each of them must be reduced. s The conflict solution: following the new norms. s We consider complete information situations.

92. 92 Negotiations Protocols u Simultaneous responses: an agent responding to an offer is not informed of the other responses. u Sequential responses: an agent responding to an offer is informed of the responses of the preceding agents (assuming that the agents are ordered).

93. 93 Negotiations strategies for simultaneous responses u As in the data allocation case:  For each possible agreement x that is better to all the plants than the conflict solution there is a subgame-perfect equilibrium of the bargaining game, with the outcome x offered and unanimously accepted in period 0.

94. 94 Negotiations strategies for sequential responses u Assumption: there is a time period, T where negotiation cannot continue anymore. In T the conflict allocation is implemented. u Perfect equilibrium by backward induction: s At T-1 if negotiations hasn’t ended, A T-1 suggests the best agreement to itself which is better to all agents than the conflict solution (denoted by O T-1 ); the other agents accept. s At T-2, A T-2 suggests the best agreement to itself which is better to all agents than the conflict solution and O T-1 (denoted by O T-2 ). The other agents accept. s By induction, at the first time period A 0 O 0 the others accept.

95. 95 Assumptions about the environment u Profit is a linear function of the number of items of each product produced by the plant u Pollution is a linear function of the number of items of each product produced.

96. 96 Techniques which were checked  Strategic negotiations:  Sequential responses: backtracking s Simultaneous response: Maximization of the sum with guaranties of default profit : –Simplex method - method for linear optimization u Nash Product:  Praxis - method for multi-variable nonlinear function minimization.  Hill Climbing

97. 97 Simulation Parameters u Number of plants is varied from 5 to 20. u Number of pollution types is varied from 5 to 20. For each product pollution of some type is produced with probability 1/2. u Each plant produces Max_prod different types of products. Max_prod is varied from 5 to 20. Pollution and profit per item of product and pollution constraints are set randomly. u Results: Average of 25 simulation runs.

98. 98 Plants’ utility as the function of the number of plants

99. 99 Standard Deviation as the function of the number of plants

100. 100 Computation time as a function of number of plants

101. 101 Plants’ utility as the function of the number of pollution substances

102. 102 Standard deviation as the function of the number of pollution substances

103. 103 Computation time as a function of the number of pollution substances

104. 104 Plants’ utility as a function of the number of products

105. 105 Standard deviation as a function of the number of products

106. 106 Computation time as the function of the number of products

107. 107 Computation time as a function of the number of products

108. 108 Conclusions u Maximizing the sum yields the highest average utility, but also the highest standard deviation; requires agreement between the designers on selecting a solution. u Backward induction yields a reasonable average utility with low standard deviations and no need for designers agreement on detailed protocol. u On going work: incomplete information.

109. 109 Sharing Resources Through Negotiation Joint resource: public communication system; satellite; Agents: self motivated. Environment: no central controller.

110. 110 Environment Description u Two agents must share a joint resource; the resource can only be used by one agent at a time. No central controller. u One agent (A) is using the resource, and the second (W) wants to use it too. u The agents negotiate to reach an agreement: a schedule that divides the usage of the resource;.

111. 111 Environment Description -cont u A continues to use the resource as the negotiation proceeds: A gains over time. u W is not able to use the resource: W loses over time. u Opting out causes damage to the resource: both agents wait q time steps. u Additional option: an agent can leave the negotiation.

112. 112 Applying the strategic model u We developed a detailed utility function for the agents (U_A; U_W). Parameters: type of goal, dead-lines, costs of negotiation, gains from goal, etc. u Main factor in the negotiation: the best agreement for A, which is still better for W than Opting out (O_n).

113. 113 Perfect equilibrium strategies u O_n depends on the specific situation; we proved lemmas which specify the value of O_n as a function of the utility function parameters. u Complete information: Negotiation ends at most after one step with an agreement, or W leaves. u The strategies are simple.

114. 114 Experiments Using MINUET Agent 1 Agent 2 Working on goal 102 #### Send request Receive request Resources 1001 - free 1002 - busy

115. 115 Experiments Results Nego.EDFMetric Utility score 91% Abandon goals9.68.4 Nego./Alter. 21.2 15.5

116. 116 Summary u A strategic model of negotiation, taking the passage of time into account. u We consider wide range of situations: complete /incomplete information; N>2 agents; agents lose over time/some lose and some gain over time;

117. 117 Summary--cont. u The model was applied to different domains. u We found simple and stable strategies. u Negotiation ends without delay.