1. CS483/683 Multi-Agent Systems Lectures 9-10: Distributed Constraint Optimization: Auction-based solutions 16 February 2009 Instructor: Kostas Bekris Computer Science & Engineering, University of Nevada, Reno

2. CS483 Contract Nets Assume you have a global problem that consists of multiple tasks  assign the tasks to different agents Each agent has different capabilities ∀ agents i, ∃ c i (T): cost for agent i to execute all tasks in the set T Agents start with an initial assignment and negotiate in order to get an optimum assignment Negotiation  Agents repeatedly contract tasks one to another ✓ e.g., exchange tasks for money or tasks for other tasks  Questions ✓ How does the bidding process work? ✓ Which contracts are successful given the bidding process? Requirement for an anytime solution

3. CS483 Negotiation One simple approach: Each agent bids for a task the cost it would incur to execute it - c(i) Tasks are allocated to lowest bidders Repeat It can be shown that there always exists a sequence of contracts that results in the optimal allocation but... if each agent contracts out a single task for money then the optimal sequence might contain “money-losing” contracts there are however more complex contracts  contracting a bundle of tasks (“cluster contracts”)  swap tasks among 2 agents (“swap contracts”)  simultaneous transfers among many agents (“multi-agent contracts”) the more complex contracts allow for optimal sequences without “money- losing” contracts

4. CS483 Challenges 1. What is the connection between the following 2 contradictory notions:  Solving a common global problem  Minimize the total cost to each individual agent ➡ We will see that finding an optimal solution is closely related to the individual utilities of agents 2. When do agents make offers? What is the method by which the contracts are decided upon? ➡ We will have to specify the following: a) Bidding rules: Permissible ways of making offers b) Market clearing rules: Definition of the outcome based on the offers c) Information dissemination rules: the information made available to the agents throughout the process ➡ We will assume an explicit centralized component ‣ an auctioneer

5. CS483 Challenges 3. Since we have cooperative agents, why does it matter if agents “lose- money”?  Auctions in general take place so that each agent will maximize a personal payoff  Game-theoretic notions ➡ In the optimization context: ✓ agents do not deviate from their prescribed bidding protocol for personal gains ✓ but game-theoretic agents can be used as a guidance We will study 2 classical optimization problems: 1. Task Assignment problems (linear program) 2. Scheduling problems (integer linear program) and how we can solve them with auctions and negotiations

6. CS483 An Intro to Linear and Integer Programming Linear Program: 1. Real-valued variables:  {x 1,…,x n } with each x i ∈ R 2. Linear objective function given constants w 1, …, w n :  maximize ∑ i w i x i  minimization problems can be solved with the weights all being negative 3. A set of linear constraints given constants a 1j,…,a nj and a constant b j :  subject to ∑ i a ij x i ≤ b j ∀ j = 1, …, m  x i ≥ 0 ∀ i = 1,..., n

7. CS483 Linear Programming Similar problems: minimization problems:  all the weights should be negative greater-than-or-equal constraints  negate all the constants in constraints equality constraints  satisfy both less-than-or-equal and greater-than-or-equal constraints ignore variables  set corresponding constants to zero piecewise-linear constraints (e.g., max function)  can be constructed by defining additional constraints and fake variables

8. CS483 Linear Programming Matrix form w: n × 1 vector containing the weights w i x: n × 1 vector containing the variables x i A: m × n matrix of constants a ij b: m × 1 vector of constants b j maximize w T x subject to Ax ≤ b x ≥ 0 Dual problem: minimize b T y subject to A T y ≥ w y ≥ 0

9. CS483 Economic Interpretation Imagine a production economy: set of resources set of products each product consumes a certain amount of each resource each product is sold at a certain price Interpret x i as the amount of product i produced and c i as the price of product i: maximize profit ∑ i c i x i Interpret b j as the available amount of resource j and a ij as the amount of resource j needed to produce a unit product i: satisfy ∑ i a ij x i ≤b j Dual variable y: shadow price of resource i High shadow price means that a resource has a high impact on the optimal solution

10. CS483 Linear Programming Each linear constraint corresponds to: a hyper-plane in an n-dimensional space Feasible solutions: the region bounded by all the hyperplanes (a convex polyhedron) Because the objective function is also linear: any local optimum in the feasible region is a global optimum at least one optimal solution will exist at a vertex of the polyhedron Popular solution: Simplex algorithms start from a vertex of the polyhedron start moving uphill (i.e., improve the objective function) towards other vertices until an optimum is found Exponential complexity but efficient in practice

11. CS483 Integer Programming Like linear programs only that the variables have to be... integers! much harder computational problem (worst case: undecidable) A useful subclass: 0-1 integer programs variables can only take 0 and 1 integer values Any problem in the NP class is equivalent to an Integer programming problem Another useful subclass: mixed-integer program some variables can be reals, some must be integers There is no real efficient solution for integer programs commonly used: branch-and-bound search (variation of backtracking search)  difference from backtracking: additional pruning takes place ✓ if the linear program relaxation does not allow certain integer values then these values are not considered in the search procedure

12. CS483 1. Assignment Problem a.k.a. weighted matching in a bipartite graph a set N of n agents a set X of n objects a set M ⊆ N × X of possible assignment pairs; and a function υ: M ⇒ R giving the value of each assignment pair An assignment is a set of pairs S ⊆ M so that each agent i ∈ N and each object j ∈ X is in at most one pair in S. A feasible assignment is one in which all agents are assigned an object A feasible assignment is optimal if it maximizes: ∑ (i,j) ∈ S u(i,j)

13. CS483 Example Utilities i u(i,x 1 ) u(i,x 2 ) u(i,x 3 ) 1240 2150 3132 Optimal assignment: (1,x 1 ), (2,x 2 ), (3,x 3 )

14. CS483 Encoding as a Linear Program Define indicator matrix x: x i,j = 1 indicates that the pair (i,j) is selected x i,j = 0 otherwise Express the linear program as follows: maximize ∑ (i,j) ∈ M u(i,j)x i,j subject to ∑ j|(i,j) ∈ M x i,j ≤ 1 ∀ i ∈ N ∑ i|(i,j) ∈ M x i,j ≤ 1 ∀ j ∈ X Why linear program and not an integer program though? Turns out: the LP formulation allows integral solutions any optimal fractional solution can be turned into an integral one

15. CS483 Properties of Linear Programming Formulation Since any LP can be solved in polynomial time the assignment problem can be solved in polynomial time as well But! the complexity is O(n 3 ), which may be too high in some cases no obvious parallelization not adaptive to changes in the problem specification So, lets take a different approach... one that has some competitive notions

16. CS483 Competitive Equilibrium Imagine each object in X has an associated price: price vector p = (p 1, …, p n ), where p j is the price of object j Given an assignment S ⊆ M and a price vector redefine the utility from an assignment of object j to agent i as:  u(i,j) = u(i,j) - p j Then we can define the notion of a competitive equilibrium: A feasible assignment S and a price vector p are in competitive equilibrium when for every pairing (i,j) ∈ S it is the case that ∀ k, u(i,j) ≥ u(i,k) If a feasible assignment S and a price vector p satisfy the competitive equilibrium condition then S is an optimal assignment. For any optimal solution S, ∃ price vector p s.t. p and S satisfy the competitive equilibrium condition.

17. CS483 Price Vector example Price Vector: (2,4,1) i u(i,x 1 ) u(i,x 2 ) u(i,x 3 ) 1240 2150 3132 i u(i,x 1 ) - p i u(i,x 2 ) - p i u(i,x 3 ) - p i 100 2 1 3 1 In order to search for a solution, we can search for: a competitive equilibrium and the corresponding price vector

18. CS483 An auction-based solution Start with no objects allocated Each agent bids for the object that maximizes utility - cost Update the price of an object by the bidding amount Terminate once the algorithm has found a feasible solution S ← 0 ∀ j p j ← 0 repeat i: an unassigned agent // agent that will bid j ← argmax k|(i,k) ∈ M (u(i,k) -p k ) // object with maximum payoff b i ← (u(i,j) - p j ) - argmax k|(i,k) ∈ M;k≠j (u(i,k) -p k ) // b i : the difference between the best object and the 2nd best object S ← add assignment (i,j) if there is another pair (i’,j) then remove it from S p j += b i until S is feasible

19. CS483 Example The auction-based algorithm will only terminate at a competitive equilibrium But it doesn’t always terminate: when more than 1 object offers maximal value for a given agent and the objects happen to be the best for another agent round p1p1p1p1 p2p2p2p2 p3p3p3p3bidder preferred object assignment i u(i,x 1 ) u(i,x 2 ) u(i,x 3 ) 1110 2110 3110 i u(i,x 1 ) u(i,x 2 ) u(i,x 3 ) 1240 2150 3132 00001x2x2 (1,x 2 ) 10202x2x2 (2,x 2 ) 20403x3x3 (2,x 2 ),(3,x 3 ) 30411x1x1 (2,x 2 ), (3,x 3 ), (1,x 1 )

20. CS483 A terminating auction algorithm Idea: each time an agent makes a bid, add a small increment ε  b i = u(i,j) - max k|(i,k) ∈ M;k≠j u(i,k)+ ε round p1p1p1p1 p2p2p2p2 p3p3p3p3bidder preferred object assignment 0ε001x1x1 (1,x 1 ) 1ε2ε2ε02x2x2 (1,x 1 ), (2,x 2 ) 23ε3ε2ε2ε03x1x1 (3,x 1 ),(2,x 2 ) 33ε3ε4ε4ε01x2x2 (3,x 1 ), (1,x 2 ) 45ε5ε4ε4ε02x1x1 (2,x 1 ), (1,x 2 ) i u(i,x 1 ) u(i,x 2 ) u(i,x 3 ) 1110 2110 3110

21. CS483 ε-Competitive Equilibrium The algorithm terminates if an object receives a bid in k iterations, its price must exceed its initial price by at least kε for sufficiently large k, the object will become expensive enough to be judged inferior to some object that has not received a bid so far once all objects receive at least one bid, the auction terminates worst case performance is O(n 3 k) for ε= 1 / n the smaller the ε - the worse the computational performance But the competitive equilibrium property is no longer maintained agents overbid on some objects ε-competitive equilibrium: S and p satisfy an ε-competitive equilibrium when for each i ∈ N, if there exists a pair (i,j) ∈ S, then ∀ k: u(i,j) + ε ≥ u(i,k) A feasible solution S with n objects that forms an ε-competitive equilibrium with some price vector is within nε of optimal the smaller the ε - the best the approximation to the optimal solution

22. CS483 2. Scheduling Problems A tuple C = (N,X,q,u) N is a set of n agents X is a set of discrete and consecutive time slots q = (q 1,…, q m ) is a price vector, where q j is the value that must be paid to reserve time slot x j u = (u 1, …, u n ), where u i is the valuation function of agent i with 2 arguments:  d i : the deadline of agent i  λ i : the required number of time slots required by agent i A solution to a scheduling problem is a vector F = (F 0, F 1, …, F n ), where F i is the set of time slots assigned to agent i [F 0 : unassigned time slots] So for an allocation F i ⊂ 2 X we have that: u i (F i ) = w i - if F i includes λ i hours before d i ; u i (F i ) = 0 - otherwise Value of solution: V(F) = ∑ j|xj ∈ F0 q j + ∑ i ∈ N u i (F i )

23. CS483 Scheduling example Eight one hour slots: from 9:00am to 5:00pm Reserve price of $3 per hour joblengthλdeadlinedworthw 12 hours1:00pm$10.00 22 hours12:00pm$16.00 31 hour12:00pm$6.00 44 hours5:00pm$14.50 time slot agent 9:00 am2 10:00 am2 11:00 am1 12:00 pm1 13:00 pm4 14:00 pm4 15:00 pm4 16:00 pm4 Integer Program Formulation: maximize ∑ S ⊆ X,i ∈ N u i (S)x i,S subject to ∑ S ⊆ X x i,S ≤ 1 ∀ i ∈ N ∑ S ⊆ X:j ∈ S,i ∈ N x i,S ≤ 1 ∀ j ∈ X x i,S ∈ {0,1} ∀ S ⊆ X, i ∈ N

24. CS483 Generalized form of competitive equilibrium Given a scheduling problem, a solution F is in competitive equilibrium at prices p if and only if: ∀ i ∈ N: F = argmax T ⊆ X (u i (T) - ∑ j|xj ∈ T p j )  the set of time slots allocated to agent i maximizes his surplus at prices p ∀ j s.t. x j ∈ F 0 : p j = q j  the price of all unallocated time slots is the reserve price ∀ j s.t. x j ∉ F 0 : p j ≥ q j  the price of all allocated time slots is greater than the reserve price If a solution F to a scheduling problem C is in equilibrium at prices p, then F is also optimal for C But... the scheduling problem is a hard problem the existence of a (generalized) competitive equilibrium cannot be guaranteed

25. CS483 Ascending-Auction Algorithm Basic process: an auctioneer advertises an ask price agents bid the ask price for bundles of time slots that maximize their surplus at the given prices the process repeats until there is no change Bid price vector: b = (b 1, …, b m ) b j : the highest bid so far for time slot x j Allocated slots for each agent: F = (F 1, …, F n ) Difference with auction-based algorithm for the assignment problem: the bid increment here is always constant and does not depend on the difference between the best and the second-best assignment Let ε be the allowed price increment

26. CS483 Ascending-Auction Algorithm ∀ x j : b j ← q j // the initial bids are set to the reserve price ∀ i: F i ← 0 repeat foreach agent i = 1 to N do foreach slot x j do if x j ∈ F j then p j ← b j // agent gets the slot if the highest bidder else p j ← b j + ε //otherwise should increase bid S* ← argmax S ⊆ X|S ⊇ Fi (u i (S) - ∑ j ∈ S p j ) // find the best subset of slots, given outstanding bids // agent i becomes the high bidder for all slots in S* \ Fi foreach slot x j ∈ S* \ F i do b j ← b j + ε // update bidding price if there exists agent k ≠ i so that x j ∈ F k then set F k ← F k \ {x j } // update current allocations F i ← S* until F does not change

27. CS483 Example Assume an increment: ε = $0.25 roundbidderslots bid onF=(F 1,F 2,F 3,F 4 )b 01( 9, 10)({9,10},{0},{0},{0}) (3.25,3.25,3,3,3,3,3,3) 12( 10, 11)({9},{10,11},{0},{0}) (3.25,3.5,3.25,3,3,3,3,3 ) 23( 09)({0},{10,11},{9},{0}) (3.5,3.5,3.25,3,3,3,3,3) 2410 ({11,12},{9,10},{0},{12,13,14,15}) (6.25,6.25,6.25,3.25,3. 25,3.25,3.25,3.25) joblengthλdeadlinedworthw 12 hours1:00pm$10.00 22 hours12:00pm$16.00 31 hour12:00pm$6.00 44 hours5:00pm$14.50

28. CS483 Properties The same algorithm for an increment of $1.00 does not reach the competitive equilibrium Even for small increments, however, there is no guarantee of convergence Unfortunately, even when the algorithm converges... it does not necessarily converge to to an optimal solution. We cannot even provide a bound! But the algorithm terminates!  the prices of slots always increase  so at some point unassigned slots will be also selected job length λ deadline d worth w 11 hour11:00am$2.00 22 hours12:00pm$20.00 32 hours12:00pm$8.00 3 time slots 9am,10am,11am reserve price: $0job length λ deadline d worth w 11 hour10:00am$3.00 22 hours11:00am$11.00 2 time slots 9am,10am reserve prices: $1, $9