Performance of Distributed Constraint Optimization Algorithms A.Gershman, T. Grinshpon, A. Meisels and R. Zivan Dept. of Computer Science Ben-Gurion University

DCR workshop - May Optimization Problems Optimization Problems Problems are too tight No solution that satisfies all constraints exists Search for the solution with a minimal cost

DCR workshop - May Constraint Optimization Problems Constraint Optimization Problems Weighted Binary CSPs: Every pair of assignments [, ], is assigned a cost c The cost of a tupple is the sum of all costs of pairs included in it Specific case – Max-CSP: all costs c are either 0 or 1 [Larrosa & Meseguer 96], [Larrosa & Schiex 2004]

DCR workshop - May Distributed Constraint Optimization Problems (DisCOPs) Distributed Constraint Optimization Problems (DisCOPs) There are several approaches for solving DisCOPs: Branch and Bound SynchBB, AFB Using a Pseudo-Tree ADOPT, DPOP Merging partial solutions OptAPO Very different algorithms  different behavior Comparative evaluation of runtime performance is needed

DCR workshop - May AFB AFB Add text

DCR workshop - May ADOPT ADOPT Add text

DCR workshop - May DPOP DPOP Add text

DCR workshop - May OptAPO OptAPO Add text

DCR workshop - May A bit of history… DisCSPs A bit of history… DisCSPs The runtime performance of ABT [Yokoo et al. 1995] was first compared to AWC [Yokoo et al. 1998]: Performance was measured in cycles of a synchronous simulator

DCR workshop - May Runtime Performance of DisCSPs (II) In 2001 we have played with a synchronous algorithm that uses ordering heuristics and have found that it is faster than ABT – but how to measure ? In 2002 a non-concurrent runtime measure for DisCSP search – non-concurrent constraints checks (NCCCs) Synchronous CBJ (with variable ordering) was shown to be faster than ABT (NCCCs) [Brito & Meseguer 2004] All on randomly generated DisCSPs

DCR workshop - May Runtime Performance of Centralized COPs Certain search algorithms produce a phase transition for increasingly tighter (and harder) random problems Performance of standard Branch and Bound grows exponentially for ever harder problems. MaxCSPs are harder than Weighted CSPs

DCR workshop - May Evaluation of Runtime Performance – ADOPT* *[Modi et al. 2005]

DCR workshop - May Evaluation of Runtime Performance – OptAPO* Evaluation of Runtime Performance – OptAPO* *[Mailler & Lesser 2004]

DCR workshop - May What is a cycle? What is a cycle? cycles All quoted results are in cycles What is a cycle for each of the algorithms: Adopt, AFB – reading all messages and checking all local assignments against the current context or CPA DPOP – Calculating all costs of the sub tree for every combination of assignments of higher priority constrained agents OptAPO, Solving centrally a problem of the size of the mediation session

DCR workshop - May How to count NCCCs for DisCOPs ? How to count NCCCs for DisCOPs ? ADOPT, SBB and AFB perform CCs in each computation session and can be counted non- concurrently as for DisCSPs DPOP – for every row in the table sent by a DPOP agent, the number of CCs is the product of number of potential assignments times the number of constrained (up-tree) agents OptAPO - each mediation session is assigned the number of CCs needed to find the local solution

DCR workshop - May Choosing the right benchmark for DisCOPs Graph coloring problems do not cover important ranges of problem difficulty. Specific problems have special structures (MSP – equality binary constraints, Sensor Nets – very small density…) Evaluation – use random DisMaxCSPs and increase problem’s difficulty (tightness) One way to exhibit a “phase transition”

DCR workshop - May Experimental Set-up Experimental Set-up Randomly generated Max-CSPs Size: 10 variables 10 values Density: p 1 = 0.4, 0.7 Tightness: p 2 = 0.4 – 0.99 Rune time Measure: Non-Concurrent Constraint Checkes (NCCCs)

DCR workshop - May Logarithmic scale (p1 = 0.4)

DCR workshop - May Linear scale (p1 = 0.4)

DCR workshop - May Low to intermediate difficulty Low to intermediate difficulty ADOPT performs well AFB, OptAPO, and SynchBB are fairly close DPOP performs extremely poor Probably due to the lack of pruning of the search space

DCR workshop - May As difficulty grows… The runtime of ADOPT, OptAPO, and SynchBB grows at a high exponential rate ADOPT did not terminate its run on the tightest problems (p 2 ≥ 0.9) DPOP and AFB perform far better, by two orders of magnitude

DCR workshop - May Linear scale – A Closer Look

DCR workshop - May High Constraints Density (p 1 = 0.7) High Constraints Density (p 1 = 0.7)

DCR workshop - May Linear scale (p 1 = 0.7)

DCR workshop - May High Density and Low Tightness The performance of ADOPT, AFB, OptAPO, and SynchBB is fairly similar The performance of DPOP is much worse

DCR workshop - May High Density and Tightness The performance of ADOPT, OptAPO, and SynchBB deteriorates exponentially The algorithms did not terminate their run on the tightest problems (p 2 > 0.9) ADOPT is the worst, since it failed to terminate its run at a lower tightness value (0.7) DPOP’s runtime is high, but it always terminated and is independent of tightness AFB is clearly the best performing algorithm Outperforms DPOP by orders of magnitude

DCR workshop - May Analysis of ADOPT Analysis of ADOPT ADOPT is unable to solve hard problems High tightness generates an excess of (greedy) context switches along with an exponential increase in the number of messages sent Each agent in ADOPT sends out two messages following every single message it receives This causes the runtime of ADOPT to increase at a very high exponential rate

DCR workshop - May Analysis of DPOP Analysis of DPOP DPOP does not perform search or pruning Computes the same size of matrices regardless of the tightness A change in the problem’s tightness would only affect the content of the matrices

DCR workshop - May Analysis of AFB Analysis of AFB The performance of AFB exhibits a ”phase-transition” AFB’s runtime increases as the problem’s difficulty (tightness) increases, and then suddenly drops by an order of magnitude at very high values of p 2 This is very similar to COPs – deeper lookahead leads to much improved performance [Gershman et al. 2006] [Larrosa & Schiex 2004]

DCR workshop - May Conclusions Conclusions ADOPT appears to be affected the most by the increase of the problem’s tightness OptAPO performs up to three times better than SynchBB DPOP’s performance is independent of the problem’s tightness AFB performs well on the whole range of problem difficulty. It is unique in its phase transition. Probably due to its use of pruning techniques through asynchronous forward-bounding