1. Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010

2. Outline [Yu] Optimized Sensing: From Water to the Web Distributed Dynamic Programming Distributed Solutions to Markov Decision Problems

3. Optimized Sensing Problem Statement Greedy Algorithms and Submodularity Robust Sensing Optimization with Saturate Algorithm Application in Blogs

4. Problem Statement How do we detect contamination in drinking water distribution networks? Which blogs should we read to learn about the biggest, newest stories on the Web? Fundamental Question: How can we get the most useful information at minimum cost (limited resources)?

5. Solutions to Optimized Sensing Covers fields of statistics, machine learning, sensor networks, and robotics With partially observable Marko decision processes, we can get optimal solutions But it is difficult to scale POMDP to large problems Introducing a new algorithm based on submodularity

6. Formulation Sensing quality function F(A) – A: the set of sensor locations Si (i=1~k) – V: the set of all locations We can also have cost constraints – Total cost of sensor deployment no greater than the budget Goal: Find A* – This is NP-hard already

7. Greedy Algorithm Iteratively find Si This naïve algorithm actually performs pretty well – Why? Submodularity – We get near-optimal solutions Submodularity: diminishing returns

8. Diminishing Returns

9. Cost-Effective Lazy Forward-Selection (CELP) Greedy algorithm Lazy evaluations – Delaying computation until the result is required – A computational technique

10. Robust Sensing Optimization Idea: Protect system against adversaries that know of our deployment of sensors Goal: Maximize the worst-case detection performance Approach Unfortunately, this naïve extension can fail

11. Failure of Greedy Algorithm on Worst- Case Scenarios I1, I2: two contamination events S1, S2, S3: three possible sensor locations – S1: detect I1 immediately, but never I2 – S2: detect I2 immediately, but never I1 – S3: detect both I1 and I2, but only after a long time We can only place two sensors Greedy would pick S3 first and then either S1 or S2 But we know the optimal solution should be S1 and S2 Solution? Saturate algorithm

12. Saturate Algorithm Idea: reduce the non-submodular worst-case objective to a submodular optimization problem – Transform non-submodular to submodular Transformation – Guess optimal solution value C using binary search – Try to find A such that F(A) is no less than C

13. Performance of Saturate

14. From Water to the Web Blog Reading Problem: Information cascading

15. Improvements Number-of-posts (NP) model – Reading a big blog can be time-consuming, so they define the cost to be the number of posts CELP tends to choose blogs with many posts NP model tends to choose summarizer blogs – But stories appear in summarizer blogs a little late

16. Other Thoughts What if we are looking for stories to read instead of blogs to read? – We can reverse our information management goal – Find posts instead of blogs – Ref. 10 End of Paper

17. Distributed Dynamic Programming for Path Planning Asynchronous Dynamic Programming Learning Real-Time A*

18. Asynchronous Dynamic Programming Propagate costs from target to start locations

19. Learning Real-Time A* (LRTA*)

20. LRTA*(n) LRTA with n agents Faster – Agents break ties differently – They can share the same h-value table

21. LRTA*(2)

22. Distributed Solutions to Markov Decision Problems As previously mentioned in the Water to Web paper, MDPs can be difficult to scale to big problems Solution: Exploit independence properties We address the modularity of actions

23. Action Selection in multiagent MDPs

24. Implementation

25. Subtask Distribution A global problem is broken down into subtasks Subtasks are distributed among agents Each agent has different capabilities Problem 25

26. Contract Net Stages – Recognition – Announce – Bidding – Awarding & Expediting Initial assignment: Not optimal Anytime property – Improve assignment in negotiation process 26

27. Assignment problem Problem definition – A set N of n agents – A set X of n objects – A set M ⊆ N × X of possible assignment pairs, and – A function v : M → R Find optimal assignment XN M 27

28. Corresponding Linear Program Linear program (LP) formulation Profit maximization Resource constraint Optimal solution Any LP can be solved in polynomial time O(n 3 ) 28

29. Competitive Equilibrium Consider a price vector p = (p 1, …, p n ) – The utility from an assignment j to agent i is u(i, j) = v(I, j) - p j A feasible assignment S and a price vector p are in competitive equilibrium when for every pairing (i, j) ∈ S it is the case that ∀ k, u(i, j) ≥ u(i, k) 29 Every agent will not change its selection S is a optimal solution

30. Naïve Auction Algorithm Round-robin style Bid increment is the difference between the utility to i of the best and second-best object 30 The agent will not overbid

31. Problem in Naïve Auction When more than one object offers maximal utility for an agent – Bid increment is zero 31

32. Terminating Auction Algorithm Modify the bid increment – 32 ε-competitive equilibrium: u(i, j) + ε ≥ u(i, k) Agents may overbid some objects

33. Scheduling Problem Problem definition – N is a set of n agents – X is a set of m discrete and consecutive time slots – q = (q 1,..., q m ) is a reserve price vector – v = (v 1,..., v n ), where v i is the valuation function of agent I Find optimal allocation 33 F

34. Corresponding Integer Program Integer program (IP) formulation IPs are not solvable polynomial time 34

35. Competitive Equilibrium – General Form Definition – For all i ∈ N it is the case that F i = argmax T ⊆ X (v i (T) − ∑ j|x j ∈ T p j ) – For all j such that x j ∈ F ∅ it is the case that p j = q j – For all j such that x j ∈ F ∅ it is the case that p j ≥ q j May not exist competitive equilibrium 35 Has a competitive equilibrium solution ↕ The LP relaxation of the associated integer program has a integer solution.

36. Ascending Auction Algorithm Center advertise an ask price Bid increment is constant 36

37. Problem in Ascending Auction If the increment is too large May not converge to optimal solution 37

38. Social Laws and Conventions Social law – A restriction on the given strategies of the agents – Induce a sub-game Social convention – The sub-game consists of a single strategy for all agent Other topics – Social goal negotiation – Social norm negotiation – …. 38