1. Trading Agent Competition: Performance Evaluation Presented by Brett Borghetti borg@cs.umn.edu 22 March 2006

2. Think about this You own a small business You own a small business You make a bunch of strategic decisions/plans/policies You make a bunch of strategic decisions/plans/policies Your 1 st quarter net profit is $100,000 Your 1 st quarter net profit is $100,000 –Which choices helped? –Which choices hurt? –Can your decisions be examined independently? –How do you improve next quarter?

3. The Situation We sometimes have to make our plans and policies before their execution We sometimes have to make our plans and policies before their execution We don’t know fully what the market will do next quarter (uncertainty) We don’t know fully what the market will do next quarter (uncertainty) We are in competition with other businesses/entities who may act to thwart our plans We are in competition with other businesses/entities who may act to thwart our plans

4. A Solution Repeat (until good enough): Repeat (until good enough): –Predict the effects of our choices offline –Adjust our choices to optimize outcome Execute our plans Execute our plans Measure the effectiveness of our choices online Measure the effectiveness of our choices online

5. Presentation Overview TAC-SCM Overview TAC-SCM Overview Current analysis methods Current analysis methods New methods New methods Future Research Future Research

6. What is TAC-SCM? Simulation of a market supply chain Simulation of a market supply chain –Agent is the computer manufacturer –Buys parts from suppliers in auction –Manage assembly line/production schedule –Reverse Auction to sell computers –Ship computers to customers Six agents compete: maximize profit Six agents compete: maximize profit TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

7. TAC-SCM Interaction TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

8. Game Flow Diagram

9. TAC - Why is it Interesting? Complexity: Beyond human-in-the-loop capability Complexity: Beyond human-in-the-loop capability –Compete with 5 other agents selling computers –Real time: 15 sec/day x 220 days –Auctions (normal and reverse for all transactions) –8 parts suppliers with production capacity changing daily –16 different computer types to build in 3 price classes –100s of Customers with varying demand and reserve prices –Price probing, future purchase decisions..... Small market: Agents have large impact on each other Small market: Agents have large impact on each other –Explicit Competition – PROFIT! –Learning other’s habits & patterns and out-thinking them –Information denial / Decision perturbation TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

10. UMN MinneTAC Design Component-based architecture Component-based architecture –Procurement – Purchase parts from suppliers –Production – Manages the production line –Sales – Interacts with customers to make sales –Shipping – plans customer shipping schedule –Repository – centralized data storage / accesors –Oracle – decision assistance evaluators TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

11. Design pros and cons Lower module coupling = good design Lower module coupling = good design –More simultaneous developers –Easier to maintain Self interest vs. Common good Self interest vs. Common good Causality – which components responsible for a good or bad decision? Causality – which components responsible for a good or bad decision? How do we analyze and improve our global performance? How do we analyze and improve our global performance? TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

12. Current Analysis Methods Run offline simulations and tweak components to optimize profit Run offline simulations and tweak components to optimize profit –CPU intensive (1 hour per game) –Statistical significance => many games –Competition is limited –Causal analysis is complicated TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

13. New Analysis Methods What if we could measure performance of components inside of the agent? What if we could measure performance of components inside of the agent? –We could directly compare performance between two components of the same type against the same TAC market dataset –We could reduce the number of games required to show correlations / relative performance –We could more rapidly determine which ‘tweaks’ actually have an effect on game outcome TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

14. Challenges of Measuring Which metrics are actually correlated with profit? Which metrics are actually correlated with profit? How do we assign sharing of credit or blame? How do we assign sharing of credit or blame? How do we account for the varying market conditions while taking measurements over multiple games? How do we account for the varying market conditions while taking measurements over multiple games? How do we simulate various competitive environments offline? How do we simulate various competitive environments offline? TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

15. Methodology - Overview Controlling the market conditions Controlling the market conditions –Control Randomness –Control market supply / demand situation Measuring component performance Measuring component performance –Create metrics –Determine if metric is correlated with profit –Assign component responsibility TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

16. Controlling Randomness Re-design server to allow deterministic / replayable games Re-design server to allow deterministic / replayable games Three types of random processes: Three types of random processes: –Server variables (customer/supplier) –Agent-dependent variables –Dummy agent variables Each process gets its own seed Each process gets its own seed –Eliminates race conditions in replays –Allows some process true randomness while others replay TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

17. Market Manipulation Agents Goal – develop a way of manipulating supply and demand conditions during a simulation to observe how competitive agents respond Goal – develop a way of manipulating supply and demand conditions during a simulation to observe how competitive agents respond Method – Build TAC agents that are not concerned with their own profit, but rather with absorbing/releasing market share Method – Build TAC agents that are not concerned with their own profit, but rather with absorbing/releasing market share TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

18. Market Manipulation Agents Market Relief Agent Market Relief Agent –Accepts and fulfils no customer RFQs –Purchases no parts from suppliers –Result: Reduces demand on suppliers and reduces supply to customers Market Pressure Agent Market Pressure Agent –Makes more promises to customers than regular agent could handle –Buys more parts from suppliers than regular agent should –Result: Increases demand on suppliers and causes customer demand to go down TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

19. Measuring Component Performance Create suite of metrics to measure: Create suite of metrics to measure: –Replacement costs when a part is sold –Storage costs of parts/computers –Late penalties –Wasted production cycles –Remaining inventory at end of game –… TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

20. Measuring Causality How do we assign responsibility? How do we assign responsibility? For example: Why was the item late? For example: Why was the item late? Didn’t ship the product? Didn’t ship the product? Didn’t make the product? Didn’t make the product? Didn’t have the parts? Didn’t have the parts? TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

21. Implementing Metrics Allow for easy creation of new metrics Allow for easy creation of new metrics –Serialize game information –Evaluations can then be made offline –Enables us to experiment in finding metrics that are correlated with profit. But how do we even know if a metric is correlated with profit? But how do we even know if a metric is correlated with profit? –Large amount of variability in each game –Need a large sample size, which takes time TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

22. Results to date We have some preliminary data regarding how the manipulation agents cause the other agents to behave under various market conditions We have some preliminary data regarding how the manipulation agents cause the other agents to behave under various market conditions TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

23. Performance Results: Market Relief Agent vs Dummy Agents Note the scale of this graph TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

24. Performance Results: Market Relief Agent vs Dummy Agents Unexpected benefits! Unexpected benefits! –MRAs can reveal undesireable traits/logic flaws in an agent TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

25. Performance Results: Market Pressure Agent vs MinneTAC TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

26. Performance Results: Market Pressure Agent vs MinneTAC TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

27. Performance Results: Market Pressure Agent vs MinneTAC TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

28. Performance Results: Market Pressure Agent vs Competition TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

29. Conclusions We’ve created some new tools for measuring offline performance We’ve created some new tools for measuring offline performance –Replayable games –Market Condition Manipulation –Embedded Metrics Collection Started choosing what metrics contain information allowing profit prediction Started choosing what metrics contain information allowing profit prediction TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

30. Future Work Improve Market Manipulation agents Improve Market Manipulation agents –Make competition modeling more realistic Find additional metrics that have a better correlation to overall profit Find additional metrics that have a better correlation to overall profit –Better off-line prediction of on-line performance Use metrics to guide development of better components Use metrics to guide development of better components –Leads to better profit performance [build to the metric] Use on-line metrics to make live strategic decisions Use on-line metrics to make live strategic decisions –Live ‘tuning’ of components if they begin to underperform –Selection of ‘pinch-hitter’ components in certain market conditions TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

31. Acknowledgement / Info Special thanks to: –Eric Sodomka –Dr. Maria Gini –Dr. John Collins –UMN TAC team More Info at –MinneTAC website www.cs.umn.edu/tac www.cs.umn.edu/tac www.cs.umn.edu/tac –SICS website www.sics.se/tac www.sics.se/tac www.sics.se/tac