2. 1991 1997 1998 2001 2005 2008

3. Why Machine Learning and Games? Machine Learning in Video Games Drivatars™ Reinforcement Learning Machine Learning in Online Games TrueSkill™ Halo 3 History of Chess Conclusions

4. Test Beds for Machine Learning Perfect instrumentation and measurements Perfect control and manipulation Reduced cost Reduced risk Great way to showcase algorithms Improve User Experience Create adaptive, believable game AI Compose great multiplayer matches based on skill and social criteria Mitigate Network latency using prediction Create realistic character movement

5. Partially observable stochastic games States only partially observed Multiple agents choose actions Stochastic pay-offs and state transitions depend on state and all the other agents’ actions Goal: Optimise long term pay-off (reward) Just like life: complex, adversarial, uncertain, and we are in it for the long run!

6. Partially Observable Markov Decision Process (POMDP) From single player’s perspective Reinforcement Learning Unsupervised Learning Supervised Learning Approximate Solutions Always takes optimal actions Delivers best entertainment value What is the best AI?

8. Why Machine Learning and Games? Machine Learning in Video Games Drivatars™ Reinforcement Learning Machine Learning in Online Games TrueSkill™ Halo 3 History of Chess Conclusions

9. Adaptive avatar for driving Separate game mode Basis of all in-game AI Basis of “dynamic” racing line

10. XBOX Game Dynamic Racing LineDynamic Racing Line Learning a DrivatarLearning a Drivatar Using a DrivatarUsing a Drivatar

11. “Built-In” AI Behaviour Development Tool Drivatar Racing Line Behaviour Model Vehicle Interaction and Racing Strategy Controller Drivatar AI Driving Recorded Player Driving

13. Two phase process: 1.Pre-generate possible racing lines prior to the race from a (compressed) racing table. 2.Switch the lines during the race to add variability. Compression reduces the memory needs per racing line segment Switching makes smoother racing lines.

14. a4a4 a2a2 a3a3 a1a1 Segments

17. Physics (“causal”) Physics Simulation System Car Position and Speed at time t Static Car Properties Car Controls Car Position and Speed at time t+1 Desired Pos. and Speed at time t+1 Controller Control (“inverse”)

18. Why Machine Learning and Games? Machine Learning in Video Games Drivatars™ Reinforcement Learning Machine Learning in Online Games TrueSkill™ Halo 3 History of Chess Conclusions

19. game state action game state reward / punishment parameter update Agent Game Learning Algorithm

20. 3 ft Q-Table THROWKICKSTAND 1ft / GROUND 2ft / GROUND 3ft / GROUND 4ft / GROUND 5ft / GROUND 6ft / GROUND 1ft / KNOCKED 2ft / KNOCKED 3ft / KNOCKED 4ft / KNOCKED 5ft / KNOCKED 6ft / KNOCKED actions game states 13.2 10.2 - 1.3 5 ft 3.26.0 4.0 +10.0

21. Game state features Separation (5 binned ranges) Separation (5 binned ranges) Last action (6 categories) Last action (6 categories) Mode (ground, air, knocked) Mode (ground, air, knocked) Proximity to obstacle Proximity to obstacle Available Actions 19 aggressive (kick, punch) 19 aggressive (kick, punch) 10 defensive (block, lunge) 10 defensive (block, lunge) 8 neutral (run) 8 neutral (run) Q-Function Representation One layer neural net (tanh) One layer neural net (tanh) Reinforcement Learner In-Game AI Code

22. Early in the learning process … … after 15 minutes of learning Reward for decrease in Wulong Goth’s health

23. Early in the learning process … … after 15 minutes of learning Punishment for decrease in either player’s health

24. Why Machine Learning and Games? Machine Learning in Video Games Drivatars™ Reinforcement Learning Machine Learning in Online Games TrueSkill™ Halo 3 History of Chess Conclusions

25. Competition is central to our lives Innate biological trait Driving principle of many sports Chess Rating for fair competition ELO: Developed in 1960 by Árpád Imre Élő Matchmaking system for tournaments Challenges of online gaming Learn from few match outcomes efficiently Support multiple teams and multiple players per team

26. Given: Match outcomes: Orderings among k teams consisting of n 1, n 2,..., n k players, respectively Questions: Skill s i for each player such that Global ranking among all players Fair matches between teams of players

27. Latent Gaussian performance model for fixed skills Possible outcomes: Player 1 wins over 2 (and vice versa) y12y12 y12y12 p1p1 p1p1 p2p2 p2p2 s1s1 s1s1 s2s2 s2s2

28. Skill of a team is the sum of the skills of its members y12y12 y12y12 t1t1 t1t1 t2t2 t2t2 s2s2 s2s2 s3s3 s3s3 s1s1 s1s1 s4s4 s4s4

29. Possible outcomes: Permutations of the teams s1s1 s1s1 s2s2 s2s2 s3s3 s3s3 s4s4 s4s4 t2t2 t2t2 t1t1 t1t1 t3t3 t3t3 y y

30. But we are interested in the (Gaussian) posterior! s1s1 s1s1 s2s2 s2s2 s3s3 s3s3 s4s4 s4s4 t1t1 t1t1 y12y12 y12y12 t2t2 t2t2 t3t3 t3t3 y23y23 y23y23

31. y12y12 y12y12 y23y23 y23y23 s1s1 s1s1 s2s2 s2s2 s3s3 s3s3 s4s4 s4s4 t1t1 t1t1 t2t2 t2t2 t3t3 t3t3 Gaussian Prior Factors Ranking Likelihood Factors Fast and efficient approximate message passing using Expectation Propagation

32. Leaderboard Global ranking of all players Matchmaking For gamers: Most uncertain outcome For inference: Most informative Both are equivalent!

33. Data Set: Halo 2 Beta 3 game modes Free-for-All Two Teams 1 vs. 1 > 60,000 match outcomes ≈ 6,000 players 6 weeks of game play Publically available

34. 0 55 10 15 20 25 30 35 40 Level 0100200300400 Number of Games char (Halo 2 rank) SQLWildman (Halo 2 rank) char (TrueSkill™) SQLWildman (TrueSkill™)

35. 0100200300400500 0% 20% 40% 60% 80% 100% Number of games played Winning probability 5/8 games won by char char wins SQLWildman wins Both players draw

36. Why Machine Learning and Games? Machine Learning in Video Games Drivatars™ Reinforcement Learning Machine Learning in Online Games TrueSkill™ Halo 3 History of Chess Conclusions

37. Xbox 360 Live Launched in September 2005 Every game uses TrueSkill ™ to match players > 12 million players > 2 million matches per day > 2 billion hours of gameplay Halo 3 Launched on 25 th September 2007 Largest entertainment launch in history > 200,000 player concurrently (peak: 1,000,000)

38. Golf (18 holes): 60 levels Car racing (3-4 laps): 40 levels UNO (chance game): 10 levels

39. Halo 3 Game MatchmakingMatchmaking Skill StatsSkill Stats Tight MatchesTight Matches

42. Why Machine Learning and Games? Machine Learning in Video Games Drivatars™ Reinforcement Learning Machine Learning in Online Games TrueSkill™ Halo 3 History of Chess Conclusions

43. Model time-series of skills by smoothing across time History of Chess 3.5M game outcomes (ChessBase) 20 million variables (each of 200,000 players in each year of lifetime + latent variables) 40 million factors st,ist,i st,ist,i s t +1, i st,jst,j st,jst,j s t +1, j pt,jpt,j pt,jpt,j pt,ipt,i pt,ipt,i pt,jpt,j pt,jpt,j pt,ipt,i pt,ipt,i p t +1, j p t +1, i p t +1, j p t +1, i

44. 1850 1858 1866 1875 1883 1891 1899 1907 1916 1924 1932 1940 1949 1957 1965 1973 1981 1990 1998 2006 1400 1600 1800 2000 2200 2400 2600 2800 3000 Adolf Anderssen Mikhail Botvinnik Jose Raul Capablanca Robert James Fischer Anatoly Karpov Garry Kasparov Emanuel Lasker Paul Morphy Boris V Spassky Whilhelm Steinitz Year Skill estimate

45. Why Machine Learning and Games? Machine Learning in Video Games Drivatars™ Reinforcement Learning Machine Learning in Online Games TrueSkill™ Halo 3 History of Chess Conclusions

46. Mentioned in this talk Adaptive and Learning Game AI Online Gaming Interactions Not mentioned in this talk Adaptive Input Devices Dialogue Generation Computer Vision Realistic Physical Movement New Game Genres based on Machine Learning