Game Intelligence: The Future Simon M. Lucas Game Intelligence Group School of CS & EE University of Essex

Meet Adrianne from nVidia

Beautiful, but not very bright, … … yet.

Game Intelligence Group Main Activity: – General purpose intelligence for game agents 2 Academic staff 1 post-doc 10 PhD students

Approaches Evolution Reinforcement Learning Monte Carlo Tree Search

Conventional Game Tree Search Minimax with alpha-beta pruning, transposition tables Works well when: – A good heuristic value function is known – The branching factor is modest E.g. Chess: Deep Blue, Rybka Tree grows exponentially with search depth

Go Much tougher for computers High branching factor No good heuristic value function “Although progress has been steady, it will take many decades of research and development before world-championship– calibre go programs exist”. Jonathan Schaeffer, 2001

MCTS Operation (fig from CadiaPlayer, Bjornsson and Finsson, IEEE T-CIAIG) Each iteration starts at the root Follows tree policy to reach a leaf node Then perform a random roll-out from there Node ‘N’ is then added to tree Value of ‘T’ back- propagated up tree

Upper Confidence Bounds on Trees (UCT) Node Selection Policy From Kocsis and Szepesvari (2006) Aim: optimal balance between exploration and exploitation Converges to optimal policy given infinite number of roll-outs Often not used in practice!

Sample MCTS Tree (fig from CadiaPlayer, Bjornsson and Finsson, IEEE T-CIAIG)

Learning Tree Policy and Roll-Out Policy Results for Othello (IEEE CIG 2011)

Research Leadership Grants Conference Series Journal Conference Special Sessions Competitions Software Toolkits (e.g. WOX, featured in MSDN Magazine)

Research Grants AI Games Network (EPSRC, 2007 – 2010; with Imperial and Bradford) UCT for Games and Beyond (EPSRC, 2010 – 2014, joint with Imperial and Bradford; 1.5M total, 489k Essex) Plus IEEE T-CIAIG Editorial Assistant + Travel

IEEE Transactions on Computational Intelligence and AI in Games Published quarterly since March 2009 Journal has made an excellent start

Competitions Many, many competitions Most recent: Ms Pac- Man versus Ghost Team: IEEE CEC 2011 An interesting and fun AI challenge Covered by New Scientist, Slashdot etc By Philipp Rohlfshagen and David Robles

Summary AI + Games – Fantastic field to work in – The BEST test-bed for general intelligence – Monte-Carlo Tree Search + Reinforcement Learning: very promising! Reasonable standard general game playing is already a reality for many games Within the next 10 years we’ll enjoy interacting with life-like AI characters

Sample Games