A Scene Learning and Recognition Framework for RoboCup Kevin Lam Carleton University September 6, 2005 M.A.Sc Thesis Defense

2 Presentation Overview RoboCup Overview Objective and Motivation Contributions Methodology  Scene Representation  Scene Matching What We Built Experimental Results Future Work

3 RoboCup Overview

4 Objective & Motivation Based on Paul Marlow’s work  Observe other agents instead of humans Lots of RoboCup teams to choose from (subject to restrictions) Advantages: reduce development time, access existing body of knowledge Can an agent learn by observing and imitating another, with little or no intervention from designers or domain experts?

5 Contributions An extensible framework for research in RoboCup agent imitation  a conversion process (raw logs  higher-level representation)  customizable scene recognition and matching using k-nearest- neighbor  a RoboCup client agent based on this scene recognition algorithm Semi-automated imitation results Student contributions (SYSC 5103, Fall 2004) Can an agent learn by observing and imitating another, with little or no intervention from designers or domain experts? … Yes! (with caveats)

6 Current Approaches Most agent development is:  Hard coded or scripted (e.g. Krislet)  High-level behaviour descriptions (Steffens)  Supervised learning situations (Stone) Some attempts at learning by observation  ILP rule inducer (Matsui) Results not directly reused; complex rules, “OK” results  COTS data mining (Weka-based) Problems: complex trees, hard to describe complex behaviours - tuning/pruning needed

7 Methodology Model agent behaviour as function of inputs and output: y = ƒ(a, b, c) Assumptions  Deterministic  Stateless and memory-less (no memory or internal status kept)

8 Methodology Observation of Existing Agents Scenario Representation  “Scene” spatial knowledge representation Scenario Recognition  K-nearest-neighbor search  “Distance” metric definition  Scene and Action selection

9 Agent Observation (see 267 ((f c) ) ((f l t) ) (turn 85.0) (sense_body 312 (view_mode high normal) (see 312 ((f c b) ) ((f b 0) ) (dash 70.0) (see 993 ((p "Canada2" 1 goalie) ) (dash 70.0). Logged messages describe:  Objects seen  Actions sent  Agent internal states

10 Scene Representation A “scene” is a snapshot of space at a given time Up to 6,000 time slices in typical game In RoboCup, this means a list of objects:  Players  Ball  Goals  Lines  Flags Distance Direction Velocity Attributes (team, uniform number etc.)

11 Scene Representation  2723 in: “see …” out: “kick”

12 Discretized Representation Can discretize scenes into segments Size of “slices”  degree of generalization Logical notions; consistent with simulator Reduced storage (or better coverage) vs generalization Extreme Left LeftCenterRight Extreme Right FarGoal Nearby Team Mate Opponent CloseBall

13 Scene Recognition Keep stored scenes from observed agent Find best match between current situation and stored scenes Use k-nearest-neighbor search “What should I do in this situation?” “What did the observed agent do when faced with a situation like this?”

14 “Distance” Metric Object Pairing  a  c, b  d  Separate by type Continuous vs. Discrete Distance  Cosine Law (Continuous)  Euclidian (Cell-based)

15 Action Selection Get k-nearest matching scene-action pairs Only one action must be selected Choices:  Random  First available  Weighted majority Also attributes (direction, power)

16 What We Built Agent based on Krislet New Brain algorithm:  Load scene file at startup  When new info arrives, convert to scene  Compare with every stored scene  Pick the “best match” and reuse action Validator (cross-validation testing)

17 Krislet-Scenes Architecture

18 Selection Algorithms “Distance” Calculation Random Selection Continuous Distance Object Calculation Discretized Distance Object Calculation Discretized Ball-Goal Calculation (student contribution) Action Selection Random Selection First Available Weighted Vote Vote with Randomness Object Matching Simple heuristic (sorted by distance to player)

19 Experiments Logged three RoboCup agents  Krislet, NewKrislet, CMUnited Experimental Parameters  Distance calculation algorithms  Action selection algorithms  k-value  {1, 5, 15}  Object weights  {0, 1}  Discretization size fixed at (5, 3) Quantitative (validation, real) vs Qualitative

20 Experimental Results Best Statistical Results Distance CalculationObject Weightsk and action selection KrisletCellBallGoal (~93%)Ball, or ball and goal k=1 (choose first valid) NewKrisletCellBallGoal (~70%)Ball, or ball and goal k=1 or k=5 (equal vote) CMUnitedContinuous or CellBallGoal (~44%) Ball, or flags, lines, players k=5 (equal vote) Best Qualitative Results ParametersDescription KrisletCellBallGoal, k=1, ball and goal weights Looks like Krislet! Able to score goals, difficult to distinguish from original. NewKrisletCellBallGoal, or Continuous, k=1 or k=5/random, ball and goal Strong resemblance but does not copy “stop and wait” behaviour, instead runs constantly CMUnitedCell distance, k=1 or k=5/random, ball and goal weights Sits and turns frequently, wanders to ball, sometimes tries to kick, shows no sign of other “intelligent” team behaviours

21 Experimental Results Best parameters:  Continuous seems to work better than discretization  k  5 with random selection, or k=1  Object weighting is critical! Can successfully imitate Krislet client (difficult for a human observer to distinguish)  Slightly less responsive, slower to “decide” Imitates many aspects of NewKrislet Attacker Copies only very basic behaviours of CMUnited

22 Limitations Simplistic object matching algorithm  Need way to match detailed objects like players, flags Works best on stateless, deterministic, reactive agents  Does not consider memory, field position, internal state  “Single-layered” logic approach Performance (speed and coverage) Limited parameter exploration in our tests Not yet fully automated

23 Future Work Application of a “minimum weight perfect matching” algorithm (David Tudino) Hierarchical scene storage for improved search and storage performance State detection and learning (e.g. Hidden Markov Model) Pattern mining within scenes Better qualitative evaluation metrics (needed for automation)

24 Conclusions We contributed an extensible framework for research in RoboCup agent imitation  a conversion process (raw logs  higher-level representation)  customizable scene recognition and matching using k- nearest-neighbor  a RoboCup client agent based on this scene recognition algorithm Encouraging imitation results (semi-automated) Lots of direction for future work

25 Questions?

26 Krislet Behaviour Simple decision tree logic  Turn and look for ball  Run toward ball  Turn and look for goal  Kick ball to goal No concept of teams or strategies Stateless Deterministic

27 NewKrislet Behaviour Implement strategies as state machine “AntiSimpleton” prepackaged strategy  Attackers wait near center line until ball is near; then kicks the ball toward the goals  Defenders wait along the field; if the ball comes near, they kick it to the attacker and return to their position Deterministic

28 CMUnited Behaviour World Cup Championship Winner! Layered-learning model  Passing, dribbling skills at low level  Strategies at higher level Formations Communications Coach Player Modeling

29 Stateless Behaviour? Model agent as function f(a, b, c) = x If inputs (a, b, c) usually results in x, the agent is probably stateless If inputs (a, b, c) sometimes produces x, but other times produces y, there might be two states involved Subject to probability modeling

30 Discretizing Scenes: Issues Potential problems  Bias introduced  Boundary values/edging  Overfitting

31