1. Intelligent Database Systems Lab 國立雲林科技大學 National Yunlin University of Science and Technology 1 Motivated Reinforcement Learning for Non-Player Characters in Persistent Computer Game Worlds Advisor : Dr. Hsu Presenter : Chia-Hao Yang Author : Kathryn Merrick, Mary Lou Maher SIGCHI 06

2. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 2  Motivation  Objective  Introduction  Method  Experiments  Discussion  Conclusions  Habituation SOM  Q-learning Outline

3. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 3 Motivation  Many NPC possess a fixed set of pre-programmed behaviors and lack the ability to adapt and evolve in time with their surroundings.

4. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 4 Objective  To create NPC that can both evolve and adapt with their environmental.

5. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 5 Introduction  Current technologies for NPCs ─ Reflexive agents Only recognized states will produce a response  State machines & rule-based approaches  EX : Baldur Gate & Dungeon Siege ─ Learning agents It can modify their internal structure to respect to some task.  Black and White ─ Reinforcement learning agents The agent records the reward signal. Then chooses an action which attempts to maximize the long- run sum of the values of the reward signal.  Tao Feng

6. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 6 Method  Motivated reinforcement learning agents ─ It use a motivation function to directs learning. ─ Skill development is dependent on the agent’s environment & these skills are developed progressively over time. S (t) – S (t-1) S (t-1) – S (t-2) Q-learning

7. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 7 Experiments  In order to experiment with MRL agent, we implemented a village scenario in Second Life. ─ Support character Trades people  Location, object, inventory sensor  Move to object, pick up object, use object effector  Ex : the pick, when used on the mine, will produce iron which can converted to weapons when used near the forge

8. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 8 Experiments ─ Partner character Vendor character  Location, object sensor  Move to object effector  Ex : In Ultima Online players can set up vendor characters to sell the goods they have crafted.

9. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 9 Conclusions  This paper has presented MRL agents as a means of creating non-player characters which can both evolve and adapt.  MRL agents explore their environment and learn new behaviors in response to interesting experiences, allowing them to display progressively evolving behavioral patterns.

10. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 10 Habituation SOM  An HSOM consists of a standard Self-Organizing Map with an additional habituating neuron connected to every clustering neuron of the SOM. ─

11. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 11 Q-Learning  It’s a part of reinforcement learning algorithm which has been widely used for many applications such as robotics, multi agent system, game, and etc.  It allows an agent to learn through training without teacher in unknown environment. ─ Modeling the Environment ─ putting similar matrix name Q in the brain of our agent reference

12. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 12 Q-Learning ─ algorithm ─ example reference ……