A DAPTIVE I NTELLIGENT AGENT IN REAL - TIME STRATEGY GAMES

P ROJECT M EMBERS Omar Enayet Amr Saqr Ahmed Atta Abdelrahman Al-Ogail Dr. Mostafa Aref Dr. Ibrahim Fathy

A GENDA 1) Problem Definition 2) Motivations. 3) Objectives 4) Related Work 5) Our Methodology. a) Architecture overview. b) Case representation. c) Case acquisition: Learning from Human Demonstration [OFFLINE-PHASE]. d) Case retrieval: Situation Assessment + E-Greedy Selection Policy + Similarity Metric using Laplace probability estimation rule. e) Online plan expansion & execution f) Case adaptation g) Case revision: SARSA(λ) Learning with Eligibility Traces [ONLINE-PHASE] h) Case optimization through abstraction/concretization 6) Results and Discussion. 7) Conclusion and Future Work. 8) Demo. 9) References.

P ROBLEM D EFINITION Learning Make the machine learn. Planning Plan then re-plan according to new givens. Knowledge Sharing Let everyone know instantly what you knew through experience.

P ROJECT D OMAIN RTS Games Real-Time Strategy Games. Severe Time Constraints – Real-Time AI – Many Objects – Imperfect Information – Micro-Actions

M OTIVATIONS Robotics For interest for military which uses battle simulations in training programs. War Simulation For the corporation of robots. Experimental Relevance They constitute well- defined environments to conduct experiments.

P ROBLEM D EFINITION PredictabilityNon-Adaptability Computer Opponent doesn’t adapt to changes in human actions. Computer Opponent actions easily predicted.

P ROBLEM D EFINITION - CONT Experience LossStatic Scripts Computer AI relies on static scripting techniques. The Absence of sharing experience costs a lot

O BJECTIVES Adaptive A.I. Making the Computer Opponent adapt to changes like human do. Mobile Experience

R ELATED W ORK Eric Kok introduced : Adaptive Reinforcement Learning Agents in RTS Games, which merged BDI Agents technology with Reinforcement Learning. Santi Ontanon introduced Darmok 2 which is an Online Case- Based Planning system designed to play Wargus. M.Johansen devised a CBR/RL system for learning micromanagement in real-time strategy games.

I-S TRATEGIZER A RCHITECTURE Goal Game State

I-S TRATEGIZER T O W ORLD A RCHITECTURE

A RCHITECTURE O VERVIEW

C ASE R ETRIEVAL - Retrieve most efficient case through: 1) Situation Assessment: Most Suitable. 2) E-Greedy Selection: a) Explore. b) Exploit (Predicted Performance): Euclidian & Laplace Probability Estimation Rule: Most Similar. Regularization Term: Most Confident.

S ITUATION A SSESSMENT - Situation Assessment : high level info using low level data. - Situation: high level representation of the world - Shallow Features: low level data. - Deep Features: high level data. “Provides feature space reduction”

C ASE R ETRIEVAL – S ITUATION A SSESSMENT O FFLINE S TAGE State Machine Empirical

C ASE R ETRIEVAL – S ITUATION A SSESSMENT O N -L INE S TAGE

C ASE R ETRIEVAL –S ELECTION P OLICY

F UTURE W ORK 1) Cooperative AI Agents. 2) Incorporation of local experience. 3) Strategy visualization tool. 4) Generic situation assessment. 5) Learn weights of Game State through neural network. 6) Online I-Strategizers. 7) Generic Abstraction/ Concretization.

Demo!

WinWargusGoal

BuildBase1Goal TrainMeleArmyGoal AttackGoal

WinWargusGoalBuildBase1Goal Build Town Hall Action Build 2 Farms Action Train Peons Action Build Barracks Action TrainArmyGoal Train 4 Grunts Action AttackGoal Attack enemy’s city Action

Plan Failed 

WinWargusGoal

BuildBase2GoalTrainRangedArmyGoalAttackGoal

WinWargusGoalBuildBase1Goal Upgrade StrongHold Action Build Troll-Lumber Mill Action Build BlackSmith Action TrainRangedArmyGoalTrain 2 Catapult ActionTrain 2 Grunt Action Train Axe Thrower Action AttackGoal Attack enemy’s city Action

Plan succeeded !

R EFERENCES [1] Martin Johansen Gunnerud. A CBR/RL system for learning micromanagement in real-time strategy games. In Norwegian University of Science and Technology, 2009 [2] Santi Ontañón, Neha Sugandh, Kinshuk Mishra, Ashwin Ram. On-Line Case-Based Planning. In Computational Intelligence, 26(1):84-119, [3] Brain Schwab. AI Game Engine Programming. Charles River Media, [4] Santi Ontañón and Kane Bonnette and Prafulla Mahindrakar and Marco A. G´omez-Mart´ın and Katie Long and Jainarayan Radhakrishnan and Rushabh Shah and Ashwin Ram. Learning from Human Demonstrations for Real-Time Case-Based Planning. In AAAI 2008

R EFERENCES – C ONT. [5] Kinshuk Mishra, Santiago Santi Ontañón, and Ashwin Ram. Situation Assessment for Plan Retrieval in Real-Time Strategy Games. In 9th European Conference on Case- Based Reasoning (ECCBR 2009), Trier, Germany. [6] Neha Sugandh and Santiago Santi Ontañón and Ashwin Ram. On-Line Case-Based Plan Adaptation for Real-Time Strategy Games. In Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (2008) [7] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning, An Introduction. MIT press, [8] Wikipedia, the free encyclopedia. [9] Michael Buro, Call for Research in RTS AI, AAAI 2004