Current Situation and Future Plans Abdelrahman Al-Ogail & Omar Enayet October

What do we do. What did we achieve till now. The Engine. The Paper. The Future. Short Term Goals. Long Term Goals.

AI Learning Make the machine learn. AI Planning Plan then re-plan according to new givens. Knowledge Sharing Let everyone know instantly what you knew through experience. P ROJECT R ESEARCH A REA

Experience Loss Static Scripts Computer AI relies on static scripting techniques. The Absence of sharing experience costs a lot P ROBLEM D EFINITION

Adaptive A.I. Making the Computer Opponent adapt to changes like human do. Mobile Experience Import/Export your experience ! O BJECTIVES

RTS Games Real-Time Strategy Games. Severe Time Constraints – Real-Time AI – Many Objects – Imperfect Information – Micro-Actions P ROJECT D OMAIN

Robotics For interest for military which uses battle simulations in training programs. War Simulation For the corporation of robots. Experimental Relevance They constitute well- defined environments to conduct experiments. M OTIVATIONS

A RTIFICIAL I NTELLIGENCE USED Case-based Reasoning Reinforcement Learning

Developed the engine with its basic stuff. Graduation Project, Excellent Degree. Participated with a booth in ICT’10 with the GP. Project was funded by ITIDA. Another graduation project & more than a master thesis are being done to extend our work in Ain-Shams University. Published a paper in ISDA’10 Had Contact with researchers on this field all over the world.

Maintaining the project blog Maintaining the project repository: Our research stuff repository: I-Strategizer (Engine’s code): Maintaining our own blogs: OmarsBrain.wordpress.com (Omar Enayet) AbdelrahmanOgail.wordpress.com (Abdelrahman Al-Ogail)

Adding new theory in the area of “Simulation of Human Behavior”. Developing a commercial AI Engine for RTS Games specifically and games in general. We already started and we have quite experience in game development. Participate in related contests around the world for AI Engines in RTS Games (As Robocup, AAAI Starcraft Competition, ORTS Competition).RobocupAAAI Starcraft CompetitionORTS Competition Initializing a major research group in Egypt in this field and become pioneers in it world wide.

Enhancing current engine which will efficiently be able to plan and learn when playing against static AI Use it as a test-bed to publish a number of papers: 1- Introducing the whole Agent model and theory in AI related conference. 2- Introducing the whole AI Game Engine from a game industry point of view in a game-industry conference. 3- More Details & Testing concerning the hybridization of Online Case based Planning and Reinforcement Learning ( the topic of our last paper) 4- Knowledge representation for plans and experience in RTS games. 5- Enhancing agent’s situation assessment algorithm. Publishing a paper concerning : Comparing Case-Based Reasoning to Reinforcement Learning.

Include different planning algorithms/systems and let agent use them and make an intensive comparison between these panning systems. Include different learning algorithms/systems and let agent use them and make an intensive comparison between these learning systems. Multi-Agent AI : machine collaboration with other machines, or machine collaboration with human players. Knowledge (Gaming Experience) Sharing. Opponent Modeling.

Named “I-Strategizer”. Based on an open-source RTS game engine called “Stratagus”. Currently tailored to serve an open source game Wargus.(clone of an old popular game called Warcraft 2 ) Still needs a lot of research and development for simulating human behavior and making it generic for strategy games. Coded in C++ and LUA scripts.

E NGINE A RCHITECTURE

Abstract —Research in learning and planning in real-time strategy (RTS) games is very interesting in several industries such as military industry, robotics, and most importantly game industry. A recent published work on online case-based planning in RTS Games does not include the capability of online learning from experience, so the knowledge certainty remains constant, which leads to inefficient decisions. In this paper, an intelligent agent model based on both online case- based planning (OLCBP) and reinforcement learning (RL) techniques is proposed. In addition, the proposed model has been evaluated using empirical simulation on Wargus (an open-source clone of the well known RTS game Warcraft 2). This evaluation shows that the proposed model increases the certainty of the case base by learning from experience, and hence the process of decision making for selecting more efficient, effective and successful plans. Keywords - Case-based Reasoning; Reinforcement Learning; Online Case-based Planning; Real-Time Strategy Games; Sarsa (λ) Learning; Eligibility Traces; Intelligent Agent.

The Problem : Learning from human then no learning from experience. Our Solution : Learning from experience is maintained through Reinforcement Learning (RL). Old Approach : Online Case-Based Planning (OLCBP) Our Approach : Hybridizing OLCBP with RL.

OLCBP ? Other approaches done to hybrid CBR with RL.

We used an RL Temporal-difference learning approach: SARSA(λ) Learning According to certain rules, SARSA(λ) Learning and interactions from the environment, the certainty value of cases in the case-base change -> Thus Learning from experience occurs.

Agent has learnt that building a smaller heavy army in that specific situation (the existence of a towers defense) is more preferable than building a larger light army. Similarly, the agent can evaluate the entire case base and learn the right choices.

Online case-based planning was hybridized with reinforcement learning in order to introduce an intelligent agent capable of planning and learning online using temporal difference with eligibility traces: Sarsa (λ) algorithm. The empirical evaluation has shown that the proposed model –unlike Darmok System - increases the certainty of the case base by learning from experience, and hence the process of decision making for selecting more efficient, effective and successful plans.

Implementing a prototype based on the proposed model. Developing a strategy/case base visualization tool capable of visualizing agent’s preferred playing strategy according to its learning history. This will help in tracking the learning curve of the agent. Finally, designing and developing a multi-agent system where agents are able to share their experiences together.