Latest AI Research in RTS Games

Slides:



Advertisements
Similar presentations
Pat Langley Computational Learning Laboratory Center for the Study of Language and Information Stanford University, Stanford, California
Advertisements

Reinforcement Learning
ARCHITECTURES FOR ARTIFICIAL INTELLIGENCE SYSTEMS
Resource Management §A resource can be a logical, such as a shared file, or physical, such as a CPU (a node of the distributed system). One of the functions.
RL for Large State Spaces: Value Function Approximation
CS 443 Advanced OS Fabián E. Bustamante, Spring 2005 Resource Containers: A new Facility for Resource Management in Server Systems G. Banga, P. Druschel,
INTRODUCTION TO MODELING
Machine Learning in Computer Games Learning in Computer Games By: Marc Ponsen.
Introduction to Research Methodology
Artificial Intelligence in Game Design Introduction to Learning.
An Approach to Evaluate Data Trustworthiness Based on Data Provenance Department of Computer Science Purdue University.
Planning under Uncertainty
Joost Westra, Frank Dignum,Virginia Dignum Scalable Adaptive Serious Games using Agent Organizations.
Reinforcement Learning
Brent Dingle Marco A. Morales Texas A&M University, Spring 2002
1 Hybrid Agent-Based Modeling: Architectures,Analyses and Applications (Stage One) Li, Hailin.
Marakas: Decision Support Systems, 2nd Edition © 2003, Prentice-Hall Chapter Chapter 7: Expert Systems and Artificial Intelligence Decision Support.
Polyscheme John Laird February 21, Major Observations Polyscheme is a FRAMEWORK not an architecture – Explicitly does not commit to specific primitives.
Developing Ideas for Research and Evaluating Theories of Behavior
Omar Khaled Enayet – 4 th Year FCIS – Computer Science Department – August 2009 concerning planning, learning, Adaptation and opponent Modeling.
Science Inquiry Minds-on Hands-on.
The chapter will address the following questions:
Introduction to the Project AbdelRahman Al OgailOmar Khaled Enayet Under the Supervision Of : Dr. Ibrahim Fathy Moawad.
Section 2: Science as a Process
Why Analysis Process Refer to earlier chapters Models what the system will do makes it easier for understanding no environment considered (hence, system.
GENERAL CONCEPTS OF OOPS INTRODUCTION With rapidly changing world and highly competitive and versatile nature of industry, the operations are becoming.
Chapter 14: Artificial Intelligence Invitation to Computer Science, C++ Version, Third Edition.
Parallel Programming Models Jihad El-Sana These slides are based on the book: Introduction to Parallel Computing, Blaise Barney, Lawrence Livermore National.
Current Situation and Future Plans Abdelrahman Al-Ogail & Omar Enayet October
SLB /04/07 Thinking and Communicating “The Spiritual Life is Thinking!” (R.B. Thieme, Jr.)
Learning in Multiagent systems
Evaluation of software engineering. Software engineering research : Research in SE aims to achieve two main goals: 1) To increase the knowledge about.
Slides are based on Negnevitsky, Pearson Education, Lecture 12 Hybrid intelligent systems: Evolutionary neural networks and fuzzy evolutionary systems.
Introduction Many decision making problems in real life
Knowledge acquisition for adative game AI Marc Ponsen et al. Science of Computer programming vol. 67, pp , 2007 장수형.
Othello Artificial Intelligence With Machine Learning
Study on Genetic Network Programming (GNP) with Learning and Evolution Hirasawa laboratory, Artificial Intelligence section Information architecture field.
C ASE -B ASED P LANNER P LATFORM FOR RTS G AMES An Introduction Abdelrahman Al-Ogail Ahmed Atta.
Topics in Artificial Intelligence By Danny Kovach.
Machine Learning in Computer Games Marc Ponsen 11/29/04.
MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 1 Dynamic Sensor Resource Management for ATE MURI.
Ibrahim Fathy, Mostafa Aref, Omar Enayet, and Abdelrahman Al-Ogail Faculty of Computer and Information Sciences Ain-Shams University ; Cairo ; Egypt.
I Robot.
CHECKERS: TD(Λ) LEARNING APPLIED FOR DETERMINISTIC GAME Presented By: Presented To: Amna Khan Mis Saleha Raza.
1 What is OO Design? OO Design is a process of invention, where developers create the abstractions necessary to meet the system’s requirements OO Design.
Indirect Supervision Protocols for Learning in Natural Language Processing II. Learning by Inventing Binary Labels This work is supported by DARPA funding.
Introduction to Earth Science Section 2 Section 2: Science as a Process Preview Key Ideas Behavior of Natural Systems Scientific Methods Scientific Measurements.
Intelligent Database Systems Lab 國立雲林科技大學 National Yunlin University of Science and Technology 1 Evolving Reactive NPCs for the Real-Time Simulation Game.
1 Knowledge Acquisition and Learning by Experience – The Role of Case-Specific Knowledge Knowledge modeling and acquisition Learning by experience Framework.
RADHA-KRISHNA BALLA 19 FEBRUARY, 2009 UCT for Tactical Assault Battles in Real-Time Strategy Games.
Course Overview  What is AI?  What are the Major Challenges?  What are the Main Techniques?  Where are we failing, and why?  Step back and look at.
A Roadmap towards Machine Intelligence
New Product Development Page 1 Teddy Concurrent Engineering by Teddy Sjafrizal.
Reinforcement Learning AI – Week 22 Sub-symbolic AI Two: An Introduction to Reinforcement Learning Lee McCluskey, room 3/10
Artificial Intelligence: Research and Collaborative Possibilities a presentation by: Dr. Ernest L. McDuffie, Assistant Professor Department of Computer.
Othello Artificial Intelligence With Machine Learning Computer Systems TJHSST Nick Sidawy.
RADHA-KRISHNA BALLA 19 FEBRUARY, 2009 UCT for Tactical Assault Battles in Real-Time Strategy Games.
From NARS to a Thinking Machine Pei Wang Temple University.
Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 1 Authors : Siming Liu, Christopher Ballinger, Sushil Louis
Network Management Lecture 13. MACHINE LEARNING TECHNIQUES 2 Dr. Atiq Ahmed Université de Balouchistan.
Functionality of objects through observation and Interaction Ruzena Bajcsy based on Luca Bogoni’s Ph.D thesis April 2016.
 Negnevitsky, Pearson Education, Lecture 12 Hybrid intelligent systems: Evolutionary neural networks and fuzzy evolutionary systems n Introduction.
Introduction to Machine Learning, its potential usage in network area,
Object-Oriented Analysis and Design
Done Done Course Overview What is AI? What are the Major Challenges?
C.-S. Shieh, EC, KUAS, Taiwan
Chapter 5. The Bootstrapping Approach to Developing Reinforcement Learning-based Strategies in Reinforcement Learning for Adaptive Dialogue Systems, V.
Introduction Artificial Intelligent.
Market-based Dynamic Task Allocation in Mobile Surveillance Systems
Emir Zeylan Stylianos Filippou
Presentation transcript:

Latest AI Research in RTS Games Omar Khaled Enayet – July 2009

Index Why is AI Development Slow in RTS-Games. AI Areas needing more research. Adaptation Introduction Planning. Case-Based Planning. PDDL. Learning. Reinforcement Learning. Dynamic Scripting. Hierarchal Reinforcement Learning. Monte-Carlo Method. Temporal Difference Learning. Q-learning. Evolution Algorithms. Artificial Neural Networks Genetic Algorithms Hybrid CBR-RL. Opponent Modeling.

Why is AI Development slow in RTS Games ? RTS game worlds feature many objects, imperfect information, micro actions, and fast-paced action. By contrast, World–class AI players mostly exist for slow– paced, turn–based, perfect information games in which the majority of moves have global consequences and planning abilities therefore can be outsmarted by mere enumeration. Market dictated AI resource limitations. Up to now popular RTS games have been released solely by game companies who naturally are interested in maximizing their profit. Because graphics is driving games sales and companies strive for large market penetration only about 15% of the CPU time and memory is currently allocated for AI tasks. On the positive side, as graphics hardware is getting faster and memory getting cheaper, this percentage is likely to increase – provided game designers stop making RTS game worlds more realistic. Lack of AI competition. In classic two–player games tough competition among programmers has driven AI research to unmatched heights. Currently, however, there is no such competition among real–time AI researchers in games other than computer soccer. The considerable man–power needed for designing and implementing RTS games and the reluctance of game companies to incorporate AI APIs in their products are big obstacles to AI competition in RTS games.

AI Areas needing more research Adversarial real–time planning. In fine–grained realistic simulations, agents cannot afford to think in terms of micro actions such as “move one step North”. Instead, abstractions of the world state have to be found that allow AI programs to conduct forward searches in a manageable abstract space and to translate found solutions back into action sequences in the original state space. Because the environment is also dynamic, hostile, and smart — adversarial real–time planning approaches need to be investigated. Decision making under uncertainty. Initially, players are not aware of the enemies’ base locations and intentions. It is necessary to gather intelligence by sending out scouts and to draw conclusions to adapt. If no data about opponent locations and actions is available yet, plausible hypotheses have to be formed and acted upon. Opponent modeling, learning. One of the biggest shortcomings of current (RTS) game AI systems is their inability to learn quickly. Human players only need a couple of games to spot opponents’ weaknesses and to exploit them in future games. New efficient machine learning techniques have to be developed to tackle these important problems.

AI Areas needing more research (2) Spatial and temporal reasoning. Static and dynamic terrain analysis as well as understanding temporal relations of actions is of utmost importance in RTS games — and yet, current game AI programs largely ignore these issues and fall victim to simple common–sense reasoning . Resource management. Players start the game by gathering local resources to build up defenses and attack forces, to upgrade weaponry, and to climb up the technology tree. At any given time the players have to balance the resources they spend in each category. For instance, a player who chooses to invest too many resources into upgrades, will become prone to attacks because of an insufficient number of units. Proper resource management is therefore a vital part of any successful strategy

AI Areas needing more research (3) Collaboration. In RTS games groups of players can join forces and intelligence. How to coordinate actions effectively by communication among the parties is a challenging research problem. For instance, in case of mixed human/AI teams, the AI player often behaves awkwardly because it does not monitor the human’s actions, cannot infer the human’s intentions, and fails to synchronize attacks. Pathfinding. Finding high–quality paths quickly in 2D terrains is of great importance in RTS games. In the past, only a small fraction of the CPU time could be devoted to AI tasks, of which finding shortest paths was the most time consuming. Hardware graphics accelerators are now allowing programs to spend more time on AI tasks. Still, the presence of hundreds of moving objects and the urge for more realistic simulations in RTS games make it necessary to improve and generalize pathfinding algorithms. Keeping unit formations and taking terrain properties, minimal turn radii, inertia, enemy influence, and fuel consumption into account greatly complicates the once simple problem of finding shortest paths.

Introduction to Adaptation in RTS Games Current Implementation of RTS Games applies extensive usage of FSM that makes them highly predictable. Adaptation is achieved either through Learning or planning or a mixture of both Planning is beginning to appear in commercial games such as DemiGod and Latest Total War Game. Learning has limited success so far. Developers are experimenting on replacing the ordinary decision making systems (FSM, FUSM, Scripting, Decision Trees, and Markov Systems) with Learning Techniques

Case Based Planning Using Case-Based Reasoning with planning. Plan recognition refers to the act of an agent observing the actions of another agent whether it be human or computer-based with the intent of predicting its future actions, intentions, or goals. Several approaches can be used to perform plan recognition namely deductive, abductive, probabilistic, and case-based. It can also be classified as either intended or keyhole. An intended case-based plan recognition system assumes that the agent or the user is actively giving signals or input to the sensing again to denote plans and intensions. In the case of a real-time strategy game, the user or player is focused on playing the game and not focused on trying to convey his or her intention to the sensing agent, hence for this scenario, we would be classified as keyhole plan recognition wherein predictions are based on indirect observations about the users actions in a certain scenario.

Case Based Planning – Papers (1) After 2003 : CASE-BASED PLAN RECOGNITION FOR REAL-TIME STRATEGY GAMES : employs case-based plan recognition for non-player characters so as to minimize their predictability. After 2004 :On the Role of Explanation for Hierarchical Case-Based Planning in Real-Time Strategy Games : describes an application of hierarchical case-based planning that involves reasoning in the context of real-time strategy games, describes a representation for explanations in this context, and detail four types of explanations.

Case Based Planning – Papers (2) 2005 : Learning to Win: Case-Based Plan Selection in a Real-Time Strategy Game : Introduced the case-based tactician (CAT) which removes the assumption a of a static opponent which was used with Ponson and Spronck(2004) who used genetic algorithms that searches in a plan space and a weighting algorithm (dynamic scripting) that biases sub-plan retreival.CAT significantly outperforms the best among a set of genetically evolved plans when tested against random opponents. 2005 : Defeating Novel Opponents in a Real-Time Strategy Game : The Case-Based Tactician (CAT) system (created by the same authors) uses case-based reasoning to learn to win the real-time strategy game Wargus. Previous work has shown CAT’s ability to defeat a randomly selected opponent from a set against which it has trained. They now focus on the task of defeating a selected opponent while training on others. They describe CAT’s algorithm and report its cross-validation performance against a set of Wargus opponents.

Case Based Planning – Papers (3) 2007 : Case-Based Planning and Execution for Real-Time Strategy Games : presents a real-time case based planning and execution approach designed to deal with RTS games. propose to extract behavioral knowledge from expert demonstrations in form of individual cases. This knowledge can be reused via a case based behavior generator that proposes behaviors to achieve the specific open goals in the current plan. 2007 : Mining Replays of Real-Time Strategy Games to Learn Player Strategies :the authors use replays of the commercial RTS game StarCraft to evaluate human player behaviors and to construct an intelligent system to learn human-like decisions and behaviors. A case-based reasoning approach was applied for the purpose of training our system to learn and predict player strategies. Our analysis indicates that the proposed system is capable of learning and predicting individual player strategies, and that players provide evidence of their personal characteristics through their building construction order.

Case Based Planning – Papers (4) 2008 : Learning from Human Demonstrations for Real-Time Case-Based Planning : One of the main bottlenecks in deploying casebased planning systems is authoring the case-base of plans. In this paper they present a collection of algorithms that can be used to automatically learn plans from human demonstrations. Theiralgorithms are based on the basic idea of a plan dependency graph, which is a graph that captures the dependencies among actions in a plan. Such algorithms are implemented in a system called Darmok 2 (D2), a case-based planning system capable of general game playing with a focus on real-time strategy (RTS) games. Theyevaluate D2 with a collection of three different games with promising results. 2008 : On-Line Case-Based Plan Adaptation for Real-Time Strategy Games :They have developed on-line case-based planning techniques that are effective .In this paper, They extend their earlier work using ideas from traditional planning to inform the real-time adaptation of plans. In their framework, when a plan is retrieved, a plan dependency graph is inferred to capture the relations between actions in the plan. The plan is then adapted in real-time using its plan dependency graph. This allows the system to create and adapt plans in an efficient and effective manner while performing the task. The approach is evaluated using WARGUS, a well-known real-time strategy game.

PDDL (Papers) 2006 : A FIRST LOOK AT BUILD-ORDER OPTIMIZATION IN REAL-TIME STRATEGY GAMES : approaches the real-time planning problem by considering build-order optimization in real-time strategy games. This problem class can be formulated as a resource accumulation and allocation problem where an agent has to decide what objects to produce at what time in order to meet one of two goals: either maximizing the number of produced objects in a given time period or producing a certain number of objects as fast as possible. They identify challenges related to this planning problem, namely creating and destroying objects, concurrent action generation and execution, and present ideas how to address them. They use PDDL in their work.

Reinforcement Learning – Papers (1) Hierarchical RL After 2006 : Hierarchical Reinforcement Learning with Deictic representation in a computer game :Without applying abstraction and generalization techniques, many traditional machine learning techniques, such as reinforcement learning, will fail to learn efficiently. In this paper they examine extensions of reinforcement learning that scale to the complexity of computer games. In particular they look at hierarchical reinforcement learning applied to a learning task in a real time-strategy computer game. Moreover, they employ a deictic state representation that reduces the complexity compared to a propositional representation and allows the adaptive agent to learn a generalized policy, i.e., it is capable of transferring knowledge to unseen task instances. They found that hierarchical reinforcement learning significantly outperforms flat reinforcement learning for our task.

Reinforcement Learning – Papers (2) Dynamic Scripting 2006 : Goal-Directed Hierarchical Dynamic Scripting for RTS Games : This paper suggests a goal-directed hierarchical dynamic scripting approach for incorporating learning into real-time strategy games. Two alternatives for shortening the re-adaptation time when using dynamic scripting are also presented. Finally, this paper presents an effective way of throttling the performance of the adaptive artificial intelligence system. Put together, the approach entails the possibility of an artificial intelligence opponent to be challenging for a human player, but not too challenging. 2008 : Automatically Acquiring Domain Knowledge For Adaptive Game AI Using Evolutionary Learning : introduce AKAD system, which uses an evolutionary algorithm that improves dynamic scripting performance on RTS Games. They conclude that high-quality domain knowledge (i.e., tactics) can be automatically generated for strong adaptive AI opponents in RTS games. This reduces the time and effort required by game developers to create intelligent game AI, thus freeing them to focus on other important topics (e.g., storytelling, graphics).

Reinforcement Learning – Papers (2) Monte-Carlo 2003 : UCT for Tactical Assault Battles in Real-Time Strategy Games : uses a Monte-Carlo algorithm called UCT for optimization of user-specified object functions, They present an evaluation of their approach on a range of tactical assault problems with different objectives in the RTS game Wargus. The results indicate that their planner is able to generate superior plans compared to several baselines and a human player. After 2004 : Monte Carlo Planning in RTS Games : They present a framework—MCPlan—for Monte Carlo planning, identify its performance parameters, and analyze the results of an implementation in a capture– the–flag game.

Reinforcement Learning – Papers (1) Monte-Carlo VS Dynamic Scripting 2008 : Adaptive reinforcement learning agents in RTS games : compares Dynamic Scripting with Monte Carlo Methods with all details, talks about drawbacks of using TD-learning, states many future research to the general topic as well as specific algorithms.

Reinforcement Learning – Papers (3) Temporal Difference Learning After 2005 : Establishing an Evaluation Function for RTS games : Because of the high complexity of modern video games, the task to create a suitable evaluation function for adaptive game AI is quite difficult. In our research we aim at fully automatically generating a good evaluation function for adaptive game AI. This paper describes our approach, and discusses the experiments performed in the RTS game Spring. TD-learning is applied for establishing two different evaluation functions, one for a perfect-information environment, and one for an imperfect-information environment. From our results we may conclude that both evaluation functions are able to predict the outcome of a Spring game reasonably well, i.e., they are correct in about 75% of all games played. 2005 : Learning Unit Values in Wargus Using Temporal Differences :In order to use a learning method in a computer game to improve the perfomance of computer controlled entities, a fitness function is required. In real-time strategy (RTS) games it seems obvious to found the fitness function on the game score, which usually is derived from the number and type of units a player has killed. In practice the value of a unit type is set manually. This paper proposes to use Temporal Difference learning (TD-learning) to determine the value of a unit type. An experiment was performed to determine good unit values for use by the learning mechanism ‘dynamic scripting’ in Wargus. The results of this experiment demonstrate significantly improved learning performance using the newlydetermined unit values, compared to using the original unit values which were manually determined by the Wargus game developers.

Hybrid CBR-RL (Papers) 2007 : Transfer Learning in Real-Time Strategy Games Using Hybrid CBR/RL : they present a multilayered architecture named CAse-Based Reinforcement Learner (CARL). It uses a novel combination of Case-Based Reasoning (CBR) and Reinforcement Learning (RL) to achieve transfer while playing against the Game AI across a variety of scenarios in MadRTSTM, a commercial Real Time Strategy game. Their experiments demonstrate that CARL not only performs well on individual tasks but also exhibits significant performance gains when allowed to transfer knowledge from previous tasks.

Genetic Algorithms (Papers) Co-evolving Real-Time Strategy Game Playing Influence Map Trees With Genetic Algorithms : They investigate the use of genetic algorithms to play real-time computer strategy games and focus on solving the complex spatial reasoning problems found within these games. To overcome the knowledge acquisition bottleneck found in using traditional expert systems, scripts, and decision trees as done in most game AI, we use genetic algorithms to evolve game players. The spatial decision makers in these game players use influence maps as a basic building block, from which they construct and evolve influence map trees containing complex game playing strategies. With co-evolution they attain ”arms race” like progress, leading to the evolution of robust players superior to their hand coded counterparts. 2003 : Human-Like Behavior in RTS Games using genetic algorithms : In this thesis they will present the results from an experiment aiming at testing strategy game AI. Test persons played against traditional strategy game AI, a genetic algorithm AI, and other humans to see if they experienced any differences in the behavior of the opponents. 2008 : Stochastic Plan Optimization in Real-Time Strategy Games : They present a domain independent off-line adaptation technique called Stochastic Plan Optimization for finding and improving plans in real-time strategy games. Their method is based on ideas from genetic algorithms but we utilize a different representation for our plans and an alternate initialization procedure for our search process. The key to their technique is the use of expert plans to initialize their search in the most relevant parts of plan space. Their experiments validate this approach using their existing case based reasoning system Darmok in the real- time strategy game Wargus, a clone of Warcraft II.

Opponent Modeling – Papers After 2007 : OPPONENT MODELING IN REAL-TIME STRATEGY GAMES : One feature of realistic behavior in game AI is the ability to recognize the strategy of the opponent player. This is known as opponent modeling. In this paper, they propose an approach of opponent modeling based on hierarchically structured models. The top-level of the hierarchy can classify the general play style of the opponent. The bottom-level of the hierarchy can classify specific strategies that further define the opponent's behavior. Experiments that test the approach are performed in the RTS game Spring. From their results they may conclude that the approach can be successfully used to classify the strategy of an opponent in the Spring game. 2007 : Hierarchical Opponent Models for Real-Time Strategy Games : This paper describes an approach of opponent modeling using hierarchical structured models. As classifiers, two different approaches have been used for the different hierarchical levels, in order to test the effectiveness of this approach. The first classifier uses fuzzy models whereas the second classifier is a modification of the concept of discounted rewards from game theory. It is observed that the fuzzy classifier shows a stable convergence to a correct classification with high confidence rating. The experiments regarding the discounted reward approach revealed, that for some sub-models this approach showed a similar well conversion as the fuzzy classifier. However, for other sub-models only mediocre results were achieved with a late and unstable convergence. They conclude that this approach is suitable for real-time strategy games and can be reliable for each sub- model with further research. 2009 : Design of Autonomous Systems Learning adaptive play in a Real Time Strategy game : In this paper we propose a method for an adaptive game player, by modeling the opponents' strategy and adjusting our strategy accordingly. To do so we use the Battlecode framework [1], and develop three basic strategies in accordance with three standard strategies in Real-Time Strategy games. We explain how all agents organize their behavior using roles and message passing. We then propose a system for adaptive play and show that an adaptive player outperforms a static one.