1. Cognitive Model Comparisons: The Road to Artificial General Intelligence? Christian Lebiere (cl@cmu.edu)cl@cmu.edu Cleotilde Gonzalez (coty@cmu.edu)coty@cmu.edu Carnegie Mellon University Walter Warwick (wwarwick@alionscience.com)wwarwick@alionscience.com Alion Science & Technology

2. Challenges in AI & Cognitive Science Both fields have similar history of challenge problems despite compatible ends but different means Artificial Intelligence: maximize task performance – Started with ambitious but poorly defined test (Turing Test) – Evolved narrow, precise, overspecialized challenges (Chess) – Recently attempted broader tests (Robocup, Grand Challenge) Cognitive Science: fit human capabilities (design guide) – Started with ambitious, ill-defined capacities list (Newell Test) – Organized a series of complex task comparisons (AMBR, HEM) – Is taking on broader but integrated challenges (DSF?) 3/7/09Artificial General Intelligence Conference2

3. Cognitive Challenge Pitfalls Challenge is fundamentally about the task, not cognition – Too much task analysis and KE, too little cognitive theory Task is too narrow; too much data available – Reduces to data fitting – favors parameterization over principle Task is too specialized (typical cognitive psychology) – Single cognitive aspect – misses generality, integration Lack of common simulation environment – Each framework/theory only tackles what they do well Lack of comparable human data – Emphasizes functionality – loses cognitive constraints 3/7/09Artificial General Intelligence Conference3

4. Desirable Challenge Attributes Lightweight – Limit integration overhead and task analysis/knowledge eng. Fast – Rapid model development and collection of monte carlo runs Open-ended and dynamic – Less parameterization, generalization to emergent behavior Simple and tractable – Direct relation from cognitive mechanisms to behavioral data Integrated – Toward integrated agent capturing architectural interactions 3/7/09Artificial General Intelligence Conference4

5. DSF Challenge Comparison Dynamic Stocks and Flows – Instance of Dynamic Decision Making – Control a dynamic system given unexpected environmental fluctuations – Simple version of real-world situations (financial, ecological, technical, game) 3/7/09Artificial General Intelligence Conference5 Integrated tasks – Anticipate events – Control system Cognitive functions – Sequence learning - PC – Action selection - BG Implementation – VB on Windows – Text socket protocol

6. Generalization Scenarios Humans learn to control system over time for simple functions – Highly variable but quantifiable performance over learning process – Complexity of task scalable along a number of cognitive dimensions 3/7/09Artificial General Intelligence Conference6 Environmental i/o – Complex sequences – Stochastic noise – Multiple variables System dynamics – Feedback delay – Non-linear effects – Real-time control Multi-agent system – Other controllers – Payoff manipulations

7. DSF Comparison Schedule Official announcement expected March 15 Task environment with socket connection for model, data and documentation available on web site Symposium April 1 st at BRIMS conference (Sundance) Model submission by May 15 Best entries invited to symposium at European cognitive modeling conference (travel supported) Email DSFChallenge@gmail.com to be added to distribution list for official announcements/updatesDSFChallenge@gmail.com 3/7/09Artificial General Intelligence Conference7