1. Pat Langley Institute for the Study of Learning and Expertise Palo Alto, California http://www.isle.org/ A Cognitive Architecture for Complex Learning Thanks to D. Choi, K. Cummings, N. Nejati, S. Rogers, S. Sage, and D. Shapiro for their contributions. This talk reports research. funded by grants from DARPA IPTO and the National Science Foundation, which are not responsible for its contents.

2. Reasons for Studying Cognitive Architectures

3. The original goal of artificial intelligence was to design and implement computational artifacts that: handled difficult tasks that require cognitive processing; handled difficult tasks that require cognitive processing; combined many capabilities into integrated systems; combined many capabilities into integrated systems; provided insights into the nature of mind and intelligence. provided insights into the nature of mind and intelligence. Cognitive Systems Instead, modern AI has divided into many subfields that care little about cognition, systems, or intelligence. But the challenge remains and we need far more research on cognitive systems.

4. The Fragmentation of AI Research action perception reasoning learning planning language

5. The Domain of In-City Driving Consider driving a vehicle in a city, which requires: selecting routes selecting routes obeying traffic lights obeying traffic lights avoiding collisions avoiding collisions being polite to others being polite to others finding addresses finding addresses staying in the lane staying in the lane parking safely parking safely stopping for pedestrians stopping for pedestrians following other vehicles following other vehicles delivering packages delivering packages These tasks range from low-level execution to high-level reasoning.

6. Newells Critique move beyond isolated phenomena and capabilities to develop complete models of intelligent behavior; move beyond isolated phenomena and capabilities to develop complete models of intelligent behavior; demonstrate our systems intelligence on the same range of domains and tasks as humans can handle; demonstrate our systems intelligence on the same range of domains and tasks as humans can handle; view artificial intelligence and cognitive psychology as close allies with distinct but related goals; view artificial intelligence and cognitive psychology as close allies with distinct but related goals; evaluate these systems in terms of generality and flexibility rather than success on a single class of tasks. evaluate these systems in terms of generality and flexibility rather than success on a single class of tasks. However, there are different paths toward achieving such systems. In 1973, Allen Newell argued You cant play twenty questions with nature and win. Instead, he proposed that we:

7. A System with Communicating Modules action perception reasoning learning planning language software engineering / multi-agent systems

8. action perception reasoning learning planning language short-term beliefs and goals A System with Shared Short-Term Memory blackboard architectures

9. Newells vision for research on theories of intelligence was that: cognitive systems should make strong theoretical assumptions about the nature of the mind; cognitive systems should make strong theoretical assumptions about the nature of the mind; theories of intelligence should change only gradually, as new structures or processes are determined necessary; theories of intelligence should change only gradually, as new structures or processes are determined necessary; later design choices should be constrained heavily by earlier ones, not made independently. later design choices should be constrained heavily by earlier ones, not made independently. Integration vs. Unification A successful framework is all about mutual constraints, and it should provide a unified theory of intelligent behavior. He associated these aims with the idea of a cognitive architecture.

10. A System with Shared Long-Term Memory action perception reasoning learning planning language short-term beliefs and goals long-term memory structures cognitive architectures

11. A Constrained Cognitive Architecture action perception reasoning learning planning language short-term beliefs and goals long-term memory structures

12. As traditionally defined and utilized, a cognitive architecture: specifies the infrastructure that holds constant over domains, as opposed to knowledge, which varies. specifies the infrastructure that holds constant over domains, as opposed to knowledge, which varies. models behavior at the level of functional structures and processes, not the knowledge or implementation levels. models behavior at the level of functional structures and processes, not the knowledge or implementation levels. commits to representations and organizations of knowledge and processes that operate on them. commits to representations and organizations of knowledge and processes that operate on them. comes with a programming language for encoding knowledge and constructing intelligent systems. comes with a programming language for encoding knowledge and constructing intelligent systems. Aspects of Cognitive Architectures Early candidates were cast as production system architectures, but alternatives have gradually expanded the known space.

13. The I CARUS Architecture In this talk I will use one such framework I CARUS to illustrate the advantages of cognitive architectures. I CARUS incorporates a variety of assumptions from psychological theories; the most basic are that: These claims give I CARUS much in common with other cognitive architectures like ACT-R, Soar, and Prodigy. 1.Short-term memories are distinct from long-term stores 2.Memories contain modular elements cast as list structures 3.Long-term structures are accessed through pattern matching 4.Cognition occurs in retrieval/selection/action cycles 5.Performance and learning compose elements in memory

14. Memories and Representations in the I CARUS Architecture

15. A cognitive architecture makes a specific commitment to: long-term memories that store knowledge and procedures; long-term memories that store knowledge and procedures; short-term memories that store beliefs and goals; short-term memories that store beliefs and goals; sensori-motor memories that hold percepts and actions. sensori-motor memories that hold percepts and actions. Architectural Commitment to Memories Each memory holds different content the agent uses in activities. For each memory, a cognitive architecture also commits to: the encoding of contents in that memory; the encoding of contents in that memory; the organization of structures within the memory; the organization of structures within the memory; the connections among structures across memories. the connections among structures across memories.

16. concepts and skills encode different aspects of knowledge that are stored as distinct cognitive structures; concepts and skills encode different aspects of knowledge that are stored as distinct cognitive structures; cognition occurs in a physical context, with concepts and skills being grounded in perception and action; cognition occurs in a physical context, with concepts and skills being grounded in perception and action; many mental structures are relational in nature, in that they describe connections or interactions among objects; many mental structures are relational in nature, in that they describe connections or interactions among objects; long-term memories have hierarchical organizations that define complex structures in terms of simpler ones; long-term memories have hierarchical organizations that define complex structures in terms of simpler ones; each element in a short-term memory is an active version of some structure in long-term memory. each element in a short-term memory is an active version of some structure in long-term memory. Ideas about Representation Cognitive psychology makes important representational claims: I CARUS adopts these assumptions about the contents of memory.

17. I CARUS Memories Long-TermConceptualMemory Long-Term Skill Memory Short-Term Belief Memory Short-Term Goal Memory Environment PerceptualBuffer MotorBuffer

18. Representing Long-Term Structures Conceptual clauses: A set of relational inference rules with perceived objects or defined concepts in their antecedents; Conceptual clauses: A set of relational inference rules with perceived objects or defined concepts in their antecedents; Skill clauses: A set of executable skills that specify: Skill clauses: A set of executable skills that specify: a head that indicates a goal the skill achieves; a head that indicates a goal the skill achieves; a single (typically defined) precondition; a single (typically defined) precondition; a set of ordered subgoals or actions for achieving the goal. a set of ordered subgoals or actions for achieving the goal. These define a specialized class of hierarchical task networks in which each task corresponds to a goal concept. I CARUS syntax is very similar to Nau et al.s SHOP2 formalism for hierarchical task networks. I CARUS encodes two forms of general long-term knowledge:

19. I CARUS Concepts for In-City Driving ((in-rightmost-lane ?self ?clane) :percepts ((self ?self) (segment ?seg) :percepts ((self ?self) (segment ?seg) (line ?clane segment ?seg)) :relations ((driving-well-in-segment ?self ?seg ?clane) :relations ((driving-well-in-segment ?self ?seg ?clane) (last-lane ?clane) (not (lane-to-right ?clane ?anylane)))) ((driving-well-in-segment ?self ?seg ?lane) :percepts ((self ?self) (segment ?seg) (line ?lane segment ?seg)) :percepts ((self ?self) (segment ?seg) (line ?lane segment ?seg)) :relations ((in-segment ?self ?seg) (in-lane ?self ?lane) :relations ((in-segment ?self ?seg) (in-lane ?self ?lane) (aligned-with-lane-in-segment ?self ?seg ?lane) (centered-in-lane ?self ?seg ?lane) (steering-wheel-straight ?self))) ((in-lane ?self ?lane) :percepts ((self ?self segment ?seg) (line ?lane segment ?seg dist ?dist)) :percepts ((self ?self segment ?seg) (line ?lane segment ?seg dist ?dist)) :tests ((> ?dist -10) ( ?dist -10) (<= ?dist 0))) ((in-segment ?self ?seg) :percepts ((self ?self segment ?seg) (segment ?seg))) :percepts ((self ?self segment ?seg) (segment ?seg)))

20. ((in-rightmost-lane ?self ?line) :percepts ((self ?self) (line ?line)) :percepts ((self ?self) (line ?line)) :start ((last-lane ?line)) :start ((last-lane ?line)) :subgoals ((driving-well-in-segment ?self ?seg ?line))) :subgoals ((driving-well-in-segment ?self ?seg ?line))) ((driving-well-in-segment ?self ?seg ?line) :percepts ((segment ?seg) (line ?line) (self ?self)) :percepts ((segment ?seg) (line ?line) (self ?self)) :start ((steering-wheel-straight ?self)) :start ((steering-wheel-straight ?self)) :subgoals ((in-segment ?self ?seg) :subgoals ((in-segment ?self ?seg) (centered-in-lane ?self ?seg ?line) (aligned-with-lane-in-segment ?self ?seg ?line) (steering-wheel-straight ?self))) ((in-segment ?self ?endsg) :percepts ((self ?self speed ?speed) (intersection ?int cross ?cross) :percepts ((self ?self speed ?speed) (intersection ?int cross ?cross) (segment ?endsg street ?cross angle ?angle)) :start ((in-intersection-for-right-turn ?self ?int)) :start ((in-intersection-for-right-turn ?self ?int)) :actions (( steer 1))) :actions (( steer 1))) I CARUS Skills for In-City Driving

21. Representing Short-Term Beliefs/Goals (current-street me A)(current-segment me g550) (lane-to-right g599 g601)(first-lane g599) (last-lane g599)(last-lane g601) (at-speed-for-u-turn me)(slow-for-right-turn me) (steering-wheel-not-straight me)(centered-in-lane me g550 g599) (in-lane me g599)(in-segment me g550) (on-right-side-in-segment me)(intersection-behind g550 g522) (building-on-left g288)(building-on-left g425) (building-on-left g427)(building-on-left g429) (building-on-left g431)(building-on-left g433) (building-on-right g287)(building-on-right g279) (increasing-direction me)(buildings-on-right g287 g279)

22. Encoding Perceived Objects (self me speed 5 angle-of-road -0.5 steering-wheel-angle -0.1) (segment g562 street 1 dist -5.0 latdist 15.0) (line g564 length 100.0 width 0.5 dist 35.0 angle 1.1 color white segment g562) (line g565 length 100.0 width 0.5 dist 15.0 angle 1.1 color white segment g562) (line g563 length 100.0 width 0.5 dist 25.0 angle 1.1 color yellow segment g562) (segment g550 street A dist oor latdist nil) (line g600 length 100.0 width 0.5 dist -15.0 angle -0.5 color white segment g550) (line g601 length 100.0 width 0.5 dist 5.0 angle -0.5 color white segment g550) (line g599 length 100.0 width 0.5 dist -5.0 angle -0.5 color yellow segment g550) (intersection g522 street A cross 1 dist -5.0 latdist nil) (building g431 address 99 street A c1dist 38.2 c1angle -1.4 c2dist 57.4 c2angle -1.0) (building g425 address 25 street A c1dist 37.8 c1angle -2.8 c2dist 56.9 c2angle -3.1) (building g389 address 49 street 1 c1dist 49.2 c1angle 2.7 c2dist 53.0 c2angle 2.2) (sidewalk g471 dist 15.0 angle -0.5) (sidewalk g474 dist 5.0 angle 1.07) (sidewalk g469 dist -25.0 angle -0.5) (sidewalk g470 dist 45.0 angle 1.07) (stoplight g538 vcolor green hcolor red))

23. Hierarchical Structure of Long-Term Memory concepts skills Each concept is defined in terms of other concepts and/or percepts. Each skill is defined in terms of other skills, concepts, and percepts. I CARUS organizes both concepts and skills in a hierarchical manner.

24. Hierarchical Structure of Long-Term Memory conceptsskills For example, the skill highlighted here refers directly to the highlighted concepts. I CARUS interleaves its long-term memories for concepts and skills.

25. Performance and Learning in the I CARUS Architecture

26. In addition, a cognitive architecture makes commitments about: performance processes for: performance processes for: retrieval, matching, and selection retrieval, matching, and selection inference and problem solving inference and problem solving perception and motor control perception and motor control learning processes that: learning processes that: generate new long-term knowledge structures generate new long-term knowledge structures refine and modulate existing structures refine and modulate existing structures Architectural Commitment to Processes In most cognitive architectures, performance and learning are tightly intertwined.

27. humans can handle multiple goals with different priorities, which can interrupt tasks to which attention returns later; humans can handle multiple goals with different priorities, which can interrupt tasks to which attention returns later; conceptual inference, which typically occurs rapidly and unconsciously, is more basic than problem solving; conceptual inference, which typically occurs rapidly and unconsciously, is more basic than problem solving; humans often resort to means-ends analysis to solve novel, unfamiliar problems; humans often resort to means-ends analysis to solve novel, unfamiliar problems; mental problem solving requires greater cognitive resources than execution of automatized skills; mental problem solving requires greater cognitive resources than execution of automatized skills; problem solving often occurs in a physical context, with mental processing being interleaved with execution. problem solving often occurs in a physical context, with mental processing being interleaved with execution. Ideas about Performance I CARUS embodies these ideas in its performance mechanisms. Cognitive psychology makes clear claims about performance:

28. I CARUS Functional Processes Long-TermConceptualMemory Short-TermBeliefMemory Short-Term Goal Memory ConceptualInference SkillExecution Perception Environment PerceptualBuffer Problem Solving Skill Learning MotorBuffer Skill Retrieval and Selection Long-Term Skill Memory

29. Cascaded Integration in I CARUS This contrasts sharply with multi-agent approaches to building AI systems and reflects the notion of a unified cognitive architecture. conceptual inference skill execution problem solving learning I CARUS adopts a cascaded approach to system integration in which lower-level modules produce results for higher-level ones.

30. I CARUS Inference-Execution Cycle 1.places descriptions of sensed objects in the perceptual buffer; 2.infers instances of concepts implied by the current situation; 3.finds paths through the skill hierarchy from top-level goals; 4.selects one or more applicable skill paths for execution; 5.invokes the actions associated with each selected path. On each successive execution cycle, the I CARUS architecture: I CARUS agents are teleoreactive (Nilsson, 1994) in that they are executed reactively but in a goal-directed manner.

31. Inference and Execution in I CARUS concepts skills Concepts are matched bottom up, starting from percepts. Skill paths are matched top down, starting from intentions. I CARUS matches patterns to recognize concepts and select skills.

32. Traditional theories claim that human problem solving occurs in response to unfamiliar tasks and involves: the mental inspection and manipulation of list structures; the mental inspection and manipulation of list structures; search through a space of states generated by operators; search through a space of states generated by operators; backward chaining from goals through means-ends analysis; backward chaining from goals through means-ends analysis; a shift from backward to forward chaining with experience. a shift from backward to forward chaining with experience. The Standard Theory of Problem Solving These claims characterize problem solving accurately, but this does not mean they are complete.

33. The Physical Context of Problem Solving 1.places descriptions of sensed objects in the perceptual buffer; 2.infers instances of concepts implied by the current situation; 3.finds paths through the skill hierarchy from top-level goals; 4.selects one or more applicable skill paths for execution; 5.invokes the actions associated with each selected path. I CARUS is a cognitive architecture for physical, embodied agents. On each successive perception-execution cycle, the architecture: Problem solving in I CARUS builds upon this basic ability to recognize physical situations and execute skills therein.

34. Abstraction from Physical Details conceptual inference augments perceptions using high-level concepts that provide abstract state descriptions. conceptual inference augments perceptions using high-level concepts that provide abstract state descriptions. execution operates over high-level durative skills that serve as abstract problem-space operators. execution operates over high-level durative skills that serve as abstract problem-space operators. both inference and execution occur in an automated manner that demands few attentional resources. both inference and execution occur in an automated manner that demands few attentional resources. I CARUS typically pursues problem solving at an abstract level: However, concepts are always grounded in primitive percepts and skills always terminate in executable actions. I CARUS holds that cognition relies on a symbolic physical system which utilizes mental models of the environment.

35. A Successful Problem-Solving Trace (ontable A T) (on B A) (on C B) (hand-empty) (clear C) (unst. C B) (unstack C B) (clear B) (putdown C T) (unst. B A) (unstack B A) (clear A) (holding C)(hand-empty) (holding B) A B CB A C initial state goal

36. Interleaved Problem Solving and Execution chains backward off skills that would produce the goal; chains backward off skills that would produce the goal; chains backwards off concepts if no skills are available; chains backwards off concepts if no skills are available; creates subgoals based on skill or concept conditions; creates subgoals based on skill or concept conditions; pushes these subgoals onto a goal stack and recurses; pushes these subgoals onto a goal stack and recurses; executes any selected skill as soon as it is applicable. executes any selected skill as soon as it is applicable. I CARUS includes a module for means-ends problem solving that: Embedding execution within problem solving reduces memory load and uses the environment as an external store.

37. I CARUS Interleaves Execution and Problem Solving Executed plan Problem ? Skill Hierarchy Primitive Skills Reactive Execution impasse? Problem Solving yes no

38. Interleaving Reactive Control and Problem Solving Solve(G) Push the goal literal G onto the empty goal stack GS. On each cycle, If the top goal G of the goal stack GS is satisfied, Then pop GS. Else if the goal stack GS does not exceed the depth limit, Let S be the skill instances whose heads unify with G. If any applicable skill paths start from an instance in S, Then select one of these paths and execute it. Else let M be the set of primitive skill instances that have not already failed in which G is an effect. If the set M is nonempty, Then select a skill instance Q from M. Push the start condition C of Q onto goal stack GS. Else if G is a complex concept with the unsatisfied subconcepts H and with satisfied subconcepts F, Then if there is a subconcept I in H that has not yet failed, Then push I onto the goal stack GS. Else pop G from the goal stack GS and store information about failure with G's parent. Else pop G from the goal stack GS. Store information about failure with G's parent. Else if G is a complex concept with the unsatisfied subconcepts H and with satisfied subconcepts F, Then if there is a subconcept I in H that has not yet failed, Then push I onto the goal stack GS. Else pop G from the goal stack GS and store information about failure with G's parent. Else pop G from the goal stack GS. Store information about failure with G's parent. This is traditional means-ends analysis, with three exceptions: (1) conjunctive goals must be defined concepts; (2) chaining occurs over both skills/operators and concepts/axioms; and (3) selected skills are executed whenever applicable.

39. Restarting on Problems detecting when action has made backtracking impossible; detecting when action has made backtracking impossible; storing the goal context to avoid repeating the error; storing the goal context to avoid repeating the error; physically restarting the problem in the initial situation; physically restarting the problem in the initial situation; repeating this process until succeeding or giving up. repeating this process until succeeding or giving up. Even when combined with backtracking, eager execution can lead problem solving to unrecoverable states. I CARUS problem solver handles such untenable situations by: This strategy produces quite different behavior from the purely mental systematic search assumed by most models.

40. efforts to overcome impasses during problem solving can lead to the acquisition of new skills; efforts to overcome impasses during problem solving can lead to the acquisition of new skills; learning can transform backward-chaining heuristic search into more informed forward-chaining behavior; learning can transform backward-chaining heuristic search into more informed forward-chaining behavior; learning is incremental and interleaved with performance; learning is incremental and interleaved with performance; structural learning involves monotonic addition of symbolic elements to long-term memory; structural learning involves monotonic addition of symbolic elements to long-term memory; transfer to new tasks depends on the amount of structure shared with previously mastered tasks. transfer to new tasks depends on the amount of structure shared with previously mastered tasks. Claims about Learning Cognitive psychology has also developed ideas about learning: I CARUS incorporates these assumptions into its basic operation.

41. Learning from Problem Solutions operates whenever problem solving overcomes an impasse; operates whenever problem solving overcomes an impasse; incorporates only information available from the goal stack; incorporates only information available from the goal stack; generalizes beyond the specific objects concerned; generalizes beyond the specific objects concerned; depends on whether chaining involved skills or concepts; depends on whether chaining involved skills or concepts; supports cumulative learning and within-problem transfer. supports cumulative learning and within-problem transfer. I CARUS incorporates a mechanism for learning new skills that: This skill creation process is fully interleaved with means-ends analysis and execution. Learned skills carry out forward execution in the environment rather than backward chaining in the mind.

42. I CARUS Learns Skills from Problem Solving Executed plan Problem ? Skill Hierarchy Primitive Skills Reactive Execution impasse? Problem Solving yes no Skill Learning

43. I CARUS Constraints on Skill Learning What determines the hierarchical structure of skill memory? What determines the hierarchical structure of skill memory? The structure emerges the subproblems that arise during problem solving, which, because operator conditions and goals are single literals, form a semilattice. The structure emerges the subproblems that arise during problem solving, which, because operator conditions and goals are single literals, form a semilattice. What determines the heads of the learned clauses/methods? What determines the heads of the learned clauses/methods? The head of a learned clause is the goal literal that the planner achieved for the subproblem that produced it. The head of a learned clause is the goal literal that the planner achieved for the subproblem that produced it. What are the conditions on the learned clauses/methods? What are the conditions on the learned clauses/methods? If the subproblem involved skill chaining, they are the conditions of the first subskill clause. If the subproblem involved skill chaining, they are the conditions of the first subskill clause. If the subproblem involved concept chaining, they are the subconcepts that held at the subproblems outset. If the subproblem involved concept chaining, they are the subconcepts that held at the subproblems outset.

44. (ontable A T) (on B A) (on C B) (hand-empty) (clear C) (unst. C B) (unstack C B) (clear B) (putdown C T) (unst. B A) (unstack B A) (clear A) (holding C)(hand-empty) (holding B) A B CB A C 1 skill chaining Constructing Skills from a Trace

45. (ontable A T) (on B A) (on C B) (hand-empty) (clear C) (unst. C B) (unstack C B) (clear B) (putdown C T) (unst. B A) (unstack B A) (clear A) (holding C)(hand-empty) (holding B) A B CB A C 1 2 skill chaining Constructing Skills from a Trace

46. (ontable A T) (on B A) (on C B) (hand-empty) (clear C) (unst. C B) (unstack C B) (clear B) (putdown C T) (unst. B A) (unstack B A) (clear A) (holding C)(hand-empty) (holding B) A B CB A C 1 3 2 concept chaining Constructing Skills from a Trace

47. (ontable A T) (on B A) (on C B) (hand-empty) (clear C) (unst. C B) (unstack C B) (clear B) (putdown C T) (unst. B A) (unstack B A) (clear A) (holding C)(hand-empty) (holding B) A B CB A C 1 3 2 4 skill chaining Constructing Skills from a Trace

48. Learned Skills in the Blocks World Learned Skills in the Blocks World (clear (?C) :percepts((block ?D) (block ?C)) :start((unstackable ?D ?C)) :skills((unstack ?D ?C))) (clear (?B) :percepts ((block ?C) (block ?B)) :start((on ?C ?B) (hand-empty)) :skills((unstackable ?C ?B) (unstack ?C ?B))) (unstackable (?C ?B) :percepts((block ?B) (block ?C)) :start ((on ?C ?B) (hand-empty)) :skills((clear ?C) (hand-empty))) (hand-empty ( ) :percepts ((block ?D) (table ?T1)) :start ((putdownable ?D ?T1)) :skills ((putdown ?D ?T1))) Hierarchical skills are generalized traces of successful means-ends problem solving

49. Initial Results with I CARUS

50. Cognitive architectures come with a programming language that: includes a syntax linked to its representational assumptions includes a syntax linked to its representational assumptions inputs long-term knowledge and initial short-term elements inputs long-term knowledge and initial short-term elements provides an interpreter that runs the specified program provides an interpreter that runs the specified program incorporates tracing facilities to inspect system behavior incorporates tracing facilities to inspect system behavior Architectures as Programming Languages Such programming languages ease construction and debugging of knowledge-based systems. For this reason, cognitive architectures support far more efficient development of software for intelligent systems.

51. The programming language associated with I CARUS comes with: a syntax for concepts, skills, beliefs, and percepts a syntax for concepts, skills, beliefs, and percepts the ability to load and parse such programs the ability to load and parse such programs an interpreter for inference, execution, planning, and learning an interpreter for inference, execution, planning, and learning a trace package that displays system behavior over time a trace package that displays system behavior over time Programming in I CARUS We have used this language to develop adaptive intelligent agents in a variety of domains.

52. An I CARUS Agent for Urban Combat

53. Learning Skills for In-City Driving We have also trained I CARUS to drive in our in-city environment. We provide the system with tasks of increasing complexity. Learning transforms the problem-solving traces into hierarchical skills. The agent uses these skills to change lanes, turn, and park using only reactive control.

54. Skill Clauses Learning for In-City Driving Skill Clauses Learning for In-City Driving ((parked ?me ?g1152) :percepts((lane-line ?g1152) (self ?me)) :start( ) :subgoals((in-rightmost-lane ?me ?g1152) (stopped ?me)) ) ((in-rightmost-lane ?me ?g1152) :percepts((self ?me) (lane-line ?g1152)) :start ((last-lane ?g1152)) :subgoals((driving-well-in-segment ?me ?g1101 ?g1152)) ) ((driving-well-in-segment ?me ?g1101 ?g1152) :percepts((lane-line ?g1152) (segment ?g1101) (self ?me)) :start ((steering-wheel-straight ?me)) :subgoals((in-lane ?me ?g1152) (centered-in-lane ?me ?g1101 ?g1152) (aligned-with-lane-in-segment ?me ?g1101 ?g1152) (steering-wheel-straight ?me)) )

55. Learning Curves for In-City Driving

56. Cumulative Curves for Blocks World

58. Cumulative Curves for FreeCell

59. Related Research

60. Intellectual Precursors earlier research on integrated cognitive architectures earlier research on integrated cognitive architectures especially ACT, Soar, and Prodigy especially ACT, Soar, and Prodigy earlier frameworks for reactive control of agents earlier frameworks for reactive control of agents research on belief-desire-intention (BDI) architectures research on belief-desire-intention (BDI) architectures planning/execution with hierarchical transition networks planning/execution with hierarchical transition networks work on learning macro-operators and search-control rules work on learning macro-operators and search-control rules previous work on cumulative structure learning previous work on cumulative structure learning I CARUS design has been influenced by many previous efforts: However, the framework combines and extends ideas from its various predecessors in novel ways.

61. Some Other Cognitive Architectures ACTSoarP RODIGY EPICRCS APEX C LARION Dynamic Memory Society of Mind CAPS GIPS 3T

62. Similarities to Previous Architectures I CARUS has much in common with other cognitive architectures like Soar (Laird et al., 1987) and ACT-R (Anderson, 1993): These ideas all have their origin in theories of human memory, problem solving, and skill acquisition. 1.Short-term memories are distinct from long-term stores 2.Memories contain modular elements cast as symbolic structures 3.Long-term structures are accessed through pattern matching 4.Cognition occurs in retrieval/selection/action cycles 5.Learning is incremental and interleaved with performance

63. Distinctive Features of I CARUS However, I CARUS also makes assumptions that distinguish it from most other architectures: Some of these assumptions appear in Bonasso et al.s 3T, Freeds APEX, and Sun et al.s CLARION architectures. 1.Cognition is grounded in perception and action 2.Categories and skills are separate cognitive entities 3.Short-term elements are instances of long-term structures 4.Inference and execution are more basic than problem solving 5.Skill/concept hierarchies are learned in a cumulative manner These ideas have their roots in cognitive psychology, but they are also effective in building integrated intelligent agents.

64. Directions for Future Research forward chaining and mental simulation of skills; forward chaining and mental simulation of skills; learning expected utilities from skill execution histories; learning expected utilities from skill execution histories; learning new conceptual structures in addition to skills; learning new conceptual structures in addition to skills; probabilistic encoding and matching of Boolean concepts; probabilistic encoding and matching of Boolean concepts; flexible recognition of skills executed by other agents; flexible recognition of skills executed by other agents; extension of short-term memory to store episodic traces. extension of short-term memory to store episodic traces. Future work on I CARUS should introduce additional methods for: Taken together, these features should make I CARUS a more general and powerful cognitive architecture.

65. Contributions of I CARUS includes separate memories for concepts and skills; includes separate memories for concepts and skills; organizes both memories in a hierarchical fashion; organizes both memories in a hierarchical fashion; modulates reactive execution with goal seeking; modulates reactive execution with goal seeking; augments routine behavior with problem solving; and augments routine behavior with problem solving; and learns hierarchical skills in a cumulative manner. learns hierarchical skills in a cumulative manner. I CARUS is a cognitive architecture for physical agents that: These ideas have their roots in cognitive psychology, but they are also effective in building flexible intelligent agents. For more information about the I CARUS architecture, see: http://cll.stanford.edu/research/ongoing/icarus/ http://cll.stanford.edu/research/ongoing/icarus/ http://cll.stanford.edu/research/ongoing/icarus/

66. Transfer of Learned Knowledge

67. general learning in multiple domains Generality in Learning training items test items Humans exhibit general intelligence by their ability to learn in many domains.

68. general learning in multiple domains transfer of learning across domains Generality and Transfer in Learning training items test items training items test items Humans exhibit general intelligence by their ability to learn in many domains. Humans are also able to utilize knowledge learned in one domain in other domains.

69. experience performance A learner exhibits transfer of learning from task/domain A to task/domain B when, after it has trained on A, it shows improved behavior on B. learning curve for task A experience performance experience performance better intercept on task B better asymptote on task B faster learning rate on task B What is Transfer? experience w/training on A w/o training on A w/training on A w/o training on A w/training on A w/o training on Aperformance

70. What is Transfer? Transfer is a sequential phenomenon that occurs in settings which involve on-line learning. Transfer is a sequential phenomenon that occurs in settings which involve on-line learning. Thus, multi-task learning does not involve transfer. Thus, multi-task learning does not involve transfer. Transfer involves the reuse of knowledge structures. Transfer involves the reuse of knowledge structures. Thus, it requires more than purely statistical learning. Thus, it requires more than purely statistical learning. Transfer can lead to improved behavior (positive transfer). Transfer can lead to improved behavior (positive transfer). But it can also produce worse behavior (negative transfer). But it can also produce worse behavior (negative transfer). Transfer influences learning but is not a form of learning. Transfer influences learning but is not a form of learning. Thus, transfer learning is an oxymoron, much like the phrase learning performance. Thus, transfer learning is an oxymoron, much like the phrase learning performance.

71. Roots of Transfer in Psychology The notion of transfer comes from psychology, where it has been studied for over a hundred years: benefits of Latin (Thorndike & Woodworth, 1901) benefits of Latin (Thorndike & Woodworth, 1901) puzzle solving (Luchins & Luchins, 1970) puzzle solving (Luchins & Luchins, 1970) operating devices (Kieras & Bovair, 1986) operating devices (Kieras & Bovair, 1986) using text editors (Singley & Anderson, 1988) using text editors (Singley & Anderson, 1988) analogical reasoning (Gick & Holyoak, 1983) analogical reasoning (Gick & Holyoak, 1983) Some recent studies have included computational models that predict the transfer observed under different conditions.

72. Inference tasks that require multi-step reasoning to obtain an answer, such as solving physics word problems and aptitude/achievement tests. Classification tasks that involve assigning items to categories, such as recognizing types of vehicles or detecting spam. These are not very interesting. Domain Classes that Exhibit Transfer Procedural tasks that involve execution of routinized skills, both cognitive (e.g., multi-column arithmetic) and sensori-motor (e.g., flying an aircraft). Problem-solving tasks that benefit from strategic choices and heuristic search, such as complex strategy games. 654 - 321 456 - 237 821 - 549 940 - 738 601 - 459 400 - 321 From: tsenator@darpa.mil To: langley@csli.stanford.edu Subject: site visit next week Date: Nov 14, 2004 Pat – I am looking forward to hearing about your progress over the past year during my site visit next week. - Ted From: noname@somewhere.com To: langley@csli.stanford.edu Subject: special offer!!! Date: Nov 14, 2004 One week only! Buy v*i*a*g*r*a at half the price available in stores. Go now to http://special.deals.com A block sits on an inclined plane but is connected to a weight by a string through a pulley. If the angle of the plane is 30 degrees and... Which ladder is safer to climb on? Which jump should red make? What should the blue team do? What are the problem answers? What path should the plane take? Which is an emergency vehicle? Which email is spam?

73. The degree of transfer depends on the structure shared with the training tasks. Transfer requires the ability to compose these knowledge elements dynamically. Transfer requires that knowledge be represented in a modular fashion. Claims About Transfer Transfer across domains requires abstract relations among representations.

74. Dimensions of Knowledge Transfer Difference in Content Difference in Representation 0 0 Memorization Different Representations (e.g., most cross-domain transfer) Similar Representations (e.g., within- domain transfer) Knowledge Reuse First- Principles Reasoning Isomorphism We have already solved these problems. We know the solution to a similar problem with a different representation, possibly from another domain. We have not solved this before, but we know other pertinent information about this domain that uses the same representation. We have not solved similar problems, and are not familiar with this domain and problem representation. Knowledge transfer complexity is determined primarily by differences in the knowledge content and representation between the source and target problems. Problem Solver

75. Memorization target items source items E.g., solving the same geometry problems on a homework assignment as were presented in class. This is not very interesting. Improvement in which the transfer tasks are the same as those encountered during training.

76. Within-Domain Lateral Transfer target items source items E.g., solving new physics problems that involve some of the same principles but that also introduce new ones. Improvement on related tasks of similar difficulty within the same domain that share goals, initial state, or other structure.

77. Within-Domain Vertical Transfer target items source items E.g., solving new physics problems that involve the same principles but that also require more reasoning steps. Improvement on related tasks of greater difficulty within the same domain that build on results from training items.

78. Cross-Domain Lateral Transfer target items source items E.g., solving problems about electric circuits that involve some of the same principles as problems in fluid flow but that also introduce new ones. Improvement on related tasks of similar difficulty in a different domain that shares either higher-level or lower-level structures.

79. Cross-Domain Vertical Transfer target items source items E.g., solving physics problems that require mastery of geometry and algebra or applying abstract thermodynamic principles to a new domain. Improvement on related tasks of greater difficulty in a different domain that share higher-level or lower-level structures.

80. 11 12 3 45 6 12 78 9 10 Methods for cumulative learning of hierarchical skills and concepts define new cognitive structures in terms of structures learned on earlier tasks. 13 11 12 3 4 6 12 5 8 9 10 12 3 45 6 78 9 This approach is well suited to support vertical transfer to new tasks of ever increasing complexity. Learning can operate on problem-solving traces, observations of another agents behavior, and even on direct instructions. Approaches to Transfer: Cumulative Learning

81. Methods for analogical reasoning store cognitive structures that encode relations in training problems. Additional relations are then inferred based on elements in the retrieved problem. Analogical reasoning can operate over any stored relational structure, but must map training elements to transfer elements, which can benefit from knowledge. This approach is well suited for lateral transfer to tasks of similar difficulty. Approaches to Transfer: Analogical Reasoning Upon encountering a new problem, they retrieve stored experiences with similar relational structure.

82. Approaches to Transfer: Mapping Representations Mapping Process Source domain: Electricity Target domain: Fluid Flow Transfer of learned knowledge across domains may require mapping between their representations of shared content. Electrical Resistance R Voltage Drop V1V1V1V1 I V2V2V2V2 I = V 1 - V 2 R Knowledge: Ohms law Pressure Drop F = P 1 - P 2 R Knowledge: Poiseuilles law P1P1P1P1 P2P2P2P2 Resistance to Flow R F Q: If P 1 =3, P 2 =2, and R=2, then what force F is being applied, assuming we only know Ohms law for electric currents?

83. Experimental Studies of Transfer Compare results from transfer and control conditions Transfer condition Control condition Train on items from source domain Test and train on target domain items Present no items from source domain Test and train on target domain items

84. Dependent Variables in Transfer Studies Dependent variables for transfer experiments should include: Initial performance on the transfer tasks Initial performance on the transfer tasks Asymptotic performance on the transfer tasks Asymptotic performance on the transfer tasks Rate of improvement on the transfer tasks Rate of improvement on the transfer tasks These require collecting learning curves over a series of tasks. Such second-order variables build on basic metrics such as: Accuracy of response or solutions to tasks Accuracy of response or solutions to tasks Speed or efficiency of solutions to tasks Speed or efficiency of solutions to tasks Quality or utility of solutions to tasks Quality or utility of solutions to tasks Different basic measures are appropriate for different domains.

85. Transfer of Knowledge in I CARUS

86. Transfer in I CARUS What forms of knowledge does I CARUS transfer? What forms of knowledge does I CARUS transfer? Hierarchical/relational skill and concept clauses Hierarchical/relational skill and concept clauses Where does the transferred knowledge originate? Where does the transferred knowledge originate? It comes from experience on source problems and background knowledge It comes from experience on source problems and background knowledge How does I CARUS know what to transfer? How does I CARUS know what to transfer? Skills are indexed by goals they achieve, with preference for more recently learned structures Skills are indexed by goals they achieve, with preference for more recently learned structures

87. A Transfer Scenario from Urban Combat Target Problem Source Problem Here the first part of the source route transfers to the target, but the second part must be learned to solve the new task.

88. Source Target Shared structures Structures Transferred in Scenario

89. Primitive Concepts for Urban Combat ((stopped ?self) :percepts((self ?self xvel ?xvel yvel ?yvel)) :percepts((self ?self xvel ?xvel yvel ?yvel)) :tests ((< (+ (* ?xvel ?xvel) (* ?yvel ?yvel)) 1))) :tests ((< (+ (* ?xvel ?xvel) (* ?yvel ?yvel)) 1))) ((in-region ?self ?region) ((in-region ?self ?region) :percepts((self ?self region ?region))) :percepts((self ?self region ?region))) ((connected-region ?target ?gateway) ((connected-region ?target ?gateway) :percepts ((gateway ?gateway region ?target))) :percepts ((gateway ?gateway region ?target))) ((blocked-gateway ?gateway) ((blocked-gateway ?gateway) :percepts((gateway ?gateway visible1 ?v1 visible2 ?v2)) :percepts((gateway ?gateway visible1 ?v1 visible2 ?v2)) :tests((and (equal ?v1 'B) (equal ?v2 'B))) ) :tests((and (equal ?v1 'B) (equal ?v2 'B))) ) ((first-side-blocked-gateway ?gateway) ((first-side-blocked-gateway ?gateway) :percepts((gateway ?gateway type ?type visible1 ?v1 visible2 ?v2)) :percepts((gateway ?gateway type ?type visible1 ?v1 visible2 ?v2)) :tests ((equal ?type 'WALK) (equal ?v1 'B) (equal ?v2 'C))) :tests ((equal ?type 'WALK) (equal ?v1 'B) (equal ?v2 'C))) ((flag-captured ?self flag1) ((flag-captured ?self flag1) :percepts ((self ?self holding ?flag1) (entity ?flag1) :percepts ((self ?self holding ?flag1) (entity ?flag1) :tests ((not (equal ?flag NIL)))) :tests ((not (equal ?flag NIL))))

90. ((not-stopped ?self) :percepts ((self ?self)) :percepts ((self ?self)) :relations ((not (stopped ?self)))) :relations ((not (stopped ?self)))) ((clear-gateway ?gateway) ((clear-gateway ?gateway) :percepts ((gateway ?gateway type ?type visible1 ?v1 visible2 ?v2)) :percepts ((gateway ?gateway type ?type visible1 ?v1 visible2 ?v2)) :relations ((not-stopped ?self)) :relations ((not-stopped ?self)) :tests ((equal ?type 'WALK) (equal ?v1 'C) (equal ?v2 'C))) :tests ((equal ?type 'WALK) (equal ?v1 'C) (equal ?v2 'C))) ((stopped-in-region ?self ?region) ((stopped-in-region ?self ?region) :percepts ((self ?self)) :percepts ((self ?self)) :relations ((in-region ?self ?region)(stopped ?self))) :relations ((in-region ?self ?region)(stopped ?self))) ((crossable-region ?target) ((crossable-region ?target) :percepts ((self ?self) (region ?target)) :percepts ((self ?self) (region ?target)) :relations ((connected-region ?target ?gateway) (clear-gateway ?gateway))) :relations ((connected-region ?target ?gateway) (clear-gateway ?gateway))) ((in-region-able ?self ?current ?region) ((in-region-able ?self ?current ?region) :percepts ((self ?self) (region ?current) (region ?region)) :percepts ((self ?self) (region ?current) (region ?region)) :relations ((crossable-region ?region) (in-region ?self ?current))) :relations ((crossable-region ?region) (in-region ?self ?current))) Nonprimitive Urban Combat Concepts

91. ((in-region ?self ?region) :percepts((self ?self) (region ?current) (region ?region) :percepts((self ?self) (region ?current) (region ?region) (gateway ?gateway region ?region dist1 ?dist1 angle1 ?angle1 (gateway ?gateway region ?region dist1 ?dist1 angle1 ?angle1 dist2 ?dist2 angle2 ?angle2)) dist2 ?dist2 angle2 ?angle2)) :start((in-region-able ?self ?current ?region)) :start((in-region-able ?self ?current ?region)) :actions((*move-toward (max ?dist1 ?dist2) (mid-direction ?angle1 ?angle2)))) :actions((*move-toward (max ?dist1 ?dist2) (mid-direction ?angle1 ?angle2)))) ((clear-gateway ?gateway) ((clear-gateway ?gateway) :percepts ((self ?self) (gateway ?gateway type WALK)) :percepts ((self ?self) (gateway ?gateway type WALK)) :start ((stopped ?self)) :start ((stopped ?self)) :actions((*move-toward 50 0))) :actions((*move-toward 50 0))) ((clear-gateway ?gateway) ((clear-gateway ?gateway) :percepts ((gateway ?gateway region ?region dist1 ?dist1 angle1 ?angle1 :percepts ((gateway ?gateway region ?region dist1 ?dist1 angle1 ?angle1 visible1 ?v1dist2 ?dist2 angle2 ?angle2 visible2 ?v2)) visible1 ?v1dist2 ?dist2 angle2 ?angle2 visible2 ?v2)) :start ((first-side-blocked-gateway ?gateway)) :start ((first-side-blocked-gateway ?gateway)) :actions ((*move-toward ?dist2 ?angle2))) :actions ((*move-toward ?dist2 ?angle2))) ((flag-captured ?self ?flag) ((flag-captured ?self ?flag) :percepts ((self ?self) (entity ?flag dist ?dist angle ?angle)) :percepts ((self ?self) (entity ?flag dist ?dist angle ?angle)) :start ((in-region ?self region107)) :start ((in-region ?self region107)) :actions ((*move-toward ?dist ?angle))) :actions ((*move-toward ?dist ?angle))) Primitive Skills for Urban Combat

92. ((crossable-region ?target) :percepts ((region ?target) (gateway ?gateway)) :percepts ((region ?target) (gateway ?gateway)) :start ((connected-region ?target ?gateway)) :start ((connected-region ?target ?gateway)) :subgoals ((clear-gateway ?gateway))) :subgoals ((clear-gateway ?gateway))) ((in-region-able ?self ?current ?region) ((in-region-able ?self ?current ?region) :percepts ((self ?self) (region ?current) (region ?region)) :percepts ((self ?self) (region ?current) (region ?region)) :subgoals ((in-region ?self ?current) (crossable-region ?region))) :subgoals ((in-region ?self ?current) (crossable-region ?region))) ((stopped-in-region ?self ?region) ((stopped-in-region ?self ?region) :percepts ((self ?self) (region ?region)) :percepts ((self ?self) (region ?region)) :subgoals ((in-region ?self ?region) (stopped ?self))) :subgoals ((in-region ?self ?region) (stopped ?self))) ((stopped-in-region ?self ?current) ((stopped-in-region ?self ?current) :percepts ((self ?self)) :percepts ((self ?self)) :start((in-region ?self ?current)) :start((in-region ?self ?current)) :subgoals ((stopped ?self))) :subgoals ((stopped ?self))) ((flag-captured ?self ?flag) ((flag-captured ?self ?flag) :percepts ((self ?self)) :percepts ((self ?self)) :subgoals ((in-region ?self region107) (flag-captured ?self ?flag))) :subgoals ((in-region ?self region107) (flag-captured ?self ?flag))) Nonprimitive Skills for Urban Combat

93. ((in-region ?self region105) :percepts ((self ?self)) :percepts ((self ?self)) :subgoals ((in-region-able ?self region115 region105) (in-region ?self region105))) :subgoals ((in-region-able ?self region115 region105) (in-region ?self region105))) ((in-region ?self region114) ((in-region ?self region114) :percepts ((self ?self)) :percepts ((self ?self)) :subgoals ((in-region-able ?self region105 region114) (in-region ?self region114))) :subgoals ((in-region-able ?self region105 region114) (in-region ?self region114))) ((in-region ?self region110) ((in-region ?self region110) :percepts ((self ?self)) :percepts ((self ?self)) :subgoals ((in-region-able ?self region114 region110) (in-region ?self region110))) :subgoals ((in-region-able ?self region114 region110) (in-region ?self region110))) ((in-region ?self region116) ((in-region ?self region116) :percepts ((self ?self)) :percepts ((self ?self)) :subgoals ((in-region-able ?self region110 region116) (in-region ?self region116))) :subgoals ((in-region-able ?self region110 region116) (in-region ?self region116))) ((in-region ?self region107) ((in-region ?self region107) :percepts ((self ?self)) :percepts ((self ?self)) :subgoals ((in-region-able ?self region116 region107) (in-region ?self region107))) :subgoals ((in-region-able ?self region116 region107) (in-region ?self region107))) Route Knowledge for Urban Combat

94. Experimental Protocol Source Problems in Random Order Target Problems in Random Order Transfer Condition Non-Transfer Condition Agent Transfer ScoresHuman Transfer Scores Statistical Analysis Human Subjects Agents % Agent/Human Transfer Ratios HumanAgent Learning Curves

95. TL LevelMetric(s)Goal(s) 1 - 8Time to completion (300s max) plus penalty for health effects: +2s for minor hazards (glass) +30s for major hazards (landmines) plus penalty for resource use: +2s for each ammunition round Find IED Minimize score Domain Performance Metrics and Goals Each minor hazard costs the time required to traverse two regions at a walking pace. Penalty chosen to equalize expected difficulty of navigation and problem solving tasks. Major hazards and ammunition costs chosen to enhance realism.

96. TL Level 1 Task: Parameterization Solution preserved, task parametrically changed SourceTarget Source Problem: Find IED Target Problem: Find IED, location in goal region changed Transferred knowledge: Route from source to goal region Solution for surmounting obstacles, if any Performance Goal : Time to completion Background Knowledge : Primitive actions, exploration skills, relational concepts (e.g., object close, path clear) Start IED IED moved in region

97. Transfer Level 1: ISLE Raw Curves Urban Combat

98. TL Level 2 Task: Extrapolation Solution preserved up to obstacle, then changed SourceTarget Source Problem: Find IED given obstacle Target Problem: Find IED, obstacle extended Transferred knowledge: Route from source to goal region Performance Goal : Time to completion Background Knowledge : Primitive actions, exploration skills, relational concepts (e.g., object close, path clear) Gap in wall removed Start IED

99. Transfer Level 2: ISLE Raw Curves Urban Combat

100. TL Level 3 Task: Restructuring Discover, then reuse action sequences in new order SourceTarget Source Problem: Find IED given obstacles Target Problem: Find IED, obstacles rearranged Transferred knowledge: Route from source to goal region Solution for surmounting obstacles, if any Performance Goal : Time to completion Background Knowledge : Primitive actions, exploration skills, relational concepts (e.g., object close, path clear) Start IED Breakable Boxes Pit Unclimbable wall Role of pit and boxes reversed Box

101. Transfer Level 3: ISLE Raw Curves Urban Combat

102. TL Level 4 Task: Extending Discover and repeatedly reuse solutions SourceTarget Source Problem: Find IED given obstacles Target Problem: Find IED, same obstacles multiplied Transferred knowledge: Route from source to goal region Solution for surmounting obstacles, if any Performance Goal : Time to completion Background Knowledge : Primitive actions, exploration skills, relational concepts (e.g., object close, path clear) Start IED Jumpable wall Unclimbable barrier Multiple walls Multiple barriers

103. Transfer Level 4: ISLE Raw Curves Urban Combat

104. ISLE Agent Transfer Scores & P values for UCT TL Metrics Level 1 Level 2 Level 3 Level 4 ScoreP-valueScoreP-ValueScoreP-valueScoreP-value Jump start 84.50000.007818.50000.3126281.1200.1198256.5780.1148 ARR (narrow) 0.00000.27700.16121.42850.013400.31 ARR (wide) 0.99130.01040.868890.18461.0730.00180.7250.1064 Ratio0.512910.78480.99920.8020.89880.83160.99 Transfer ratio 7.37688.0E-043.66520.0031.870.06282.79490.01 Truncated transfer ratio 7.27170.01064.00540.02821.870.19462.79490.08 Transfer difference 10565.604740.180.001211044.00.19467329.60.01 Scaled transfer difference -111.5691-30.5670.9954-34.4260.866-22.8220.988 Asymptotic advantage 94.2000039.50000.05511.00.4550.8950.02 TL1: 50 trials/condition; TL2: 37; TL3: 35; TL4: 45 Data preprocessed by subtracting from 0

105. Human Transfer Scores & P values for UCT TL Metrics Level 1 Level 2 Level 3 Level 4 ScoreP-valueScoreP-valueScoreP-valueScoreP-value Jump start -0.64710.5448-42.3330.8126-14.94740.6896104.7500.0014 ARR (narrow) 0.00000.0364-1E+310.58180.00000.22040.00000.2720 ARR (wide) -1E+310.4028-1E+310.2750-1E+310.33340.89680.0656 Ratio0.36660.99800.73010.88820.63870.88500.55020.9354 Transfer ratio 6.45230.00282.22370.08982.42740.16782.26990.1670 Truncated transfer ratio 6.45230.03742.32070.09722.51230.21762.26990.1748 Transfer difference 163.0880.0044104.6670.1240193.9740.1166264.9690.0748 Scaled transfer difference -10.11860.9966-2.11880.8598-3.74920.9004-6.88230.9788 Asymptotic advantage 8.94120.105434.40000.047840.42110.0942108.8750.0014 TL1: 17 trials/condition; TL2 15; TL3 19; TL4 16 Data preprocessed by subtracting from 0

106. We have also tested ICARUS in domains like FreeCell solitaire. Other Transfer Results with I CARUS Experiments suggest that learned knowledge transfers well here.

107. Key Ideas about Transfer in I CARUS The most important transfer concerns goal-directed behavior that involves sequential actions aimed toward an objective. The most important transfer concerns goal-directed behavior that involves sequential actions aimed toward an objective. Transfer mainly involves the reuse of knowledge structures. Transfer mainly involves the reuse of knowledge structures. Organizing structures in a hierarchy aids reuse and transfer. Organizing structures in a hierarchy aids reuse and transfer. Indexing skills by goals they achieve determines relevance. Indexing skills by goals they achieve determines relevance. One can learn hierarchical, relational, goal-directed skills by analyzing traces of expert behavior and problem solving. One can learn hierarchical, relational, goal-directed skills by analyzing traces of expert behavior and problem solving. Skill learning can build upon structures acquired earlier. Skill learning can build upon structures acquired earlier. Successful transfer benefits from knowledge-based inference to recognize equivalent situations. Successful transfer benefits from knowledge-based inference to recognize equivalent situations.

108. Open Research Problems Goal transfer across tasks with distinct but related objectives Goal transfer across tasks with distinct but related objectives Negative transfer minimizing use of inappropriate knowledge Negative transfer minimizing use of inappropriate knowledge Context handling avoiding catastrophic interference Context handling avoiding catastrophic interference Representation mapping Representation mapping Lateral Deep analogy that involves partial isomorphisms Lateral Deep analogy that involves partial isomorphisms Vertical Bootstrapped learning that builds on lower levels Vertical Bootstrapped learning that builds on lower levels There remain many research issues that we must still address: These challenges should keep our field occupied for some time.

109. Closing Remarks involves the sequential reuse of knowledge structures involves the sequential reuse of knowledge structures takes many forms depending on source/target relationships takes many forms depending on source/target relationships has been repeatedly examined within psychology/education has been repeatedly examined within psychology/education has received little attention in AI and machine learning has received little attention in AI and machine learning requires a fairly sophisticated experimental method requires a fairly sophisticated experimental method Transfer of learned knowledge is an important capability that: Transfer originated in psychology, and it is best studied in the context of cognitive architectures, which have similar roots.

110. End of Presentation