1. Pat Langley Institute for the Study of Learning and Expertise Palo Alto, CA http://www.isle.org Cumulative Learning of Relational and Hierarchical Skills from Problem Solving This research was funded by Grant HR0011-04-1-0008 from the DARPA Information Processing Technology Office, which may not agree with the points made in this talk.

2. Research Objectives We are designing and implementing new learning methods that: operate over relational, hierarchical knowledge structures operate over relational, hierarchical knowledge structures support reasoning, reactive control, and problem solving support reasoning, reactive control, and problem solving are embedded within a broader architectural framework are embedded within a broader architectural framework utilize existing knowledge to increase learning rates utilize existing knowledge to increase learning rates acquire this knowledge in an incremental, cumulative manner acquire this knowledge in an incremental, cumulative manner are applicable to a variety of challenging domains are applicable to a variety of challenging domains We hope to develop learning mechanisms that support horizontal and vertical transfer both within and across domains.

3. The I CARUS Architecture * Long-TermConceptualMemory Long-Term Skill Memory Short-TermConceptualMemory Goal/SkillStack Categorization and Inference SkillExecution Perception Environment PerceptualBuffer * without learning Means-EndsAnalysis MotorBuffer SkillRetrieval

4. Organization of Long-Term Memory concepts skills Each concept is defined in terms of other concepts and/or percepts. Each skill is defined in terms of other skills, concepts, and percepts. I CARUS organizes both concepts and skills in a hierarchical manner.

5. Concepts from In-City Driving Domain (in-segment (?self ?sg) :percepts((self ?self segment ?sg) (segment ?sg))) (aligned-with-lane (?self ?lane) :percepts((self ?self) (lane-line ?lane angle ?angle)) :positives((in-lane ?self ?lane)) :tests((> ?angle 0.05) ( ?angle 0.05) (< ?angle 0.05)) ) (on-street (?self ?packet) :percepts((self ?self) (packet ?packet street ?street) (segment ?sg street ?street)) :positives((not-delivered ?packet) (current-segment ?self ?sg))) (increasing-direction (?self) :percepts((self ?self)) :positives((increasing ?b1 ?b2)) :negatives((decreasing ?b3 ?b4)) )

6. Organization of Long-Term Memory conceptsskills For example, the skill highlighted here refers directly to the highlighted concepts. I CARUS interleaves its long-term memories for concepts and skills.

7. Skills from In-City Driving Domain (turn-around-on-street (?self ?packet) :percepts((self ?self segment ?segment direction ?dir) (building ?landmark)) :start ((on-street-wrong-direction ?packet)) :effects((on-street-right-direction ?packet)) (building ?landmark)) :start ((on-street-wrong-direction ?packet)) :effects((on-street-right-direction ?packet)) :ordered((get-in-U-turn-lane ?self) (prepare-for-U-turn ?self) :ordered((get-in-U-turn-lane ?self) (prepare-for-U-turn ?self) (steer-for-U-turn ?self ?landmark)) ) (steer-for-U-turn ?self ?landmark)) ) (get-aligned-in-segment (?self ?sg) :percepts((lane-line ?lane angle ?angle)) :requires((in-lane ?self ?lane)) :effects((aligned-with-lane ?self ?lane)) :actions(( steer ( times ?angle 2))) ) (steer-for-right-turn (?self ?int ?endsg) :percepts((self ?self speed ?speed) (intersection ?int cross ?cross) (segment ?endsg street ?cross angle ?angle)) :start((ready-for-right-turn ?self ?int)) :effects((in-segment ?self ?endsg)) :actions(( times steer 2)) ) :actions(( times steer 2)) )

8. Basic I CARUS Processes concepts skills Concepts are matched bottom up, starting from percepts. Skill paths are matched top down, starting from intentions. I CARUS matches patterns to recognize concepts and select skills.

9. 1 2 8 5 3 6 4 7 9 1011 A Trace of Means-Ends Problem Solving The resulting traces provide the material for learning new relational skills and concepts in terms of simpler components. An impasse causes I CARUS to invoke a means-ends problem solver.

10. Learning Skills from Means-Ends Traces 1 2 8 5 3 6 4 7 A 9 1011 concept chaining I CARUS learns skills for ordering subgoals from concept chaining.

11. Learning Skills from Means-Ends Traces 1 2 8 5 3 6 4 7 A B 9 1011 skill chaining I CARUS learns skills for ordering subskills from skill chaining.

12. Learning Skills from Means-Ends Traces 1 2 8 5 3 6 4 7 A B 9 1011 C concept chaining Each level of skill learning builds upon results from prior levels.

13. Learning Skills from Means-Ends Traces 1 2 8 5 3 6 4 7 A B D 9 1011 C skill chaining This leads I CARUS to extend its skill hierarchy in a cumulative way.

14. Learning Skills from Means-Ends Traces 1 2 8 5 3 6 4 7 A B D E 9 1011 C concept chaining This in turn supports transfer both within and across problems.

15. Transfer Results in FreeCell FreeCell is a complex solitaire game in which all cards are visible. We let I CARUS practice on versions with a small set of cards, then examined its transfer to problems with more cards.

16. Transfer Results in FreeCell Experiments revealed substantial transfer to the harder problems. This held both for the percentage of problems solved and for the effort required on successful attempts.

17. Directions for Future Research vertical transfer to domains that utilize others as components; vertical transfer to domains that utilize others as components; horizontal transfer to domains to share knowledge elements; horizontal transfer to domains to share knowledge elements; horizontal transfer to tasks that require representation mapping. horizontal transfer to tasks that require representation mapping. Our initial results suggest I CARUS can transfer knowledge learned on simple problems to complex ones from the same domain. In future work, we intend to examine the additional issues of: The final problem is a key challenge in developing robust methods for reusing learned knowledge. We hope to evaluate our ideas on both action-oriented domains like strategy games and inferential tasks like physics problems.

18. The General Game-Playing Testbed supports a wide variety of wide variety of N-person games; supports a wide variety of wide variety of N-person games; describes each game setting in a standard logical formalism; describes each game setting in a standard logical formalism; specifies the rules of each game in a related formalism; specifies the rules of each game in a related formalism; manages matches between players and records activities; manages matches between players and records activities; provides sample games for debugging candidate systems. provides sample games for debugging candidate systems. Genesereth and Love (2005) have developed a framework that: They have designed this framework to encourage research on general approaches to intelligent behavior. However, it also provides an excellent testbed for evaluating the ability of learning systems to transfer within and across domains. See http://games.stanford.edu for more details and examples.