1 USC INFORMATION SCIENCES INSTITUTE Gil & Kim Interactive Knowledge Acquisition Tools: A Tutoring Perspective Yolanda Gil Jihie Kim USC/Information Sciences Institute August 9, 2002

2 USC INFORMATION SCIENCES INSTITUTE Gil & Kim Motivation: Investigate Synergies between Instructional Systems and Acquisition Tools Instructional System Acquisition Tool teaches ? ? Good Tutoring Principles Good Learning Principles SOFTWAREUSER

3 USC INFORMATION SCIENCES INSTITUTE Gil & Kim Our Previous Work in Knowledge Acquisition: The EXPECT Project at USC/ISI: EXPECT architecture for knowledge-based systems exploits highly declarative representations  [Swartout & Gil, KAW-95], [Gil & Melz, AAAI-96] [Blythe et al, IUI- 01]  Research focus: interactive knowledge acquisition (KA) tools that help end users to develop knowledge bases  Deriving models of knowledge interdependencies to detect knowledge gaps and errors [Kim & Gil, AAAI-99] [Kim & Gil, IUI- 2000] [Kim & Gil, AAAI-2000]  KA dialogue scripts to guide users by following up on effects of complex changes [Gil & Tallis, AAAI-97] [Tallis & Gil, AAAI-99] [Tallis, IJHCS-2001]  Exploiting background theories to understand how new knowledge fits [Blythe, IJCAI-2001] [Blythe, AAAI-02]

4 USC INFORMATION SCIENCES INSTITUTE Gil & Kim EXPECT: A User-Centered Framework for Developing KBSs Method instantiator Method instantiator Knowledge Base Domain ontologies and factual knowledge Problem solving methods Domain dependent KBS compiler KBS compiler Knowledge-Based System Interdependency Model (IM) EXPECT Ontologies and Method libraries KA tools EMeD Plans (PLANET) Evaluations and Critiques Evaluation PSMs Resources (OZONE) Domain ontologies CYC/Sensus Upper NL Editor Instrumentation Dialogue plans (KA Scripts) PSMTool

5 USC INFORMATION SCIENCES INSTITUTE Gil & Kim Brief Overview of Representative KA Tools (I) CHIMAERA [McGuinness et al 2000]  Acquisition of concepts, relations, instances  Diagnoses faulty definitions EXPECT [Blythe et al 2001]  Acquisition of problem solving knowledge  Exploits dialogue scripts, interdependency models, bg k INSTRUCTO-SOAR [Huffman & Laird 1995]  Acquisition of task models in Soar  Situated NL instruction is mapped to PSCM [Newell et al. 1991] KSSn [Gaines & Shaw 1993]  Acquisition of concepts, rules, data  Based on personal construct psychology [Kelly 1955] PROTOS [Bareiss et al 1990]  Acquisition and classification of new cases  Learning indexes to categories

6 USC INFORMATION SCIENCES INSTITUTE Gil & Kim Brief Overview of Representative KA Tools (II) SALT [Marcus & McDermott 1989]  Acquisition of constraints and fixes for configuration design  Exploits Problem Solving Method/ Task (Role-limiting approach) SEEK2 [Ginsberg et al. 1985]  Acquisition of rules  Uses verification and validation techniques SHAKEN [Clark et al. 2001]  Acquisition of process models  User interaction based on concept maps [Novak 1977] TAQL [Yost 1993]  Acquisition of SOAR rules  Editor for high level language for PSCM [Newell et al. 1991] TEIREISIAS [Davis 1979]  Acquires and classifies new cases  Learning indexes to categories

7 USC INFORMATION SCIENCES INSTITUTE Gil & Kim Open Challenges in KA Users remain largely responsible for the acquisition process  Decide where, what, when, how, why to enter knowledge  System checks errors, may have some short-term acquisition goals Ideally, KA tools should have student-like skills:  Formulate and pursue learning goals  Keep track of lessons and progress  Assess how much they are learning and how useful k is  If teacher is not so great, still capable of learning

8 USC INFORMATION SCIENCES INSTITUTE Gil & Kim Instructional Systems and Acquisition Tools: What Are the Synergies? Instructional System Acquisition Tool teaches ? ? Good Tutoring Principles Good Learning Principles SOFTWAREUSER Supplement Student’s limitations Supplement Teacher’s limitations

9 USC INFORMATION SCIENCES INSTITUTE Gil & Kim Teaching/Learning principleTutoring literature Start by introducing lesson topics and goals Atlas-Andes, Meno-Tutor, Human tutorial dialog Use topics of the lesson as a guide BE&E, UMFE Subsumption to existing cognitive structure Human learning, WHY, Atlas-Andes Immediate FeedbackSOPHIE, Auto-Tutor, Lisp tutor, Human tutorial dialog, human learning Generate educated guessesHuman tutorial dialog, QUADRATIC, PACT Keep on trackGUIDON, SHOLAR, TRAIN-Tutor Indicate lack of understandingHuman tutorial dialog, WHY Tutoring and Learning Principles Relevant to KA [Kim & Gil, ITS 02] (I)

10 USC INFORMATION SCIENCES INSTITUTE Gil & Kim Teaching/Learning principleTutoring literature Detect and fix “buggy” knowledgeSCHOLAR, Meno-Tutor, WHY, Buggy, CIRCSIM Learn deep modelPACT, Atlas-Andes Learn domain languageAtlas-Andes, Meno-Tutor Keep track of correct answersAtlas-Andes Prioritize learning tasksWHY Limit the nesting of the lesson to a handful Atlas Summarize what was learnedEXCHECK, TRAIN-Tutor, Meno- Tutor Provide overall assessment of learning knowledge WEST, Human tutorial dialog Tutoring and Learning Principles Relevant to KA [Kim & Gil, ITS 02] (II)

11 USC INFORMATION SCIENCES INSTITUTE Gil & Kim ASSIMILATE INSTRUCTION TRIGGER GOALS PROPOSE STRATEGIES PRIORITIZE GOALS & STRATEGIES PRESENTATION DESIGN USER INTERFACE KNOWLEDGE ACQUISITION BACKEND Five Main Functions of KA Tools Knowledge Base

12 USC INFORMATION SCIENCES INSTITUTE Gil & Kim ASSIMILATE INSTRUCTION TRIGGER GOALS PROPOSE STRATEGIES PRIORITIZE GOALS & STRATEGIES PRESENTATION DESIGN USER INTERFACE KNOWLEDGE ACQUISITION BACKEND Guidance Exploited by KA Tools Guidance from Knowledge Base Problem Solving & Task Knowledge Domain Knowledge General Background Knowledge Example Cases Guidance from Meta Knowledge Knowledge Repres. Model Diagnosis & Debugging Principles Tutoring & Learning Principles

13 USC INFORMATION SCIENCES INSTITUTE Gil & Kim Acqu. goals Acqu. strats Guess Generators Interaction Guidelines Operational Principles Priority Schemes General Tutoring & Learning Principles Knowledge Editor Dialogue - Goals & Strats - State - Suggestions - History USER INTERFACE KNOWLEDGE ACQUISITION BACKEND Tutoring and Learning Principles in KA Tools: Basic Conceptual Framework Knowledge Base ASSIMILATE INSTRUCTION TRIGGER GOALS PROPOSE STRATEGIES PRIORITIZE GOALS & STRATEGIES PRESENTATION DESIGN

14 USC INFORMATION SCIENCES INSTITUTE Gil & Kim Tutoring and Learning Principles Implicit in KA tools KSSnAssess learned knowledge Summarize what is learned EXPECTPrioritize learning tasks SEEK2Keep track of answers Learn domain language Learn deep models EXPECT,CHIMERATAQLDetect and fix “buggy” K INSTRUCTO- SOAR Indicate lack of understanding Keep on track EXPECTTEIREISIASGenerate educated guesses EXPECTTEIREISIASINSTRUCTO-SOARPROTOSImmediate feedback PROTOS, SALT TEIREISIAS PROTOSSubsumption to existing cog. structure SALTEXPECTSEEK2SALTUse topics of the lesson as a guide EXPECT, SEEK2Introduce topics & goals Design Presentation Prioritize Goals & Strats Propose Strategies Trigger Goals Assimilate Instruction Tutoring/Learning principle Limit the nesting of lessons

15 USC INFORMATION SCIENCES INSTITUTE Gil & Kim Tutoring and Learning Principles in KA Tools Observation: Some learning and tutoring principles are used in some aspects of the dialogue by some tools Opportunity: Incorporate principles more thoroughly in all aspects of the dialogue Observation: These principles are implicit in the tool’s code and thus are limited Opportunity: Exploit declarative representations of learning state, goals, and strategies

16 USC INFORMATION SCIENCES INSTITUTE Gil & Kim USER INTERFACE KB Proactive Dialogue Window Active Acquisition Goals & Strategies Awareness Annotations SLICK Dialogue Manager KB State Dial. History SLICK (Skills for Learning and Interactively Capture Knowledge) KNOWLEDGE ACQUISITION BACKEND Tutoring & Learning Principles

17 USC INFORMATION SCIENCES INSTITUTE Gil & Kim Conclusions Analysis of existing KA tools shows they use tutoring/learning principles  Sparsely  Implicitly Current capabilities of KA tools can be improved by:  Representing tutoring/learning principles declaratively  Organizing the dialogue around lesson topics  Keeping track of how knowledge improves through dialogue  Exposing what knowledge has been assimilated and what areas need improvement or testing  Assessing their competence and confidence on question answering