1. Author: Fang Wei, Glenn Blank Department of Computer Science Lehigh University July 10, 2007 A Student Model for an Intelligent Tutoring System Helping Novices Learn Object Oriented Design

2. Intelligent Tutoring System (ITS) A computer-based instructional system A computer-based instructional system has knowledge bases for instructional content and teaching strategies has knowledge bases for instructional content and teaching strategies uses a student’s level of mastery of topics to adapt instruction dynamically uses a student’s level of mastery of topics to adapt instruction dynamically A cost-effective means of one-on-one tutoring to provide novices with individualized attention A cost-effective means of one-on-one tutoring to provide novices with individualized attention Computer Assisted Instruction (CAI) system does not model what a student is learning and cannot adapt to student Computer Assisted Instruction (CAI) system does not model what a student is learning and cannot adapt to student CAI provides same instruction, problems and feedback to every student CAI provides same instruction, problems and feedback to every student

3. Intelligent Tutoring System Typically contains three main components: Typically contains three main components: An expert evaluator that observes a student’s work and identifies errors in his/her solution An expert evaluator that observes a student’s work and identifies errors in his/her solution A student model that diagnoses gap in student’s knowledge A student model that diagnoses gap in student’s knowledge A pedagogical advisor that provides feedback to student A pedagogical advisor that provides feedback to student

4. Student Model Maintains a model of students’ current knowledge state by representing and updating Maintains a model of students’ current knowledge state by representing and updating Provides information for intelligent pedagogical decisions and actions including: Provides information for intelligent pedagogical decisions and actions including: curriculum sequencing curriculum sequencing interactive problem solving support interactive problem solving support pedagogical tutoring customized to each individual student’s learning state pedagogical tutoring customized to each individual student’s learning state

5. Authors System Context Consider history Diagnose Concept Pre-requisitesRealTime Murray (1998) Desktop Associate skills√ VanLehn et al.(2001, 2005) Solve physics problems rules, not concepts √ Butz et al. (2004) C++ programming √ No evaluation Millan et al.(2002, 2005) CAT for math √√ Post- process Reye(1996, 1998, 2004) Theoretical analysis √√ Wei&Blank (2006,2007) OO Design (UML) √√√√ Student Model in Wei & Blank (2006,2007) compared with other BN Student Models

6. Layers of Student Knowledge (Self 1994) Domain knowledge layer Domain knowledge layer explain all vocabulary for discussing or solving problems explain all vocabulary for discussing or solving problems Reasoning knowledge layer Reasoning knowledge layer contain reasoning relationships between propositions in domain knowledge contain reasoning relationships between propositions in domain knowledge Monitoring knowledge layer Monitoring knowledge layer specify how to solve a problem using reasoning knowledge and domain knowledge specify how to solve a problem using reasoning knowledge and domain knowledge Reflective knowledge layer Reflective knowledge layer specify appropriate strategies students should have in a learning environment specify appropriate strategies students should have in a learning environment

7. Three Layered Architecture CM recognizes cognitive strategies that a student is using HM simulates students’ hierarchical knowledge in a history PDM simulates current students’ hierarchical knowledge

8. actor actor_object object object_class class class_attribute attribute attribute_constructor constructor double int numeric datatype datatype string datatype_variable variable variable_parameter parameter variable_returntype returntype pass in only class_method method method_constructor class_constructor object_constructor method_parameter variable_attribute object_attribute object_method double_int int_string double_string method_returntype datatype_returntype attribute_method attribute_parameter actor_method A is prerequisite of B A B Curriculum Information Network

9. Two kinds of concepts Unique concept, such as attribute or parameter Unique concept, such as attribute or parameter Relationship concepts, such as attribute_parameter Relationship concepts, such as attribute_parameter Relationships emerge because of student’s confusions between concepts Relationships emerge because of student’s confusions between concepts E.g., student defines movieTitle as a parameter when he has already defined movieTitle as an attribute E.g., student defines movieTitle as a parameter when he has already defined movieTitle as an attribute

10. Prerequisite relationships Prerequisite is relationship between concepts: Prerequisite is relationship between concepts: The concepts a learner needs to understand before understanding a concept The concepts a learner needs to understand before understanding a concept E.g., one needs to understand int and double in order to understand numericDatatype E.g., one needs to understand int and double in order to understand numericDatatype Relationship concepts are prerequisites of unique concepts and vice versa Relationship concepts are prerequisites of unique concepts and vice versa E.g., class_constructor -> constructor E.g., class_constructor -> constructor Understanding constructor doesn’t imply understanding of class, just how to define a constructor for a class Understanding constructor doesn’t imply understanding of class, just how to define a constructor for a class

11. Connecting Knowledge with Performance Student action unit and knowledge unit make a pair(KU,AU) Student action unit and knowledge unit make a pair(KU,AU) Infer understanding of a concept (KU) from a student solution step (AU) Infer understanding of a concept (KU) from a student solution step (AU) Action unit (AU): Action unit (AU): A single action or step in a student’s solution A single action or step in a student’s solution E.g., add an attribute to a class E.g., add an attribute to a class Knowledge unit (KU) – concept a student need to learn Knowledge unit (KU) – concept a student need to learn KU directly causes a student action unit KU directly causes a student action unit KU is a concept in Curriculum Information Network (CIN) KU is a concept in Curriculum Information Network (CIN) au ku

12. …… au ku d-prereq(ku ) 1 d-prereq(ku ) 2 d-prereq(ku ) N Atomic Bayesian Network (ABN) Noisy-and generalizes logical-and Students must understand all direct prerequisites of the concept ku in order to understand ku

13. How to generate an ABN Student model generates an ABN in response to a student solution step Student model generates an ABN in response to a student solution step First, define the structure of an ABN, i.e., the causal relationship between KU and AU, and the direct-prerequisites of KU First, define the structure of an ABN, i.e., the causal relationship between KU and AU, and the direct-prerequisites of KU Second, determine conditional probability tables for this ABN Second, determine conditional probability tables for this ABN

14. … au ku d-p(ku) 1 d-p(ku) 2 d-p(ku) N … au ku d-p(ku) 1 d-p(ku) 2 d-p(ku) N 0 0 0 0 0 1 1 1 1 1 Atomic Dynamic Bayesian Network (ADBN) for HM layer

15. How to generate an ADBN Student model generates an ADBN in response to a student solution step Student model generates an ADBN in response to a student solution step First, look for the ABN in response to previous student solution step First, look for the ABN in response to previous student solution step Second, generate an ABN in response to current student solution step Second, generate an ABN in response to current student solution step Third, determine conditional probability tables for the ADBN Third, determine conditional probability tables for the ADBN

16. Concrete Example Student defined movieTitle as a parameter for method displayMovieTitle after she has already defined movieTitle as an attribute to a class TicketMachine Student defined movieTitle as a parameter for method displayMovieTitle after she has already defined movieTitle as an attribute to a class TicketMachine EE determines that movieTitle should not be a parameter EE determines that movieTitle should not be a parameter SM determines that the center concept of an ABN is attribute_parameter, and finds all direct prerequisites, attribute and parameter, from CIN SM determines that the center concept of an ABN is attribute_parameter, and finds all direct prerequisites, attribute and parameter, from CIN

17. Concrete Example attribute’s prior can be found from the database attribute’s prior can be found from the database parameter’s prior is 0.5, students’ knowledge state is assessed based on the difference between prior and posterior probabilities (VanLehn et al. 1998, Millán & Pérez-de-la-Cruz 2002) parameter’s prior is 0.5, students’ knowledge state is assessed based on the difference between prior and posterior probabilities (VanLehn et al. 1998, Millán & Pérez-de-la-Cruz 2002) SM determines: SM determines: student has good understanding of class, attribute, methods, and parameter but low understanding of attribute_parameter student has good understanding of class, attribute, methods, and parameter but low understanding of attribute_parameter the tutoring need is: explanation of attribute_parameter the tutoring need is: explanation of attribute_parameter

18. Concrete Example feedback “Since you have added movieTitle as an attribute to the class TicketMachine, you shouldn’t also make it a parameter to the method displayMovieTitle. To decide whether movieTitle should be an attribute or a parameter, remember: attributes are accessible anywhere within the scope of a class, while parameters are accessible only within the scope of a method” “Since you have added movieTitle as an attribute to the class TicketMachine, you shouldn’t also make it a parameter to the method displayMovieTitle. To decide whether movieTitle should be an attribute or a parameter, remember: attributes are accessible anywhere within the scope of a class, while parameters are accessible only within the scope of a method”

19. Conclusions Student models with ADBNs can diagnose student knowledge states accurately in real-time Student models with ADBNs can diagnose student knowledge states accurately in real-time Accuracy of ADBN-based student model is significantly higher than ABN student model Accuracy of ADBN-based student model is significantly higher than ABN student model

20. Future work Implement cognitive model to simulate monitoring knowledge and reflective knowledge Implement cognitive model to simulate monitoring knowledge and reflective knowledge Consider students learning gain from reviewing feedback Consider students learning gain from reviewing feedback how do we determine the conditional probability table for the ADBN so as to simulate the real student learning? how do we determine the conditional probability table for the ADBN so as to simulate the real student learning? how do we update the new ADBN? how do we update the new ADBN? how do we convey empirical studies with simulated students and human subjects? how do we convey empirical studies with simulated students and human subjects? Diagnose students’ learning state in other domains, such as object-oriented programming Diagnose students’ learning state in other domains, such as object-oriented programming