1. Author: Fang Wei Advisor: Prof. Blank Department of Computer Science Lehigh University May 10, 2007 A Student Model for an Intelligent Tutoring System Helping Novices Learn Object Oriented Design

2. Presentation Outline Publications Publications Background Background Research questions Research questions Methodology Methodology Evaluation Evaluation Conclusions and future work Conclusions and future work

3. Publications Wei, F. & Blank, G.D. (2007) Atomic Dynamic Bayesian Networks for a Responsive Student Model, Proceedings of the 13th International Conference on Artificial Intelligence in Education, AIED 2007 Wei, F. & Blank, G.D. (2007) Atomic Dynamic Bayesian Networks for a Responsive Student Model, Proceedings of the 13th International Conference on Artificial Intelligence in Education, AIED 2007 Wei, F. & Blank, G.D. (2006) Student Modeling with Atomic Bayesian Networks, Proceedings of the 8th International Conference on Intelligent Tutoring Systems, ITS 2006, pp. 491- 502 Wei, F. & Blank, G.D. (2006) Student Modeling with Atomic Bayesian Networks, Proceedings of the 8th International Conference on Intelligent Tutoring Systems, ITS 2006, pp. 491- 502 Wei, F., Moritz, S., Parvez, S., & Blank, G.D. (2005) A Student Model for Object-Oriented Design and Programming. The Journal of Computing Sciences in Colleges (CCSC), Vol. 20, pp. 260-273. "Best Paper" Award Wei, F., Moritz, S., Parvez, S., & Blank, G.D. (2005) A Student Model for Object-Oriented Design and Programming. The Journal of Computing Sciences in Colleges (CCSC), Vol. 20, pp. 260-273. "Best Paper" Award Blank, G. D., Parvez, S., Wei, F., Moritz, S (2005) A web-based ITS for object-oriented design. Poster for 12th International Conference on Artificial Intelligence in Education, Workshop of Adaptive Systems for Web-Based Education: Tools and reusability, Amsterdam, The Netherlands, June Blank, G. D., Parvez, S., Wei, F., Moritz, S (2005) A web-based ITS for object-oriented design. Poster for 12th International Conference on Artificial Intelligence in Education, Workshop of Adaptive Systems for Web-Based Education: Tools and reusability, Amsterdam, The Netherlands, June Moritz, S., Wei, F., Parvez, S., & Blank, G. D. (2005), From objects-first to design-first with multimedia and intelligent tutoring. The Tenth Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE), Monte da Caparica, Portugal, June. Moritz, S., Wei, F., Parvez, S., & Blank, G. D. (2005), From objects-first to design-first with multimedia and intelligent tutoring. The Tenth Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE), Monte da Caparica, Portugal, June.

4. Presentation Outline Publications Publications Background Background Research questions Research questions Methodology Methodology Evaluation Evaluation Conclusions and future work Conclusions and future work

5. Intelligent Tutoring System (ITS) A computer-based instructional system A computer-based instructional system has knowledge bases for instructional content and teaching strategies has knowledge bases for instructional content and teaching strategies uses a student’s level of mastery of topics to adapt instruction dynamically uses a student’s level of mastery of topics to adapt instruction dynamically A cost-effective means of one-on-one tutoring to provide novices with individualized attention A cost-effective means of one-on-one tutoring to provide novices with individualized attention Computer Assisted Instruction (CAI) system does not model what a student is learning and cannot adapt to student Computer Assisted Instruction (CAI) system does not model what a student is learning and cannot adapt to student CAI provides same instruction, problems and feedback to every student CAI provides same instruction, problems and feedback to every student

6. Intelligent Tutoring System Typically contains three main components: Typically contains three main components: An expert evaluator that observes a student’s work and identifies errors in his/her solution An expert evaluator that observes a student’s work and identifies errors in his/her solution A student model that diagnoses gap in student’s knowledge A student model that diagnoses gap in student’s knowledge A pedagogical advisor that provides feedback to student A pedagogical advisor that provides feedback to student

7. Student Model Maintains a model of students’ current knowledge state by representing and updating Maintains a model of students’ current knowledge state by representing and updating Provides information for intelligent pedagogical decisions and actions including: Provides information for intelligent pedagogical decisions and actions including: curriculum sequencing curriculum sequencing interactive problem solving support interactive problem solving support pedagogical tutoring customized to each individual student’s learning state pedagogical tutoring customized to each individual student’s learning state

8. Motivation Novices in high school and college have much difficulty in object oriented design Novices in high school and college have much difficulty in object oriented design ITSs can aid learners with complex problem- solving ITSs can aid learners with complex problem- solving DesignFirst-ITS developed to help novices learn object-oriented design (Blank et al. 2005, Moritz & Blank 2005) DesignFirst-ITS developed to help novices learn object-oriented design (Blank et al. 2005, Moritz & Blank 2005) Research has shown that an ITS that adapts to accurate student knowledge enhances learning (Corbett et al. 2000) Research has shown that an ITS that adapts to accurate student knowledge enhances learning (Corbett et al. 2000) Corbett’s ITS focuses on adaptive problem sequencing rather than adaptive feedback Corbett’s ITS focuses on adaptive problem sequencing rather than adaptive feedback Our ITS focuses on adaptive feedback Our ITS focuses on adaptive feedback

9. Authors System Context Consider history Diagnose Concept Pre-requisitesRealTime Murray (1998) Desktop Associate skills√ VanLehn et al.(2001, 2005) Solve physics problems rules, not concepts √ Butz et al. (2004) C++ programming √ No evaluation Millan et al.(2002, 2005) CAT for math √√ Post- process Reye(1996, 1998, 2004) Theoretical analysis √√ Wei&Blank (2006,2007) OO Design (UML) √√√√ Student Model in Wei & Blank (2006,2007) compared with other BN Student Models

10. Layers of Student Knowledge (Self 1994) Domain knowledge layer Domain knowledge layer explain all vocabulary for discussing or solving problems explain all vocabulary for discussing or solving problems Reasoning knowledge layer Reasoning knowledge layer contain reasoning relationships between propositions in domain knowledge contain reasoning relationships between propositions in domain knowledge Monitoring knowledge layer Monitoring knowledge layer specify how to solve a problem using reasoning knowledge and domain knowledge specify how to solve a problem using reasoning knowledge and domain knowledge Reflective knowledge layer Reflective knowledge layer specify appropriate strategies students should have in a learning environment specify appropriate strategies students should have in a learning environment

11. Common Problems with Student Models Do not consider relationship between individual concepts Do not consider relationship between individual concepts Do not represent layered knowledge (Self 1994) Do not represent layered knowledge (Self 1994) Do not simulate students’ knowledge history Do not simulate students’ knowledge history Separate the inferred students’ knowledge from closed- and open-ended exercises Separate the inferred students’ knowledge from closed- and open-ended exercises Do not consider students’ cognitive strategies including general and domain-specific Do not consider students’ cognitive strategies including general and domain-specific Bayesian student models require exponential updating time and hence cannot provide real-time tutoring adaptive to individual student’s learning state Bayesian student models require exponential updating time and hence cannot provide real-time tutoring adaptive to individual student’s learning state

12. Presentation Outline Publication Publication Background Background Research questions Research questions Methodology Methodology Evaluation Evaluation Conclusions and future work Conclusions and future work

13. Research Questions (1 of 2) Can this student model provide information for pedagogical decisions? Can this student model provide information for pedagogical decisions? How should this student model represent a student’s current knowledge state and the student’s knowledge structure? How should this student model represent a student’s current knowledge state and the student’s knowledge structure? How will the student model track students’ knowledge state over time? Under this research question there are two sub questions: How will the student model track students’ knowledge state over time? Under this research question there are two sub questions: Would tracking a history of students’ knowledge state be useful for pedagogical decisions? Would tracking a history of students’ knowledge state be useful for pedagogical decisions? Can a history be maintained efficiently enough to be responsive in real-time? Can a history be maintained efficiently enough to be responsive in real-time?

14. Research Questions (2 of 2) How to synthesize information from two different sources, open-ended problem solving (object-oriented class diagram design) and closed-ended exercises (multiple choice quizzes or drag-and-drop exercises)? How to synthesize information from two different sources, open-ended problem solving (object-oriented class diagram design) and closed-ended exercises (multiple choice quizzes or drag-and-drop exercises)? What cognitive strategies should the student model consider and how to consider them? What cognitive strategies should the student model consider and how to consider them?

15. Presentation Outline Publication Publication Background Background Research questions Research questions Methodology Methodology Evaluation Evaluation Conclusions and future work Conclusions and future work

16. Three Layered Architecture CM recognizes cognitive strategies that a student is using HM simulates students’ hierarchical knowledge in a history PDM simulates current students’ hierarchical knowledge

17. actor actor_object object object_class class class_attribute attribute attribute_constructor constructor double int numeric datatype datatype string datatype_variable variable variable_parameter parameter variable_returntype returntype pass in only class_method method method_constructor class_constructor object_constructor method_parameter variable_attribute object_attribute object_method double_int int_string double_string method_returntype datatype_returntype attribute_method attribute_parameter actor_method A is prerequisite of B A B Curriculum Information Network

18. Two kinds of concepts Unique concept, such as attribute or parameter Unique concept, such as attribute or parameter Relationship concepts, such as attribute_parameter Relationship concepts, such as attribute_parameter Relationships emerge because of student’s confusions between concepts Relationships emerge because of student’s confusions between concepts E.g., student defines movieTitle as a parameter when he has already defined movieTitle as an attribute E.g., student defines movieTitle as a parameter when he has already defined movieTitle as an attribute

19. Prerequisite relationships Prerequisite is relationship between concepts: Prerequisite is relationship between concepts: The concepts a learner needs to understand before understanding a concept The concepts a learner needs to understand before understanding a concept E.g., one needs to understand int and double in order to understand numericDatatype E.g., one needs to understand int and double in order to understand numericDatatype Relationship concepts are prerequisites of unique concepts and vice versa Relationship concepts are prerequisites of unique concepts and vice versa E.g., class_constructor -> constructor E.g., class_constructor -> constructor Understanding constructor doesn’t imply understanding of class, just how to define a constructor for a class Understanding constructor doesn’t imply understanding of class, just how to define a constructor for a class

20. Connecting Knowledge with Performance Student action unit and knowledge unit make a pair(KU,AU) Student action unit and knowledge unit make a pair(KU,AU) Infer understanding of a concept (KU) from a student solution step (AU) Infer understanding of a concept (KU) from a student solution step (AU) Action unit (AU): Action unit (AU): A single action or step in a student’s solution A single action or step in a student’s solution E.g., add an attribute to a class E.g., add an attribute to a class Knowledge unit (KU) – concept a student need to learn Knowledge unit (KU) – concept a student need to learn KU directly causes a student action unit KU directly causes a student action unit KU is a concept in Curriculum Information Network (CIN) KU is a concept in Curriculum Information Network (CIN) au ku

21. …… au ku d-prereq(ku ) 1 d-prereq(ku ) 2 d-prereq(ku ) N Atomic Bayesian Network (ABN) Noisy-and generalizes logical-and Students must understand all direct prerequisites of the concept ku in order to understand ku

22. How to generate an ABN Student model generates an ABN in response to a student solution step Student model generates an ABN in response to a student solution step First, define the structure of an ABN, i.e., the causal relationship between KU and AU, and the direct-prerequisites of KU First, define the structure of an ABN, i.e., the causal relationship between KU and AU, and the direct-prerequisites of KU Second, determine conditional probability tables for this ABN Second, determine conditional probability tables for this ABN

23. … au ku d-p(ku) 1 d-p(ku) 2 d-p(ku) N … au ku d-p(ku) 1 d-p(ku) 2 d-p(ku) N 0 0 0 0 0 1 1 1 1 1 Atomic Dynamic Bayesian Network (ADBN) for HM layer

24. How to generate an ADBN Student model generates an ADBN in response to a student solution step Student model generates an ADBN in response to a student solution step First, look for the ABN in response to previous student solution step First, look for the ABN in response to previous student solution step Second, generate an ABN in response to current student solution step Second, generate an ABN in response to current student solution step Third, determine conditional probability tables for the ADBN Third, determine conditional probability tables for the ADBN

25. Concrete Example Student defined movieTitle as a parameter for method displayMovieTitle after she has already defined movieTitle as an attribute to a class TicketMachine Student defined movieTitle as a parameter for method displayMovieTitle after she has already defined movieTitle as an attribute to a class TicketMachine EE determines that movieTitle should not be a parameter EE determines that movieTitle should not be a parameter SM determines that the center concept of an ABN is attribute_parameter, and finds all direct prerequisites, attribute and parameter, from CIN SM determines that the center concept of an ABN is attribute_parameter, and finds all direct prerequisites, attribute and parameter, from CIN

26. Concrete Example attribute’s prior can be found from the database attribute’s prior can be found from the database parameter’s prior is 0.5, students’ knowledge state is assessed based on the difference between prior and posterior probabilities (VanLehn et al. 1998, Millán & Pérez-de-la-Cruz 2002) parameter’s prior is 0.5, students’ knowledge state is assessed based on the difference between prior and posterior probabilities (VanLehn et al. 1998, Millán & Pérez-de-la-Cruz 2002) SM determines: SM determines: student has good understanding of class, attribute, methods, and parameter but low understanding of attribute_parameter student has good understanding of class, attribute, methods, and parameter but low understanding of attribute_parameter the tutoring need is: explanation of attribute_parameter the tutoring need is: explanation of attribute_parameter

27. Concrete Example feedback “Since you have added movieTitle as an attribute to the class TicketMachine, you shouldn’t also make it a parameter to the method displayMovieTitle. To decide whether movieTitle should be an attribute or a parameter, remember: attributes are accessible anywhere within the scope of a class, while parameters are accessible only within the scope of a method” “Since you have added movieTitle as an attribute to the class TicketMachine, you shouldn’t also make it a parameter to the method displayMovieTitle. To decide whether movieTitle should be an attribute or a parameter, remember: attributes are accessible anywhere within the scope of a class, while parameters are accessible only within the scope of a method”

28. Presentation Outline Publication Publication Background Background Research questions Research questions Methodology Methodology Evaluation Evaluation Conclusions and future work Conclusions and future work

29. Evaluation of ABNs with simulated students Hypotheses: Hypotheses: Pre-setting slip and guess values will lead to a reliable student model Pre-setting slip and guess values will lead to a reliable student model Varying slip and guess values will affect the accuracy of the student model Varying slip and guess values will affect the accuracy of the student model

30. Pre-setting slip and guess values to same (relatively small e.g. <=0.1) values produces accuracy of at least 93%, confirming the first hypothesis Pre-setting slip and guess values to same (relatively small e.g. <=0.1) values produces accuracy of at least 93%, confirming the first hypothesis Changing of presetting slip and guess causes accuracy to change from 79.1% to 94.3%, confirming the second hypothesis Changing of presetting slip and guess causes accuracy to change from 79.1% to 94.3%, confirming the second hypothesis Correct diagnostic rates are higher when slip p, guess p and slip e, guess e take same value Correct diagnostic rates are higher when slip p, guess p and slip e, guess e take same value No significant difference when slip and guess take a same small value (<=0.1) No significant difference when slip and guess take a same small value (<=0.1)

31. Evaluation of ADBNs with simulated students Hypotheses: Hypotheses: Pre-setting slip and guess values can lead to a reliable student model Pre-setting slip and guess values can lead to a reliable student model Modeling learning history with ADBNs will enhance the accuracy of the student model Modeling learning history with ADBNs will enhance the accuracy of the student model

32. The significant difference between correct diagnostic rates using ABNs versus using ADBNs demonstrates that ADBNs enhance the accuracy of the student model, confirming the second hypothesis The significant difference between correct diagnostic rates using ABNs versus using ADBNs demonstrates that ADBNs enhance the accuracy of the student model, confirming the second hypothesis

33. Pre-setting slip p/e, guess p/e, to relatively small (e.g. <=0.1) values produces accuracy of at least 97%, confirming the first hypothesis. Pre-setting slip p/e, guess p/e, to relatively small (e.g. <=0.1) values produces accuracy of at least 97%, confirming the first hypothesis. The accuracy is not sensitive to change of slip and guess values so along as the values are relatively small (<=0.1) The accuracy is not sensitive to change of slip and guess values so along as the values are relatively small (<=0.1)

34. Evaluation of student model for CIMEL multimedia with real students Hypotheses: Hypotheses: Pre-setting slip, guess will lead to a reliable student model Pre-setting slip, guess will lead to a reliable student model Pre-setting slip and guess to various values will affect the accuracy of the student model Pre-setting slip and guess to various values will affect the accuracy of the student model The student model will perform in real-time, i.e., it will support responses as students are working on exercises The student model will perform in real-time, i.e., it will support responses as students are working on exercises Results: Results: The average response time of the student model after a students enters the solution step is 0.24 seconds The average response time of the student model after a students enters the solution step is 0.24 seconds

35. Presetting slip and guess with relatively small values (<=0.1) can produce accuracy of up to 80.1%. Presetting slip and guess with relatively small values (<=0.1) can produce accuracy of up to 80.1%. Changing the presetting slip and guess causes the accuracy to change from 71.7% to 80.1% Changing the presetting slip and guess causes the accuracy to change from 71.7% to 80.1%

37. Has a higher value of r than student model by Corbett et al. (2000) Has a higher value of r than student model by Corbett et al. (2000)

38. Hypotheses: Hypotheses: Pre-setting slip, guess will lead to a reliable student model Pre-setting slip, guess will lead to a reliable student model The student model will perform in real-time, i.e., it will support responses as students are working on problem-solving steps The student model will perform in real-time, i.e., it will support responses as students are working on problem-solving steps Results: Results: The average response time of the student model after a student enters a solution step is 0.63 seconds The average response time of the student model after a student enters a solution step is 0.63 seconds Evaluation of student model for DesignFirst-ITS with real students

39. Presetting slip and guess with relatively small values (<=0.1) can produce accuracy of up to 81.8% Presetting slip and guess with relatively small values (<=0.1) can produce accuracy of up to 81.8% Varying the slip and guess value does not affect the accuracy of the student model, so long as slip and guess values are relatively small (<=0.1) Varying the slip and guess value does not affect the accuracy of the student model, so long as slip and guess values are relatively small (<=0.1)

40. Has a higher value of r than student model by Corbett et al. (2000) Has a higher value of r than student model by Corbett et al. (2000)

41. Evaluation of diagnoses integration (multimedia and DesignFirst-ITS) with real students Hypothesis: Hypothesis: Integrating diagnoses from closed-ended questions will enhance the accuracy of diagnoses of student model for open-ended questions Integrating diagnoses from closed-ended questions will enhance the accuracy of diagnoses of student model for open-ended questions

42. Accuracy increases 7.7% when adding diagnoses from closed-ended exercises in multimedia, confirming hypothesis Accuracy increases 7.7% when adding diagnoses from closed-ended exercises in multimedia, confirming hypothesis

43. Comparing with non-advanced- numerical student models Non-advanced-numerical techniques include match, summation and subtraction Non-advanced-numerical techniques include match, summation and subtraction Advanced-numerical techniques include Bayesian networks Advanced-numerical techniques include Bayesian networks Hypotheses: Hypotheses: Straightforward algorithm will not lead to a reliable student model Straightforward algorithm will not lead to a reliable student model ADBNs will perform better than “match” student model ADBNs will perform better than “match” student model

44. From same set of evidences, ADBNs perform more than two times better than “match” student models From same set of evidences, ADBNs perform more than two times better than “match” student models

45. Presentation Outline Publications Publications Background Background Related Research Related Research Research questions Research questions Methodology Methodology Evaluation Evaluation Conclusions and future work Conclusions and future work

46. Conclusions Student models with ADBNs can diagnose student knowledge states accurately in real-time Student models with ADBNs can diagnose student knowledge states accurately in real-time Accuracy of ADBN-based student model is significantly higher than ABN student model Accuracy of ADBN-based student model is significantly higher than ABN student model Integrating diagnoses from closed- and open- ended exercises is an effective way to increase accuracy of student models Integrating diagnoses from closed- and open- ended exercises is an effective way to increase accuracy of student models Student models using ADBNs perform much better than the student models that use straightforward algorithm Student models using ADBNs perform much better than the student models that use straightforward algorithm

47. Future work Implement cognitive model to simulate monitoring knowledge and reflective knowledge Implement cognitive model to simulate monitoring knowledge and reflective knowledge Consider students learning gain from reviewing feedback Consider students learning gain from reviewing feedback how do we determine the conditional probability table for the ADBN so as to simulate the real student learning? how do we determine the conditional probability table for the ADBN so as to simulate the real student learning? how do we update the new ADBN? how do we update the new ADBN? how do we convey empirical studies with simulated students and human subjects? how do we convey empirical studies with simulated students and human subjects? Diagnose students’ learning state in other domains, such as object-oriented programming Diagnose students’ learning state in other domains, such as object-oriented programming

48. An ADBN considers feedback

49. Contributions (1 of 2) A novel way to represent students’ knowledge structure, where both concepts and relationship between concepts are knowledge units that students need to learn A novel way to represent students’ knowledge structure, where both concepts and relationship between concepts are knowledge units that students need to learn A novel three-layered architecture which can be standardized in modeling various stratums of students’ knowledge A novel three-layered architecture which can be standardized in modeling various stratums of students’ knowledge ABN – a novel Atomic Bayesian network that provides a refined representation of prerequisite relationships, diagnoses student’s knowledge structure, and guarantees real-time responsiveness ABN – a novel Atomic Bayesian network that provides a refined representation of prerequisite relationships, diagnoses student’s knowledge structure, and guarantees real-time responsiveness

50. Contributions (2 of 2) ADBN – an innovative dynamic Bayesian network that represents refined representation of prerequisite relationships and diagnoses students’ knowledge structure in real-time considering learning history ADBN – an innovative dynamic Bayesian network that represents refined representation of prerequisite relationships and diagnoses students’ knowledge structure in real-time considering learning history A unique student model that integrates knowledge from open-ended problem solving (object-oriented class diagram design) and closed-ended exercises A unique student model that integrates knowledge from open-ended problem solving (object-oriented class diagram design) and closed-ended exercises A general approach for student models that help students learn complex problem solving in real time A general approach for student models that help students learn complex problem solving in real time

51. Questions?