SimStudent: A computational model of learning for Intelligent Authoring and beyond Noboru Matsuda Human-Computer Interaction Institute Carnegie Mellon University

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 2 CTAT: Cognitive Tutor Authoring Tools Example-Tracing Tutor –Zero programming –A cognitive model specific to a particular problem –Limited generalization by editing a behavior graph Model-Tracing Tutor –Requires a cognitive model –Cognitive task analysis is hard –Writing production rules is even more challenging Performing a task is much easier…

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 3 Next Generation Authoring Build a tutor GUI Teaching a solution SimSt. learning Rule simplify-LHS: IFis-equation( Eq ), is-lhs( Eq, Lhs ), polynomial( Lhs ), all-var-terms( Lhs ) Thensimplify( Lhs, S-lhs ), enter( S-lhs ) Production Rules Rule simplify-LHS: IFis-equation( Eq ), is-lhs( Eq, Lhs ), polynomial( Lhs ), all-var-terms( Lhs ) Thensimplify( Lhs, S-lhs ), enter( S-lhs ) Rule simplify-LHS: IFis-equation( Eq ), is-lhs( Eq, Lhs ), polynomial( Lhs ), all-var-terms( Lhs ) Thensimplify( Lhs, S-lhs ), enter( S-lhs )

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 4 SimStudent Machine learning agent –Learns problem-solving steps by … –Observes model solutions / solving problems, and … –Outputs a set of production rules Fundamental technology –Programming by Demonstration –Inductive Logic Programming Lau & Weld (1998). Blessing (1997).

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) Authoring Strategies Authoring by demonstration –Demonstrate whole solutions –Learning from worked-out examples Demonstrated solutions as positive examples Authoring by tutoring –Interactively tutor with immediate feedback and hint –Learning by tutored problem-solving Learning by generalizing hint with taking feedback into account 5

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) Demo! 6 Authoring by Tutoring

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) Learning Production Rules in 3 parts: What-When-How If among this and that GUI elements such and such constraints hold Then do actions with the GUI elements When What How

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) Production Rule in JESS GUI elements Constraints Actions

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 9 Background Knowledge Domain concepts to “explain” demonstrations –Operators –Feature predicates External Jess function written in Java (defrule multi-lhs … ?var22140 <- (column (cells ? ? ?var22143 ? ? ? ? ?)) ?var22143 <- (cell (value ?val0&~nil)) (test (fraction-term ?val0 )) => (bind ?val2 (denominator ?val0)) (bind ?input (mul-term-by ?val0 ?val2)) … )

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 10 Example ： Algebra domain 16 Feature predicates & 28 operators Feature Predicates for LHS conditionsOperators for RHS actions HasCoefficient HasConstTerm HasVarTerm Homogeneous IsFractionTerm IsConstant IsDenominatorOf IsNumeratorOf IsPolynomial Monomial NotNull VarTerm IsSkillAdd IsSkillSubtract IsSkillDivide IsSkillMultiply AddTerm AddTermBy Coefficient CopyTerm Denominator DivTerm DivTermBy EvalArithmetic FirstTerm FirstVarTerm GetOperand InverseTerm LastConstTerm LastTerm LastVarTerm MulTerm MulTermBy Numerator ReverseSign RipCoefficient SkillAdd SkillClt SkillDivide SkillMultiply SkillRf SkillMt SkillSubtract VarName

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 11 Learning Results # of training tasks % Correct rule firings (10 test tasks) Authoring by tutoring Authoring by demonstration Better than

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 12 Authoring Time Authoring by Tutoring Authoring by Demonstration Authoring by tutoring took 86 minutes Authoring by demonstration took 238 minutes A 2.8x speed-up!

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 13 Example: Stoichiometry Tutor

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) SimStudent Projects Intelligent Authoring –Building a Cognitive Tutor as a CTAT Plug-in Student Modeling and Simulation –Controlled educational studies –Error formation study –Prerequisite conceptual knowledge study Teachable Peer Learner –Learning by teaching 14

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) SimStudent Projects Intelligent Authoring –Building a Cognitive Tutor as a CTAT Plug-in Student Modeling and Simulation –Controlled educational studies –Error formation study –Prerequisite conceptual knowledge study Teachable Peer Learner –Learning by teaching 15

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) Model of Incorrect Learning Identify errors students commonly make Weaken SimStudent’s background knowledge Let SimStudent make an induction error 16

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) Weak Prior Knowledge Hypothesis Multiple ways to make sense of examples 17 3x=5 “divide by 3” Get a coefficient and divide Get a number and divide Get a denominator and multiply 4/x=5 strong prior knowledge weak prior knowledge “divide by 4” “multiply by x”

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) Target Common Errors 18

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) Results: Learning Rate Steps Score = 0 (if there is no rule applicable) # correct rule applications / # all rule applications # training problems Step Score

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) SimStudent Projects Intelligent Authoring –Building a Cognitive Tutor as a CTAT Plug-in Student Modeling and Simulation –Controlled educational studies –Error formation study –Prerequisite conceptual knowledge study Teachable Peer Learner –Learning by teaching 20

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) Learning by Teaching SimStudent

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) Demo! 22 Learning by Teaching

SimStudent Project PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) Learn more about SimStudents Project Web – Download & Tutorial – (linked from project web) Contact us! 23