1 Cognitive Tutor Development Kit (TDK) Tutorial Cognitive Modeling & Intelligent Tutoring Systems Ken Koedinger Vincent Aleven

2 Overview Basics of TDK Notation Matching: $ and = variables Modeling alternative strategies “Interface constraints” for model tracing Control flow and production chains Writing hints and bug productions Creating a curriculum

3 Production Rules in the Tutor Development Kit (TDK) How do you go about writing a production rule? The TDK Production System –Working Memory: Make up of “Working Memory Elements” (WMEs) –Production Rules –Interpreter (Match, Conflict Resolution, Fire)

4 Implementing a Production Rule Model in TDK Simple example: a model for single- column addition without carrying! How would you define: –Working Memory representation for the problem states –Production rules that transform working memory Done

5 A Notation for Working Memory Elements (“WMEs”) in TDK PROBLEM4+3> ISA SINGLE-COLUMN-ADDITION-PROBLEM FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT nil DONE nil WME Name Slot Values Slots WME Type Separator

6 Working Memory Transitions PROBLEM4+3> ISA SINGLE-COLUMN- ADDITION-PROBLEM FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT nil DONE nil Done PROBLEM4+3> ISA SINGLE-COLUMN- ADDITION-PROBLEM FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT 7 DONE nil PROBLEM4+3> ISA SINGLE-COLUMN- ADDITION-PROBLEM FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT 7 DONE T Production: ADD Production: DONE

7 TDK Production Rule Notation (defproduction add single-column-addition (=problem) =problem> isa single-column-addition-problem result NIL first-addend =num1 second-addend =num2 ==> !eval! =sum (+ =num1 =num2) =problem> result =sum ) ADD (English version) IF The goal is to do =problem, a single-column addition problem And no result has been found yet And the first addend is =num1 And the second added is =num2 THEN Set =sum to the sum of =num1 and =num2 Write =sum as the result

8 TDK Production Rule Notation (defproduction add single-column-addition (=problem) =problem> isa single-column-addition-problem result NIL first-addend =num1 second-addend =num2 ==> !eval! =sum (+ =num1 =num2) =problem> result =sum ) If-part Then-part

9 TDK Production Rule Notation (defproduction add single-column-addition (=problem) =problem> isa single-column-addition-problem result NIL first-addend =num1 second-addend =num2 ==> !eval! =sum (+ =num1 =num2) =problem> result =sum ) If-part Then-part WME pattern Computation in Lisp

10 TDK Production Rule Notation (defproduction add single-column-addition (=problem) =problem> isa single-column-addition-problem result NIL first-addend =num1 second-addend =num2 ==> !eval! =sum (+ =num1 =num2) =problem> result =sum ) WME pattern Variables (“bound” to values through matching) Separator

11 Matching a Production Rule to Working Memory—Find Values for Each Variable Working Memory problem4+3> isa single-column- addition-problem first-addend 4 second-addend 3 result NIL done NIL Find value for each variable Variable =problem =num1 =num2 =sum Value Problem Match! Modify WME problem4+3> isa single-column- addition-problem first-addend 4 second-addend 3 result 7 done NIL Production Rule add =problem> isa single-column- addition-problem result NIL first-addend =num1 second-addend =num2 ==> !eval! =sum (+ =num1 =num2) =problem> result =sum ) This WME pattern in the THEN-part means change slot value of existing WME What changes are made to Working Memory?

12 Summary—Components of a TDK Production Rule Model Working Memory is a collection of WMEs wme-name> isa wme-type slot1 value1 slot2 value2 Production rules specify working memory transitions –WME patterns in the IF-part are matched against Working Memory =wme-variable> isa wme-type slot1 value-pattern1 slot2 value-pattern2 –WME patterns in the THEN-part specify changes to Working Memory

13 Overview Basics of TDK Notation Matching: $ and = variables Modeling alternative strategies “Interface constraints” for model tracing Control flow and production chains Writing hints and bug productions Creating a curriculum

14 WME Representation Done Value of a slot in a WME can be a list. Initial State PROBLEM26+51> ISA MULTI-COLUMN-ADDITION-PROBLEM COLUMNS (COLUMN2 COLUMN1) DONE nil COLUMN1> ISA COLUMN FIRST-ADDEND 6 SECOND-ADDEND 1 RESULT nil COLUMN2> ISA COLUMN FIRST-ADDEND 2 SECOND-ADDEND 5 RESULT nil Final State PROBLEM26+51> ISA MULTI-COLUMN-ADDITION-PROBLEM COLUMNS (COLUMN2 COLUMN1) DONE T COLUMN1> ISA COLUMN FIRST-ADDEND 6 SECOND-ADDEND 1 RESULT 7 COLUMN2> ISA COLUMN FIRST-ADDEND 2 SECOND-ADDEND 5 RESULT 7

15 Production Rules Set New Goals & Perform Actions FOCUS-ON-FIRST-COLUMN, FOCUS-ON-NEXT-COLUMN ADD Goal: Solve the addition problem Goal: Process column C Action: Write the sum Action: Mark as done DONE

16 Goals in Production Rules FOCUS-ON-FIRST-COLUMN IF The goal is to do a multi-column addition problem And C is the rightmost column of the problem And there are numbers to add in C And there is no result yet in C THEN Set a subgoal to process column C FOCUS-ON-NEXT-COLUMN IF The goal is to do a multi-column addition problem And C is the rightmost column with numbers to add and no result THEN Set a subgoal to process column C WRITE-RESULT IF There is a goal to process column C THEN Set Sum to the sum of the addends in column C Write Sum as the result in column C DONE IF The goal is to do a multi-column addition problem And there is no column left with numbers to add and no result THEN Mark the problem as done

17 Figuring Out Which Column FOCUS-ON-FIRST-COLUMN IF The goal is to do a multi-column addition problem And C is the rightmost column of the problem And there are numbers to add in C And there is no result yet in C THEN Set a subgoal to process column C FOCUS-ON-NEXT-COLUMN IF The goal is to do a multi-column addition problem And C is the rightmost column with numbers to add and no result THEN Set a subgoal to process column C WRITE-RESULT IF There is a goal to process column C THEN Set Sum to the sum of the addends in column C Write Sum as the result in column C DONE IF The goal is to do a multi-column addition problem And there is no column left with numbers to add and no result THEN Mark the problem as done

Example Production Rule (defproduction focus-on-next-column multi-column-addition (=problem) =problem> isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> isa column - result NIL =next-column> isa column result NIL first-addend =num1 second-addend =num2 !eval! (or =num1 =num2) ==> =process-column-goal> isa process-column-goal column =next-column !chain! multi-column-addition (=process-column-goal)) FOCUS-ON-NEXT-COLUMN IF The goal is to do =problem, a multi-column addition problem And =next-column and =previous-column are columns next to each other And there is a result in =previous-column And there is no result in =next-column And =num1 is the 1st addend in =next-column And =num2 is the 2nd addend in =next-column And there is no result yet in =next-column And =num1 or =num2 is not blank THEN Set a subgoal to process =next-column

TDK Production Rule Notation (defproduction focus-on-next-column multi-column-addition (=problem) =problem> isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> isa column - result NIL =next-column> isa column result NIL first-addend =num1 second-addend =num2 !eval! (or =num1 =num2) ==> =process-column-goal> isa process-column-goal column =next-column !chain! multi-column-addition (=process-column-goal)) If-part Then-part WME pattern

TDK Production Rule Notation (defproduction focus-on-next-column multi-column-addition (=problem) =problem> isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> isa column - result NIL =next-column> isa column result NIL first-addend =num1 second-addend =num2 !eval! (or =num1 =num2) ==> =process-column-goal> isa process-column-goal column =next-column !chain! multi-column-addition (=process-column-goal)) Variables multiple occurrences of same variable in IF-part must match the same value

TDK Production Rule Notation (defproduction focus-on-next-column multi-column-addition (=problem) =problem> isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> isa column - result NIL =next-column> isa column result NIL first-addend =num1 second-addend =num2 !eval! (or =num1 =num2) ==> =process-column-goal> isa process-column-goal column =next-column !chain! multi-column-addition (=process-column-goal)) Variables within a list whose name starts with $ match any number of items (0 or more).

22 Matching a Production Rule to Working Memory—Find Values for Each Variable (1) Production Rule focus-on-next-column =problem> isa multi-column- addition-problem columns ($ =next-col =prev-col $) =prev-col> isa column - result NIL =next-col> isa column result NIL first-addend =num1 second-addend =num2 !eval! (or =num1 =num2) ==> =process-column-goal> isa process-column-goal column =next-col Working Memory problem26+51> isa multi-column- addition-problem columns (column2 column1) done nil column1> isa column first-addend 6 second-addend 1 result 7 column2> isa column first-addend 2 second-addend 5 result nil Find value for each variable Variable =problem 1st $ =next-col =prev-col 2nd $ =num1 =num2 Value problem26+51 () column2 column1 () 6 1 Match! Create new WME isa process-column-goal column column2 What changes are made to Working Memory? When a WME pattern with “isa” is in the THEN- part of a rule, it means create a new WME rather than modify an existing one.

23 Value to be matched: (COLUMN4 COLUMN3 COLUMN2 COLUMN1) Pattern: ($ =next-column =previous-column $) Match 1 Left $() =next-column COLUMN4 =previous-column COLUMN3 Right $(COLUMN2 COLUMN1) Match 2 Left $(COLUMN4) =next-column COLUMN3 =previous-column COLUMN2 Right $(COLUMN1) Match 3 Left $(COLUMN4 COLUMN3) =next-column COLUMN2 =previous-column COLUMN1 Right $() Pattern Matching with Lists—Patterns with $ variable(s) often create multiple matches Common pattern to extract the next element of a list.

24 TDK Production Rule Notation “==>” separates if-part & then-part If- & then-parts are made up of WME patterns of form: =wme-variable> isa wme-type slot1 value-pattern1 slot2 value-pattern2 An atom starting with either “=” or “$” is a variable –Ex: “=num” “$columns” –“=” vars match atoms –“$” variables match 0 or more elements of a list –“=” or “$” by itself is just an unnamed (dummy) variable

25 The TDK Production System Interpreter Match phase: –Attempt to match each WME pattern with 1 or more WMEs in working memory –Variables are “bound” to a value. Subsequent references to those variables must have the same value. –A set of “bindings” for all variables in the if-part is called a “instance” of a production. –If a consistent set of bindings is found, the production “matches”, otherwise it “fails”. –A production can match working memory in more than one way, that is, it can have multiple instances (or rule instantiations). Conflict resolution phase: –In stand-alone mode: Determined by hand or arbitrarily –In tutor mode: Production priorities & model tracing heuristics Act phase (fire a production): –Use the variable bindings to modify or create the WME indicated in the then-part. –If “isa” slot is present, create a WME, otherwise modify.

26 Overview Basics of TDK Notation Matching: $ and = variables Modeling alternative strategies “Interface constraints” for model tracing Control flow and production chains Writing hints and bug productions Creating a curriculum

27 Modeling Alternative Strategies Consider multiple solution paths –Reordering of steps Write sum first or write carry in next column first –Alternative strategies Left to right strategy Use calculator and write result strategy –Shortcuts Implicit carrying Which of these will the model cover? Motivate choices pedagogically!

28 Execution Space Diagram Illustrating Flexibility

29 Production Rules Set New Goals & Perform Actions Goal: Solve the addition problem ADD-CARRY FOCUS-ON-FIRST-COLUMN, FOCUS-ON-NEXT-COLUMN Goal: Process column C Goal: Write carry in next column MUST-CARRY WRITE-SUM Action: Write the sum Action: Write the carry WRITE-CARRY Goal: Write sum in column C ADD-ADDENDS

30 Model Tracing 1 — Using a Cognitive Model to Evaluate Students’ Solution Steps Cognitive model is an expert system that –is psychologically plausible –accounts for all reasonable solutions –accounts for commonly occurring errors (“bugs”) To verify solution step by student –Use model to generate correct and buggy next steps –If student step matches Correct step: Accept –update Working Memory to reflect new problem state Buggy step: Give error-specific feedback No step: Reject without feedback

31 Overview Basics of TDK Notation Matching: $ and = variables Modeling alternative strategies “Interface constraints” for model tracing Control flow and production chains Writing hints and bug productions Creating a curriculum

32 Communicating About Student Solution Steps between Interface and Tutor Represent each observable action in the user interface as a selection-action-input triple. Selection = (column2) Action = write-carry Input = Selection = (column1) Action = write-sum Input =

33 Execution Space Diagram with Selection-Action-Input Triples Selection: Column1 Action:Write-Result Input: 0 Selection: Column2 Action:Write-Result Input: 8 Selection: Column3 Action:Write-Result Input: 9 Selection: Column2 Action:Write-Carry Input: 1 Selection: Column1 Action:Write-Result Input: 0 Selection: Column2 Action:Write-Carry Input: Done. Selection: - Action:Done Input: -

34 Interface Constraints—Matching Student Steps Against Model Steps Production rule writer can attach “model-tracing” or “interface constraints” to rules. –Specify tests on the selection-action-input triples. During model-tracing TDK will not consider productions whose constraints are violated.

35 CONSTRAINTS—consider rule only if: Selection is: Action is: Input is: Example of Interface Constraints in English WRITE-SUM IF There is a goal to write Sum as the result in column C And Sum < 10 And the carry into column C (if any) has been added THEN Write Sum as the result in column C And remove the goal Same as value of variable C Write-Result Equal to the value of variable Sum Q: Do we really need a constraint on the Selection AND a constraint on the Action AND a constraint on the Input? What if we left out one? A: Yes, each is necessary. Without a constraint on the Selection, the rule will accept the student step if the number typed is correct, regardless of the column in which it was typed. Without a constraint on the Action, the the rule will accept the step if the right number was typed in the right column, regardless of whether it was in the carry spot or at the bottom of the column. Etc.

36 Example of Interface Constraints in English WRITE-SUM IF There is a goal to write Sum as the result in column C And Sum < 10 And the carry into column C (if any) has been added THEN Write Sum as the result in column C And remove the goal CONSTRAINTS—student step matches only if: Selection is: Action is: Input is: Same as value of variable C Write-Result Equal to the value of variable Sum

37 More Detail on Interface Constraints In Tutor Mode –When evaluating student input TDK will not consider productions whose constraints are violated. –When responding to a help request Tries to find the production path with the least severe constraint violation. – from least to most severe: input, action, selection. In Stand-Alone Mode, interface constraints are ignored Convenient for development—can run model without having to supply input.

TDK Notation for Interface Constraints (defproduction write-sum addition (=problem) =problem> isa addition-problem... =subgoal> isa write-sum-goal... ==> =column> result =sum =problem> subgoals ($sg1 $sg2) ; the remaining subgoals :nth-selection 0 =column :action 'write-result :input =sum #'equal-value-p) See Example 4 from the Examples menu. The first item of Selections must be equal to the value of =column. The Action must be write-result. The value of Input must be equal to the value of =sum. Use function equal-value-p to compare.

39 Syntax of TDK Notation for Interface Constraints :nth-selection n value [test-fn] :action value [test-fn] :input value [test-fn]

40 Overview Basics of TDK Notation Matching: $ and = variables Modeling alternative strategies “Interface constraints” for model tracing Control flow and production chains Writing hints and bug productions Creating a curriculum

41 Creating a Control Flow Diagram Control Flow Diagram elements: –Boxes are goals or actions –Arrows are production rules –A “path” or “chain” through the diagram starts at a goal and ends with an action Crucial idea: –Each chain should produce exactly one action –When you “cycle” the TDK production system a “chain” of productions fire

42 Production Rules Set New Goals & Perform Actions Goal: Solve the addition problem ADD-CARRY FOCUS-ON-FIRST-COLUMN, FOCUS-ON-NEXT-COLUMN Goal: Process column C Goal: Write carry in next column MUST-CARRY WRITE-SUM Action: Write the sum Action: Write the carry WRITE-CARRY Goal: Write sum in column C ADD-ADDENDS

43 (defproduction focus-on-next-column multi-column-addition (=problem) =problem> isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> isa column - result NIL =next-column> isa column result NIL first-addend =num1 second-addend =num2 !eval! (or =num1 =num2) ==> =process-column-goal> isa process-column-goal column =next-column !chain! multi-column-addition (=process-column-goal)) Chaining of Production Rules —Multiple Productions per Step (defproduction write-result multi-column-addition (=subgoal) =subgoal> isa process-column-goal column =column =column> isa column first-addend =num1 second-addend =num2 ==> !eval! =sum (+ (or =num1 0) (or =num2 0)) =column> result =sum) Parameters !chain! specifies the production set and parameters and initiates another attempt to match productions to this parameter & WM. Name of the production rule set No !chain! so this rule ends the chain.

44 Important Design Principle Design your production rule model so that there is exactly one cycle per observable (student) action.

45 Overview Basics of TDK Notation Matching: $ and = variables Modeling alternative strategies “Interface constraints” for model tracing Control flow and production chains Writing hints and bug productions Creating a curriculum

46 Example Hint Sequences for Addition Tutor Start with the column on the right. This is the "ones" column. You need to add the two digits in this column. Adding 4 and 6 gives 10. The sum that you have, 10, is greater than 9. You need to 10 of the 10 into the second column. And you need to write the rest of the 10 at the bottom of the first column. Write 0 at the bottom of the rightmost column. You need to complete the work on the first column. Write the carry from the first to the next column. Write 1 at the top of the second column from the right. Now move on to the second column from the right. This is the "tens" column. You need to add the two digits in this column. Adding 6 and 1 gives 7. There is a carry into this column, so you need to add the value carried in. This gives equals 8. Write 8 at the bottom of the second column from the right

47 Design of Hint Messages –What makes a good hint sequence? Short Clear (minimum of jargon, short sentences) Tells student what to do But also why/when that action is appropriate Goes from the general to the specific –Writing hints can help in designing the production rule model. –Hint sequences often combine hints from multiple productions in a chain.

48 (defproduction focus-on-next-column addition (=problem) =problem> isa addition-problem... =previous-column> isa column... name =name position =pos !eval! (or =num1 =num2 =carry) ==>... !chain! addition (=problem) :messages (help `(Now move on to the #\space,=pos column from the right #\. ~n ~n This is the #\space,=name column #\.))) Example Hint Template Attach list of hint templates to the rule.

49 Example hints 1. Now move on the the second column from the right. This is the “tenths” column. 2. Next big event: the second column from the right!!! Each template is a Lisp form that evaluates to a list. :messages (help `(Now move on to the #\space,=pos column from the right #\. ~n ~n This is the #\space,=name column #\.) `(Next big event #\: the #\space,=pos column from the right #\! #\! #\! ) ) 2nd Example Hint Template

50 Example hints 1. Now move on the the second column from the right. This is the “tenths” column. 2. Next big event: the second column from the right!!! Each template is a Lisp form that evaluates to a list. :messages (help `(Now move on to the #\space,=pos column from the right #\. ~n ~n This is the #\space,=name column #\.) `(Next big event #\: the #\space,=pos column from the right #\! #\! #\! ) ) 2nd Example Hint Template

51 Example hints 1. Now move on the the second column from the right. This is the “tenths” column. 2. Next big event: the second column from the right!!! Each template is a Lisp form that evaluates to a list. :messages (help `(Now move on to the #\space,=pos column from the right #\. ~n ~n This is the #\space,=name column #\.) `(Next big event #\: the #\space,=pos column from the right #\! #\! #\! ) ) 2nd Example Hint Template

52 Example hints from this template 1. Now move on the the second column from the right. This is the “tenths” column. 2. Next big event: the second column from the right!!! Each template is a Lisp form that evaluates to a list. :messages (help `(Now move on to the #\space,=pos column from the right #\. ~n ~n This is the #\space,=name column #\.) `(Next big event #\: the #\space,=pos column from the right #\! #\! #\! ) ) 2nd Example Hint Template For punctuation and spaces, use #\ notation. To insert value of variable: comma followed by variable name. ~n produces new line. Backquote. When inserting a string, put a space in front of it.

53 Sequences Combining Hints from Multiple Productions Start with the column on the right. This is the "ones" column. You need to add the two digits in this column. Adding 4 and 6 gives 10. The sum that you have, 10, is greater than 9. You need to move 10 of the 10 into the second column. And you need to write the rest of the 10 at the bottom of the first column. Write 0 at the bottom of the rightmost column. You need to complete the work on the first column. Write the carry from the first to the next column. Write 1 at the top of the second column from the right. Now move on to the second column from the right. This is the "tens" column. You need to add the two digits in this column. Adding 6 and 1 gives 7. There is a carry into this column, so you need to add the value carried in. This gives equals 8. Write 8 at the bottom of the second column from the right FOCUS-ON-FIRST- COLUMN ADD-ADDENDS MUST-CARRY WRITE-SUM WRITE-CARRY FOCUS-ON-NEXT- COLUMN ADD-ADDENDS ADD-CARRY WRITE-SUM

54 How could you model students who don’t carry? Instead of doing the addition correctly: Can you model a student who writes: Can you write a “bug rule” that models this error?

55 Production Rules Set New Goals & Perform Actions Goal: Solve the addition problem ADD-CARRY FOCUS-ON-FIRST-COLUMN, FOCUS-ON-NEXT-COLUMN Goal: Process column C Goal: Write carry in next column MUST-CARRY WRITE-SUM Action: Write the sum Action: Write the carry WRITE-CARRY Goal: Write sum in column C ADD-ADDENDS

56 How to come up with a bug rule? Intuition: Need to model the “forgetting” of the subgoal to carry into the next column. Normal processing: If you then You will pretty soon be in situation: Subgoal 1: write-sum-goal write 0 as result in the first column. Subgoal 2: write-carry-goal write 1 as carry in second column Forget Subgoal 2, and continue as if nothing has happened

Solution 1: Explicit Forgetting Rule ;; BUG-FORGOT-WRITE-CARRY-GOAL ;; IF The goal is to do an addition problem ;; And there is a pending subgoal to write a carry ;; THEN Delete that pending subgoal (w/o carrying it out). ;; (And carry on with the addition problem in the usual manner.) (defproduction-bug BUG-FORGOT-WRITE-CARRY-GOAL addition (=problem) =problem> isa addition-problem subgoals ($sg1 =write-carry-goal $sg2) =write-carry-goal> isa write-carry-goal ==> =problem> subgoals ($sg1 $sg2) ; Remove the write-carry-goal. !chain! addition (=problem) :messages (bug `(First #\, you need to write the carry from the previous column #\. ) )) Use defproduction-bug instead of defproduction. Use :messages (bug … ) instead of :messages (help … )

;; BUG-FORGOT-WRITE-CARRY-GOAL ;; IF The goal is to do an addition problem ;; And there is a pending subgoal to write a carry ;; THEN Delete that pending subgoal (w/o carrying it out). ;; (And carry on with the addition problem in the ;; usual manner.) (defproduction-bug BUG-FORGOT-WRITE-CARRY-GOAL addition (=problem) =problem> isa addition-problem subgoals ($sg1 =write-carry-goal $sg2) =write-carry-goal> isa write-carry-goal ==> =problem> subgoals ($sg1 $sg2) !chain! addition (=problem) :messages (bug `(First #\, you need to write the carry from the previous column #\. ) )) Solution 1: Explicit Forgetting Rule Some questions & details Why does the rule chain? Why does the rule fire only when the student forgets to add a carry, and not for any wrong answer? Does this work when action “done” follows after 97+42=39? Does this rule fire under different circumstances than expected? YES - after MUST-CARRY in the same cycle. Not intended but not a problem. Some questions & details Why does the rule chain? Why does the rule fire only when the student forgets to add a carry, and not for any wrong answer? Does this work when action “done” follows after 97+42=39? Does this rule fire under different circumstances than expected? YES - after MUST-CARRY in the same cycle. Not intended but not a problem.

59 Overview Basics of TDK Notation Matching: $ and = variables Modeling alternative strategies “Interface constraints” for model tracing Control flow and production chains Writing hints and bug productions Creating a curriculum

60 Setting Up a Curriculum Organize curriculum early on during the development of your tutoring system. Organize the tutor curriculum –Sequencing of skills –Break down curriculum into lessons and sections –Each section involves a handful of “skills” –Assign problem sets to each section

61 Setting Up Knowledge Tracing and Mastery Learning Indicate how to group the crucial productions into traced skills. –Each traced skill corresponds to one or more productions. For each skill, provide values for the four knowledge tracing parameters. For each problem, indicate which productions are needed to solve it. –Enables tutor to select problems that involve un-mastered skills.

Example: Addition Skills Start — Start in the rightmost column FOCUS-ON-FIRST-COLUMN Add — Add the addends in a column ADD-ADDENDS Write — Write the sum in a column WRITE-SUM ADDITION Done — Determine that all columns are done DONE ADDITION Next — Focus on the next column FOCUS-ON-NEXT-COLUMN Regroup — Realize that 10 goes to next column MUST-CARRY Write Carry — Write the carry WRITE-CARRY Add Carry — Add the carry to the sum ADD-CARRY ADDITION

63 Example from Addition Tutor Structure curriculum, based on skills Section1 — one column, no carry new skills: Start, Add, Write, Done Section2 — two columns, no carry new skills: Next Section3 — two columns, possible carry new skills: Regroup, Mark Carry, Add Carry Section4 — three columns, possible carry new skills: -

64 Excerpt from Lesson File (Make-Section :Required-Problems '(Problem11 Problem12 Problem15) :Remedial-Problems '(Problem13 Problem14 Problem16 Problem17 Problem18 Problem19) :Skills '(((Focus-On-First-Column Addition) :Description "Start") ((Done Addition) :Description "Done") ((Add-Addends Addition) :Description "Add") ((Write-Sum Addition ) :Description "Write") ))

65 Set Knowledge-Tracing Parameters Parameter (setf *p-initial* 0.33) (setf *p-learn* 0.20) (setf *p-guess* 0.20) (setf *p-slip* 0.10) (setf *p-known-p* 0.95) Represents probability that the student has learned the skill prior to the instruction the student learns the skill when successfully applying it the student gets the answer right even though the skill is not mastered the student gets the answer wrong even though the skill is mastered Represents mastery threshold

66 Provide Production Counts For problem (or any other 2-column problem without carrying) (set-problem-production-counts problem26+51 '(((FOCUS-ON-FIRST-COLUMN ADDITION). 1) ((FOCUS-ON-NEXT-COLUMN ADDITION). 1) ((ADD-ADDENDS ADDITION). 2) ((WRITE-SUM ADDITION). 2) ((DONE ADDITION). 1) ))))

67 Collecting Logs of Student-Tutor Interactions in TDK Create “protocol collectors” when starting tutor (make-instance 'global protocol-collector :broadcast-station broadcast-station) (make-instance 'protocol-collector :broadcast-station broadcast-station) ; add to function ; make-addition-tutor Set directory parameters (setf *global-protocol-directory* "Home:Globals;") (setf *protocol-directory* "Home:Protocols;") Flip the switch (setf *collect-protocols* t)

Example Protocol from Addition Tutor *000:00:01 START STUDENT; "Student3", USER-DATA; NIL, PROBLEM; PROBLEM15, NPRESENTED; (0 0), START-TIME; "07/18/01 21:57", FILE-NAME; "Macintosh HD:Desktop Folder:Vincent:CIRCLE Summer School:TDK:Protocols:LESSON1:Student3:1%PR OBLEM15%0.p-0", LESSON; LESSON1, LESSON-PRINT-NAME; "", SECTION; SECTION155, SECTION-PRINT-NAME; "",. *000:00:24 INITIAL-PROBABILITIES p/p-1; ( (ADD-ADDENDS ADDITION)), p/p-2; ( (DONE ADDITION)), p/p-3; ( (FOCUS-ON-FIRST- COLUMN ADDITION)), p/p-4; ( (WRITE-SUM ADDITION)),. *000:00:40 FOCUS SELECTED-WMES; (COLUMN1),. *000:14:926 APPLY-ACTION WINDOW; ADDITION-WINDOW, CONTEXT; ROOT, SELECTIONS; (COLUMN1), ACTION; WRITE-RESULT, INPUT; "5",. *000:14:939 UPDATE-P-KNOW META; NIL, PRODUCTION; (FOCUS-ON-FIRST-COLUMN ADDITION), SUCCESS?; NIL, P-KNOW; ,. *000:14:945 UPDATE-P-KNOW META; NIL, PRODUCTION; (ADD-ADDENDS ADDITION), SUCCESS?; NIL, P-KNOW; ,. *000:14:952 UPDATE-P-KNOW META; NIL, PRODUCTION; (WRITE-SUM ADDITION), SUCCESS?; NIL, P-KNOW; ,. *000:14:958 NO-CONFLICT-SET. *000:14:962 READY. *000:17:41 SUGGESTED-SELECTION ITEMS; (WRITE-RESULT COLUMN1),. *000:17:46 HELP SELECTION; (COLUMN1), NHINT; 0,. #000:17:50 HELP-TEXT You need to add the two digits in this column. Adding 3 and 3 gives 6.. *000:17:53 HELP-PATH P-1; (FOCUS-ON-FIRST-COLUMN ADDITION), P-2; (ADD-ADDENDS ADDITION), P-3; (WRITE-SUM ADDITION),. *000:17:68 READY. *000:19:246 SUGGESTED-SELECTION ITEMS; (WRITE-RESULT COLUMN1),. *000:19:252 HELP SELECTION; (COLUMN1), NHINT; 1,. #000:19:256 HELP-TEXT Write 6 at the bottom of the rightmost column.. *000:19:259 HELP-PATH P-1; (FOCUS-ON-FIRST-COLUMN ADDITION), P-2; (ADD-ADDENDS ADDITION), P-3; (WRITE-SUM ADDITION),. *000:19:270 READY. *000:21:94 WINDOW-SELECT WINDOW-TITLE; ADDITION-WINDOW,. *000:23:765 APPLY-ACTION WINDOW; ADDITION-WINDOW, CONTEXT; ROOT, SELECTIONS; (COLUMN1), ACTION; WRITE-RESULT, INPUT; "6",. *000:23:774 GOOD-PATH. *000:23:778 HISTORY P-1; (FOCUS-ON-FIRST-COLUMN ADDITION), P-2; (ADD-ADDENDS ADDITION), P-3; (WRITE-SUM ADDITION),. *000:23:782 READY. *000:25:84 APPLY-ACTION WINDOW; ADDITION-WINDOW, CONTEXT; ROOT, SELECTIONS; NIL, ACTION; DONE, INPUT; NIL,. *000:25:94 UPDATE-P-KNOW META; NIL, PRODUCTION; (DONE ADDITION), SUCCESS?; T, P-KNOW; ,. *000:25:99 GOOD-PATH. *000:25:102 HISTORY P-1; (DONE ADDITION),. *000:25:106 READY. *000:25:109 PROBLEM-COMPLETED. *000:25:113 FINAL-PROBABILITIES p/p-1; ( (ADD-ADDENDS ADDITION)), p/p-2; ( (DONE ADDITION)), p/p-3; ( (FOCUS-ON-FIRST- COLUMN ADDITION)), p/p-4; ( (WRITE-SUM ADDITION)),.

69 Example Uses of Log Data To improve model and tutor –Is skill decomposition accurate? If “learning curves” for skills do not follow power law, crucial difficulty factors may have been overlooked. –Tune knowledge tracing parameters Fit the parameters to the log data, use the new parameter values next time around. –What kind of errors do students make? Any use for new bug messages? –Do we have enough different problems? For research –Relation between help use and learning outcomes. –Compare learning rates in comparative studies.

70 END