How does model tracing work? CPI 494 Feb 10 Kurt VanLehn

Example tracing Tutor’s domain knowledge = – an explicit representation of the prescribed solution paths to the problem – Called “example” and “behavior graph” To check if a student’s step is correct, – Tutor searches graph for matching step (caveats) When asked what step to hint, – Tutor picks the next step along the (which?) path

Model tracing Tutor domain knowledge = – A procedure for solving problems – Called a “cognitive model” or “Jess production rules” To check if a student’s step is correct – Tutor searches (how?) for a match among the steps that it would generate next When asked what step to hint, – Tutor picks one of the steps that it would generate next

Key differences Example tracing tutor has all correct steps generated and stored as data Model tracing tutor can generate such steps dynamically as needed If example has branches (multiple solution paths, orders), then model must be a non- deterministic procedure

A simple example of a non- deterministic procedure Do in order: – Sit down. – Do both in either order: Don left shoe. Don right shoe. – Do both in either order: Tie left shoe Tie right shoe. – Stand up.

Using production (if-then) rules to represent non-deterministic procedure If not sitting, then sit down. If sitting and left foot exposed, then don left shoe. If sitting and right foot exposed, then don right shoe. If left shoe on foot and untied, then tie left shoe. If right shoe is on foot and untied, then tie right shoe. If left shoe tied and right shoe tied, then stand up

Production rules can have variables that are “bound” when condition matches If not sitting, then sit down. If sitting and ?X foot exposed, then don ?X shoe. If ?X shoe on foot and untied, then tie ?X shoe. If left shoe tied and right shoe tied, then stand up

Can have fancy conditions If not sitting, then sit down. If sitting and ?X foot exposed, then don ?X shoe. If ?X shoe on foot and untied, then tie ?X shoe. If not (there exist an exposed foot ?foot), and not(there exists an untied shoe ?shoe), then stand up

Why prefer production rules over other non- deterministic programming languages? If not sitting, then sit down. If sitting and ?X foot exposed, then don ?X shoe. If ?X shoe on foot and untied, then tie ?X shoe. If not (there exist an exposed foot ?foot), and not(there exists an untied shoe ?shoe), then stand up Each rule can be learned independently Conditions express when the rule applies, so may apply even outside this procedure Can write hints for a rule without referring to other rules If same rule appears in multiple procedures, can count number of times applied & hence measure competence

What do conditions match against? Working memory, which is a set of facts – E.g., left_shoe_on A fact can represent – Something on the student’s screen E.g., numerator of first given fraction is 4 – Goals (intentions: specific to this problem, this time) E.g., I will convert the first given fraction to have 12 as the denominator – Temporary, mentally held info E.g., the least common denominator is 12

What happens during execution WM (working memory) starts with representation of student’s initial screen Loop until Exit action executed: – Collect all rules whose conditions match in WM – called the conflict set – Pick one member of the conflict set – Execute it: With variables bound by the condition, substitute their values into the action side, do the actions thus modifying WM

Example of execution 1 WM “Left shoe untied” “Left foot exposed” “Right shoe untied” “Right foot exposed” Rules that match (Conflict set) : – 1 1.If not(“sitting”), then add “sitting”. 2.If “sitting” and “?X foot exposed”, then remove “?X foot exposed” & add “?X shoe on foot”. 3.If “?X shoe on foot” and “?X shoe untied”, then remove “?X shoe untied” 4.If not (“?X foot exposed”), and not(“?X shoe untied”), then remove “sitting”

Example of execution 2 WM “Left shoe untied” “Left foot exposed” “Right shoe untied” “Right foot exposed” “sitting” Rules that match: – 2 with ?X=left – 2 with ?X=right 1.If not(“sitting”), then add “sitting”. 2.If “sitting” and “?X foot exposed”, then remove “?X foot exposed” & add “?X shoe on foot”. 3.If “?X shoe on foot” and “?X shoe untied”, then remove “?X shoe untied” 4.If not (“?X foot exposed”), and not(“?X shoe untied”), then remove “sitting”

Example of execution 3 WM “Left shoe untied” “Right shoe untied” “Right foot exposed” “sitting” “left shoe on foot” Rules that match: – 2 with ?X=right – 3 with ?X=left 1.If not(“sitting”), then add “sitting”. 2.If “sitting” and “?X foot exposed”, then remove “?X foot exposed” & add “?X shoe on foot”. 3.If “?X shoe on foot” and “?X shoe untied”, then remove “?X shoe untied” 4.If not (“?X foot exposed”), and not(“?X shoe untied”), then remove “sitting”

Example of execution 4 WM “Left shoe untied” “Right shoe untied” “sitting” “left shoe on foot” “right shoe on foot” Rules that match: – 3 with ?X=left – 3 with ?X=right 1.If not(“sitting”), then add “sitting”. 2.If “sitting” and “?X foot exposed”, then remove “?X foot exposed” & add “?X shoe on foot”. 3.If “?X shoe on foot” and “?X shoe untied”, then remove “?X shoe untied” 4.If not (“?X foot exposed”), and not(“?X shoe untied”), then remove “sitting”

Example of execution 5 WM “Right shoe untied” “sitting” “left shoe on foot” “right shoe on foot” Rules that match: – 3 with ?X=right 1.If not(“sitting”), then add “sitting”. 2.If “sitting” and “?X foot exposed”, then remove “?X foot exposed” & add “?X shoe on foot”. 3.If “?X shoe on foot” and “?X shoe untied”, then remove “?X shoe untied” 4.If not (“?X foot exposed”), and not(“?X shoe untied”), then remove “sitting”

Example of execution 6 WM “sitting” “left shoe on foot” “right shoe on foot” Rules that match: – 4 1.If not(“sitting”), then add “sitting”. 2.If “sitting” and “?X foot exposed”, then remove “?X foot exposed” & add “?X shoe on foot”. 3.If “?X shoe on foot” and “?X shoe untied”, then remove “?X shoe untied” 4.If not (“?X foot exposed”), and not(“?X shoe untied”), then remove “sitting”

Example of execution 7 WM “left shoe on foot” “right shoe on foot” Rules that match: – 1 (opps) 1.If not(“sitting”), then add “sitting”. 2.If “sitting” and “?X foot exposed”, then remove “?X foot exposed” & add “?X shoe on foot”. 3.If “?X shoe on foot” and “?X shoe untied”, then remove “?X shoe untied” 4.If not (“?X foot exposed”), and not(“?X shoe untied”), then remove “sitting”

It like ordinary programming… Except that must check that all solution paths are desired Except that can see WM facts

How to match student’s step? Pre-generate the examples, then use example tracing At any given time, see if any of the rules that can fire produce an action that matches the student’s step – Complexity: Can chain through invisible facts such as goals and temporary info. Must keep student on a correct solution path

How to select a correct, pedagogically appropriate next step if student asks for a hint? Sequence or prioritize rules so that first one in the conflict set is best Store hints on that rule Give first hint of the sequence – Can substute in the values of ?X variables