Constraint-based tutoring CPI 494 Feb 17, 2009 Kurt VanLehn ASU

Outline SQL-Tutor (today) ER-Tutor & Normit (next class) Technical details (next class) Evaluation & Dissemination (next class)

Framework for comparing step-based tutoring systems FeatureConstraint basedCognitive Step granularity & user interface Interpreting a student’s step Suggesting good steps Feedback and hints on steps Task selection Assessment Evaluations Dissemination

In class-exercises Install SQL tutor Try at least 3 problems Check our your student model periodically Try different levels of help Try different ways of expressing the same step Try different ordering of steps What steps does it suggest? Try to game it Try fancy ways to say simple things: E.g., instead of referring to Stanley Kubrick by name, retrieve the director of “Clockwork Orange”

Step granularity & user interface Type of user interface? Canvas, forms (i.e., goal boxes), text, simulation based? Learnability? Every problem the same UI? In general, what are the steps? The Answer? On the problem “Display the titles of movies that are comedies or dramas”, what are the steps? – Would a finer granularity make sense? For who?

Framework for comparing step-based tutoring systems FeatureConstraint basedCognitive Step granularity & user interfaceLarge, fixed-size goal boxesGoal boxes, granularity varies Interpreting a student’s step Suggesting good steps Feedback and hints on steps Task selection Assessment Evaluations Dissemination

Interpreting student steps: How does tutor know what is correct? Is it example-based or knowledge-based or both? “Constraint-based modeling (CBM) is…not interested in the exact sequence of states in the problem space…” “CBM does not require an executable domain model, and is applicable in situations in which such a model would be difficult to construct.” “CBM eliminates the need for bug libraries”

A syntactic constraint Msg: "The SELECT clause is a mandatory one. Specify the attributes/expressions to retrieve from the database.“ Clause applied to: SELECT Relevance condition: t Satisfaction condition: (not (null (select-clause ss)))

A semantic constraint Msg: "Check whether you should have ascending or descending order!“ Clause: “ORDER BY“ Relevance: (and (not (null (order-by is))) (not (null (order-by ss))) (bind-all ’?n (names (order-by ss)) bindings) (match ’(?*d1 ?n "DESC" ?*d2) (order-by ss) bindings) (not (qualified-name ’?n))) Satisfaction: (match ’(?*d3 ?n "DESC" ?*d4) (order-by is) bindings) Comment: “verifies that if the results of a query are sorted in decreasing order in the ideal solution (is), the the student’s solution (ss) uses the same ordering.”

Interpreting a student’s step: Details of the matching How is an individual step interpreted? Spaces? Capitalization? Ordering of conjuncts and disjuncts? Ordering of steps? Fancy ways to say simple things? E.g., instead of referring to Stanley Kubrick by name, retrieve the director of “Clockwork Orange”

Framework for comparing step-based tutoring systems FeatureConstraint basedCognitive Step loop: Granularity & user interfaceLarge, fixed-size goal boxes Goal boxes, granularity varies Step loop: Interpreting student stepsConstraints + ideal solution; CB matching EB or KB; Exact, intervals, regular expressions Step loop: Suggesting good steps Step loop: Feedback and hints Task selection Assessment Evaluations Dissemination

Suggesting good steps What does its method/ordering appear to be? What if there are multiple correct solutions?

Framework for comparing step-based tutoring systems FeatureConstraint basedCognitive Step granularity & user interfaceLarge, fixed-size goal boxes Goal boxes, granularity varies Interpreting a student’s stepString matchExact, intervals, regular expressions Suggesting good stepsNext blank stepNext step in order Feedback and hints on steps Task selection Assessment Evaluations Dissemination

Feedback and hints What are the hint levels? How can one game? Does it detect cut-and-paste? Does it detect common mistakes e.g., quotes, “WHERE” as first word of WHERE clause… Are the hints pedagogical?

Framework for comparing step-based tutoring systems FeatureConstraint basedCognitive Step granularity & user interfaceLarge, fixed-size goal boxes Goal boxes, granularity varies Interpreting a student’s stepString matchExact, intervals, regular expressions Suggesting good stepsNext blank stepNext step in order Feedback and hints on stepsFew bugs; selectable hint from sequence Few bugs; hint sequence Task selection Assessment Evaluations Dissemination

Task selection What happens if you select next task in order? Anyone graduate from novice to higher? Do clauses make a good index for student task selection? Does student have enough info to do task selection?

Framework for comparing step-based tutoring systems FeatureConstraint basedCognitive Step granularity & user interfaceLarge, fixed-size goal boxes Goal boxes, granularity varies Interpreting a student’s stepString matchExact, intervals, regular expressions Suggesting good stepsNext blank stepNext step in order Feedback and hints on stepsFew bugs; selectable hint from sequence Few bugs; hint sequence Task selectionSome student choiceMastery learning Assessment Evaluations Dissemination

Assessment What are the KCs? Clauses or constraints? – Displayed? – Learning curves? – How many? Are the colored sections of the bars informative? How updated? Is history window useful?

Framework for comparing step-based tutoring systems FeatureConstraint basedCognitive Step granularity & user interfaceLarge, fixed-size goal boxes Goal boxes, granularity varies Interpreting a student’s stepString matchExact, intervals, regular expressions Suggesting good stepsNext blank stepNext step in order Feedback and hints on stepsFew bugs; selectable hint from sequence Few bugs; hint sequence Task selectionSome student choiceMastery learning Assessment?Knowledge tracing Evaluations Dissemination

Outline SQL-Tutor (today) ER-Tutor & Normit (next class) Technical details (next class) Evaluation & Dissemination (next class)