Constraint-based tutoring CPI 494 Feb 17, 2009 Kurt VanLehn ASU

Outline SQL-Tutor (previous class) ER-Tutor & Normit (today) Evaluation & Dissemination (today)

Framework for comparing step-based tutoring systems Feature SQL tutor Cognitive Step granularity & user interface Goal boxes; large & fixed granularity Goal boxes, granularity varies Interpreting a student’s step Ideal solution + constraints; String match Rules that gen. solution; Exact, intervals, regular expressions, etc. Suggesting good steps Next blank step Next step in order Feedback and hints on steps Few bugs; selectable hint from sequence Few bugs; hint sequence Task selection Some student choice Mastery learning Assessment Clauses? Constraints? Knowledge tracing Evaluations Dissemination

Let’s add two other Constraint-based tutors Feature NORMIT ER-Tutor SQL tutor Cognitive Step granularity & user interface Goal boxes; large & fixed granularity Goal boxes, granularity varies Interpreting a student’s step Ideal solution + constraints; String match Rules that gen. solution; Exact, intervals, regular expressions, etc. Suggesting good steps Next blank step Next step in order Feedback and hints on steps Few bugs; selectable hint from sequence Few bugs; hint sequence Task selection Some student choice Mastery learning Assessment Clauses? Constraints? Knowledge tracing Evaluations Dissemination

Steps? Looks like one goal box per page Sequence of pages is fixed: “…determine candidate keys, the closure of a set of attributes, prime attributes, simplify functional dependencies, determine normal forms, and, if necessary, decompose the table. The sequence is fixed: the student will only see a Web page corresponding to the current task.”

“self-explanation” of steps The first time a step is done, requires selecting an explanation from a menu. On subsequent occurrences, explanation is required only if the step is incorrect If explanation is incorrect, asked to select correct definition of a concept Menu is stored on constraints

Normit Feature NORMIT ER-Tutor SQL tutor Cognitive Step granularity & user interface Goal boxes + explanations Goal boxes; large & fixed granularity Goal boxes, granularity varies Interpreting a student’s step Generates solution + constraints; exact match Ideal solution + constraints; String match Rules that gen. solution; Exact, intervals, regular expressions, etc. Suggesting good steps Next step in strict sequence Next blank step Next step in order Feedback and hints on steps On demand On demand; Few bugs; selectable hint from sequence Immdiate: Few bugs; hint sequence Task selection ? Some student choice Mastery learning Assessment Clauses? Constraints? Knowledge tracing Evaluations Dissemination

ER-Tutor Whenever a new object is createed, student highlights text in problem statement Ideal solution

Normit Feature NORMIT ER-Tutor SQL tutor Cognitive Step granularity & user interface Goal boxes + explanations Canvas, but nouns act as goals Goal boxes; large & fixed granularity Goal boxes, granularity varies Interpreting a student’s step Generates solution + constraints; exact match Ideal solution + constraints; exact match on text, node type, links Ideal solution + constraints; String match Rules that gen. solution; Exact, intervals, regular expressions, etc. Suggesting good steps Next step in strict sequence ? Next blank step Next step in order Feedback and hints on steps On demand On demand; Few bugs; selectable hint from sequence Immdiate: Few bugs; hint sequence Task selection Some student choice Mastery learning Assessment Clauses? Constraints? Knowledge tracing Evaluations Dissemination

Outline SQL-Tutor (previous class) ER-Tutor & Normit (today) Evaluation & Dissemination (today)

Evaluation vs. Assessment “Evaluation” used for determining effectiveness of instruction “Assessment” used for determining competence of a student

Classic evaluation design Experimental instruction Baseline/control/comparison instruction Pre-training (e.g., read short textbook) Pre-training (ditto) Pre-test Training (e.g., Use SQL-tutor) Training (e.g., use paper & pencil) Post-test

Three types Control number of problems solved Control training time Time to complete training varies Score on post-tests varies Control training time Problems completed varies Control post-test score by repeating training until mastery Time to mastery varies Problems solved varies

Classic stats If pre-test scores not significantly different, then can use T-test on post-test Better: Gain_score = Post-test_score – pre-test_score T-test on gain_score: Most say “don’t use” ANCOVA on post-test, with pre-test as covariate

Effect size Cohen’s d: d=1.0 is considered excellent; one letter grade d = mean(gain_score(expt)) – mean(gain_score(control)) / standard_deviation(gain_score(control)) d=1.0 is considered excellent; one letter grade d=0.5 is good d=0.2 is questionable unless classroom/field

Evaluation of SQL-tutor Participation voluntary 20 of 49 in course volunteered 2 hours for training No pre-test, no post-test; course exam used as post-test; only questions relevant to SQLT counted T-test on means: p=0.006 Effect size: d=0.66 self-selection bias

Classroom evaluation of Lisp Cognitive Tutor Tutor vs. Lisp standard environment Random assignment Whole minicourse Final exam; no pretest T-test on means: Highly significant Effect size: d=1.0

Lab evaluation of Lisp Tutor Same exercises, same text; tutor vs. Lisp alone Control students did not successfully complete all exercises

Evaluation of Geometry Cognitive Tutor Tutor vs. baseline Whole classes Same teacher Whole minicourse Final exam; used prior year’s grade as covariate Contribution of tutor to regression was reliable Effect size: d= “more than 1.0”

Second evaluation of Geometry Tutor 4 conditions Tutor + project teacher (best) Tutor + nonproject teacher Paper + project teacher Paper + nonproject teacher First condition better than all others, which tied Effect size: d = 1.0

Evaluation of first Algebra Tutor Tutor vs. baseline Whole classes No significant differences