1. Constraint-based tutoring
CPI 494 Feb 17, 2009
Kurt VanLehn
ASU

2. Outline SQL-Tutor (previous class) ER-Tutor & Normit (today)
Evaluation & Dissemination (today)

3. Framework for comparing step-based tutoring systems
Feature
SQL tutor
Cognitive
Step granularity & user interface
Goal boxes; large & fixed granularity
Goal boxes, granularity varies
Interpreting a student’s step
Ideal solution + constraints; String match
Rules that gen. solution; Exact, intervals, regular expressions, etc.
Suggesting good steps
Next blank step
Next step in order
Feedback and hints on steps
Few bugs; selectable hint from sequence
Few bugs; hint sequence
Task selection
Some student choice
Mastery learning
Assessment
Clauses? Constraints?
Knowledge tracing
Evaluations
Dissemination

4. Let’s add two other Constraint-based tutors
Feature
NORMIT
ER-Tutor
SQL tutor
Cognitive
Step granularity & user interface
Goal boxes; large & fixed granularity
Goal boxes, granularity varies
Interpreting a student’s step
Ideal solution + constraints; String match
Rules that gen. solution; Exact, intervals, regular expressions, etc.
Suggesting good steps
Next blank step
Next step in order
Feedback and hints on steps
Few bugs; selectable hint from sequence
Few bugs; hint sequence
Task selection
Some student choice
Mastery learning
Assessment
Clauses? Constraints?
Knowledge tracing
Evaluations
Dissemination

6. Steps? Looks like one goal box per page Sequence of pages is fixed:
“…determine candidate keys, the closure of a set of attributes, prime attributes, simplify functional dependencies, determine normal forms, and, if necessary, decompose the table. The sequence is fixed: the student will only see a Web page corresponding to the current task.”

8. “self-explanation” of steps
The first time a step is done, requires selecting an explanation from a menu.
On subsequent occurrences, explanation is required only if the step is incorrect
If explanation is incorrect, asked to select correct definition of a concept
Menu is stored on constraints

9. Normit Feature NORMIT ER-Tutor SQL tutor Cognitive
Step granularity & user interface
Goal boxes + explanations
Goal boxes; large & fixed granularity
Goal boxes, granularity varies
Interpreting a student’s step
Generates solution + constraints; exact match
Ideal solution + constraints; String match
Rules that gen. solution; Exact, intervals, regular expressions, etc.
Suggesting good steps
Next step in strict sequence
Next blank step
Next step in order
Feedback and hints on steps
On demand
On demand; Few bugs; selectable hint from sequence
Immdiate: Few bugs; hint sequence
Task selection ?
Some student choice
Mastery learning
Assessment
Clauses? Constraints?
Knowledge tracing
Evaluations
Dissemination

11. ER-Tutor
Whenever a new object is createed, student highlights text in problem statement
Ideal solution

12. Normit Feature NORMIT ER-Tutor SQL tutor Cognitive
Step granularity & user interface
Goal boxes + explanations
Canvas, but nouns act as goals
Goal boxes; large & fixed granularity
Goal boxes, granularity varies
Interpreting a student’s step
Generates solution + constraints; exact match
Ideal solution + constraints; exact match on text, node type, links
Ideal solution + constraints; String match
Rules that gen. solution; Exact, intervals, regular expressions, etc.
Suggesting good steps
Next step in strict sequence ?
Next blank step
Next step in order
Feedback and hints on steps
On demand
On demand; Few bugs; selectable hint from sequence
Immdiate: Few bugs; hint sequence
Task selection
Some student choice
Mastery learning
Assessment
Clauses? Constraints?
Knowledge tracing
Evaluations
Dissemination

13. Outline SQL-Tutor (previous class) ER-Tutor & Normit (today)
Evaluation & Dissemination (today)

14. Evaluation vs. Assessment
“Evaluation” used for determining effectiveness of instruction
“Assessment” used for determining competence of a student

15. Classic evaluation design
Experimental instruction
Baseline/control/comparison instruction
Pre-training (e.g., read short textbook)
Pre-training (ditto)
Pre-test
Training (e.g., Use SQL-tutor)
Training (e.g., use paper & pencil)
Post-test

16. Three types Control number of problems solved Control training time
Time to complete training varies
Score on post-tests varies
Control training time
Problems completed varies
Control post-test score by repeating training until mastery
Time to mastery varies
Problems solved varies

17. Classic stats
If pre-test scores not significantly different, then can use T-test on post-test
Better:
Gain_score = Post-test_score – pre-test_score
T-test on gain_score: Most say “don’t use”
ANCOVA on post-test, with pre-test as covariate

18. Effect size Cohen’s d: d=1.0 is considered excellent; one letter grade
d = mean(gain_score(expt)) – mean(gain_score(control)) / standard_deviation(gain_score(control))
d=1.0 is considered excellent; one letter grade
d=0.5 is good
d=0.2 is questionable unless classroom/field

19. Evaluation of SQL-tutor
Participation voluntary
20 of 49 in course volunteered
2 hours for training
No pre-test, no post-test;
course exam used as post-test;
only questions relevant to SQLT counted
T-test on means: p=0.006
Effect size: d=0.66
self-selection bias

20. Classroom evaluation of Lisp Cognitive Tutor
Tutor vs. Lisp standard environment
Random assignment
Whole minicourse
Final exam; no pretest
T-test on means: Highly significant
Effect size: d=1.0

21. Lab evaluation of Lisp Tutor
Same exercises, same text; tutor vs. Lisp alone
Control students did not successfully complete all exercises

22. Evaluation of Geometry Cognitive Tutor
Tutor vs. baseline
Whole classes
Same teacher
Whole minicourse
Final exam; used prior year’s grade as covariate
Contribution of tutor to regression was reliable
Effect size: d= “more than 1.0”

23. Second evaluation of Geometry Tutor
4 conditions
Tutor + project teacher (best)
Tutor + nonproject teacher
Paper + project teacher
Paper + nonproject teacher
First condition better than all others, which tied
Effect size: d = 1.0

24. Evaluation of first Algebra Tutor
Tutor vs. baseline
Whole classes
No significant differences