1. Evaluating Adaptive Generation of Problems in Programming Tutors – Two Studies Amruth KUMAR Ramapo College of New Jersey, Mahwah, NJ 07430, USA

2. Adaptation in tutors Text Navigation Problem sequence Feedback

3. Adaptive problem generation Vector spaces [Kurhila et al ITS 02] Learning Spaces [Salton et al 75] Disadvantages: Exhaustive enumeration Adding new concepts/problems entails redesign

4. Associative adaptation Identify concepts in a topic Build overlay student model in terms of the concepts Maintain proficiency on each concept Associate each problem with concept For the next problem: Select the concept Select the problem – Round robin algorithm

5. Problets Problem-solving tutors for program analysis Available for: Expression Evaluation: Arithmetic, Relational Selection Loops – Pretest, Counter-controlled Pointers in C++ www.problets.org

6. Arithmetic Expression Concepts Correct evaluation of +, -, *, / (Integer & Real), % Precedence of +, -, *, /, % Associativity of +, -, *, /, % Coercion in +, -, *, /, % Divide by zero error for /, % % applied to real operands

7. Knowledge Model

8. Overlay Student Model

9. Cognitive Student Model

10. Problem Template Learning Objective: Modulus (%) Operator applied to real operands Template: % Example: 2 % 6.5

11. Selecting the concept for the next problem Given the last concept was Ci: If Ci has been mastered, return concept Ci + 1. If i + 1 > n, set i = 1, return C 1 If p problems already generated back to back on Ci, return Ci + 1. If i + 1 > n, set i = 1, return C1 Else, return Ci.

12. Associative adaptation Persistence p = 1: rapid context-switch - better for testing p > 3: problems predictable p = 2: better for tutoring

13. Evaluation Hypothesis - 1 Targets the concepts less well understood by students. Principle: Categorize student-concepts, not students into control and test groups

14. Tutor on Selection – 05 Protocol Pretest – 21 problems, 12 concepts, 8 minutes, no feedback Initialized student model Practice – Adaptive, 15 min max Post-Test – Similar to Pretest

15. Classifying concepts Problems SolvedPre-TestPracticePost-Test Discarded 0** **0 KnownA ≥ M 1 & R / A ≥ M 2 ** Control +0+ Test +++

16. Control Vs Test Concepts Spring 05 Results Spring 05Pre-TestPracticePost-Testp-value Prob.AveProblemsProb.AveProb.Ave Control (N = 56 student-concepts) Average1.020.8801.110.870.020.68 Std-Dev0.130.3000.31 Test (N = 135 student-concepts) Average 1.070.461.831.350.680.000 Std-Dev 0.260.471.140.480.43 p-value 0.050.000

17. Control Vs Test Concepts Fall 05 Results Fall 05Pre-TestPracticePost-Testp-value Prob.AveProblemsProb.AveProb.Ave Control (N = 26 student-concepts) Average1.150.8101.460.760.0020.55 Std-Dev0.370.3500.510.40 Test (N = 87 student-concepts) Average1.000.611.551.150.860.000 0 Std-Dev00.471.200.360.31 p-value0.040.020.0060.23

18. Pretest Loop/Spring 06 (N=67) Pre-TestPost-Testp-value ProbAveProbAveProbAve Unknown Unpractised (N=43 student concepts) Ave1.300.001.260.26 0.6220.0002 StDev0.460.000.540.44 Unknown Practised (N=220 student concepts) Ave1.880.041.800.38 0.1440.0000 StDev0.950.130.930.47 Already Known (N=236 student concepts) Ave1.310.991.340.89 StDev0.590.040.560.30

19. Counter Loop/Spring 06 (N=66) Pre-TestPost-Testp-value ProbAveProbAveProbAve Unknown Unpractised (N=54 student concepts) Ave1.180.011.040.31 0.05850.0000 StDev0.520.070.190.47 Unknown Practised (N=224 student concepts) Ave1.600.071.660.46 0.23740.0000 StDev0.740.180.840.46 Already Known (N=161 student concepts) Ave1.241.001.470.86 StDev0.510.000.590.33

20. Confounds Earlier concepts in practice are test, later concepts are control Decreasing order of difficulty – targets harder concepts Increasing order of difficulty – pre-test control average > pre-test test concept average More room for improvement on harder concepts – compare for effectiveness: Not control vs test But, pre vs post

21. Evaluation Hypothesis - 2 Helps students learn with fewer problems. Principle: Adaptation may be inclusionary or exclusionary Compare: Exclusionary adaptation against the worst-case (all- inclusive case); Inclusionary adaptation against the best-case (all-exclusive case). Gain of adaptation: Inclusionary adaptation – percentage increase due to adaptation; Exclusionary adaptation – percentage decrease due to adaptation

22. Arithmetic Expressions Tutor Spring 05 Protocol Pre-test: 21 problems, 17 concepts, 7 min, no feedback Practice: 15 minutes, p=2, detailed feedback Control group: Non-adaptive tutor Test group: Adaptive tutor Post-test: Similar to pre-test

23. Arithmetic Expressions Tutor Spring 05 Results Spring 05 Arithmetic Pre-TestPracticePost-Testp-value Prob.Ave.ProblemsProb.Ave.Prob.Ave. Control (N = 21 students) Average10.480.4637.4314.100.610.00010.0066 Std-Dev4.210.2519.104.950.24 Test (N = 35 students) Average10.110.4725.3714.230.590.00000.0004 Std-Dev4.630.2719.395.930.28 p-value0.7710.9270.0270.9310.851

24. Relational Expressions Tutor Spring 05 Protocol Pre-test: 17 problems, 17 concepts, 7 min, no feedback Practice: 15 minutes, p=1, detailed feedback Control group: Non-adaptive tutor Test group: Adaptive tutor Post-test: Similar to pre-test

25. Relational Expressions Tutor Spring 05 Results Pre-TestPractic e Post-Testp-value Prob.Ave.Prob. Ave.Prob.Ave. Control (N = 21 students) Ave.13.900.6845.0515.000.770.2560.020 StdDev3.130.2119.193.390.16 Test (N = 16 students) Ave.14.560.7314.1315.380.820.3020.023 StdDev3.440.2216.122.580.16 p-value0.5540.4740.0000.7050.355

26. Conclusions Associative adaptation: Targets the concepts less well understood by students. Helps students learn with fewer problems.

27. Associative adaptation - Advantages Easier to build Supports any learning path Scalable Domain-independent

28. Acknowledgments Partial support for this work was provided by: National Science Foundation's Educational Innovation Program under grant CNS-0426021. Ramapo Foundation