1. 1/∞ CRESST/UCLA Towards Individualized Instruction Using Technology Gregory K. W. K. Chung Annual CRESST Conference Los Angeles, CA – January 22, 2007 UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing

2. 2/∞ CRESST/UCLA Structure of Talk Problem statement Research questions Technical approach to diagnoses and prescription Study design

3. 3/∞ CRESST/UCLA Problem Statement

4. 4/∞ CRESST/UCLA Released NAEP Items NAEP 1990.69 correct Current Study Lo:.73 correct Hi:.87 correct NAEP 2003.48 correct Current Study Lo:.48 correct Hi:.69 correct NAEP 2003.52 correct Current Study Lo:.63 correct Hi:.87 correct.37 incorrect.13 incorrect.52 incorrect.31 incorrect.27 incorrect.13 incorrect.48 incorrect.52 incorrect.31 incorrect

5. 5/∞ CRESST/UCLA Even Simpler Problems Lo:.36 correct Hi:.68 correct Lo:.73 correct Hi:.88 correct Lo:.68 correct Hi:.90 correct Lo:.60 correct Hi:.60 correct Lo:.56 correct Hi:.90 correct

6. 6/∞ CRESST/UCLA Study Context Why pre-algebra? Pre-algebra provides students with the fundamental skills and knowledge that underlie algebra In fall 2004, 57% of CSU first-time freshmen needed remediation in mathematics (22,000 students) Remediation includes basic math, pre- algebra, geometry, and algebra By fall 2005, 3500 students did not complete remediation and were disenrolled

7. 7/∞ CRESST/UCLA Study Context Tasked with developing instructional and assessment supports for pre-algebra Develop an approach to support rapid diagnosis and remediation of pre-algebra knowledge and skill gaps Develop assessment and instructional tools to support classroom instruction Test approach with middle school students Do it with technology, do it now

8. 8/∞ CRESST/UCLA Research Questions

9. 9/∞ CRESST/UCLA Research Questions What is the architecture for a diagnosis and remediation system? What are necessary components of such a system, how is diagnosis performed, how is remediation performed, how is the system validated, how is effectiveness measured?

10. 10/∞ CRESST/UCLA Research Questions To what extent can a Bayesian network codify and capture the structure and properties of pre-algebra? This question addresses the use of BN for representing domain knowledge (i.e., pre- algebra) and automated reasoning.

11. 11/∞ CRESST/UCLA Research Questions What are methods for developing “instructional parcels”? What is the structure of the student interaction, what is the format of the media delivery, what are techniques that can engage students, what are the scalability issues, what are the development bottlenecks?

12. 12/∞ CRESST/UCLA Basic Approach Automated Reasoning (Bayes net of pre- algebra) Pre-algebra pretest adding fractions distributive property multiplying fractions Individualized instruction Individualized posttest

13. 13/∞ CRESST/UCLA Domain Analysis Specify domain structure as a Bayes net Specify concepts and influences among concepts Attach math items (“observables”) to concepts to form scales Develop assessment items using a step- by-step derivation of a few complex problems Internally coherent Enables precise diagnoses

14. 14/∞ CRESST/UCLA Pre-algebra Bayes Net definitions operations transformations common errors

15. 15/∞ CRESST/UCLA Domain Sampling

16. 16/∞ CRESST/UCLA Sample Items DP (OPER) CP-ADD (CE) AP-ADD (CE) DP (CE) DP (OPER) DP = distributive property CP-ADD = commutative property of addition AP-ADD = associative property of addition CE = common error OPER = operation

17. 17/∞ CRESST/UCLA Instructional Design Issues

18. 18/∞ CRESST/UCLA Study

19. 19/∞ CRESST/UCLA Study Design and Procedure Pretest (1st occasion) Assign Instructional Parcels to S Score Tests (at UCLA) Generate Predictions Assign S to Conditions Occasion 1 Occasion 2 Computer Instruction Posttest Experimental Condition Posttest Control Condition Computer Instruction 2 days

20. 20/∞ CRESST/UCLA Expected Relations PretestPosttest PretestPosttest PretestPosttest PretestPosttest Low Prior Knowledge High Prior Knowledge No Instruction Instruction Score

21. 21/∞ CRESST/UCLA Results (L/M Prior Know.) a Wilcoxan signed rank test (non-parametric paired test). Scale InstructionNo Instruction neg.pos.tiepapa neg.pos.tiepapa Adding fractions2195<.01140ns Multiplying fractions8205.06343ns Reducing fractions222ns103 Transformations5236<.01183.02 Distributive property8175.02631ns

22. 22/∞ CRESST/UCLA Results (L/M Prior Know.) a Wilcoxan signed rank test (non-parametric paired test). Scale InstructionNo Instruction neg.pos.tiepapa neg.pos.tie papa Commutative property of addition 115ns111 Commutative property of multiplication 052.04302.08 Associative property of addition 333ns115 Multiplicative inverse595ns140 Multiplicative identity6196<.01424ns

23. 23/∞ CRESST/UCLA Discussion Preliminary results promising Individualization of instruction and assessment tractable Instructional parcels appear effective but more work needed Next steps Refine items and domain sampling End-to-end automation Continue gathering validity evidence

24. 24/∞ CRESST/UCLA Questions

25. 25/∞ CRESST/UCLA