1/∞ CRESST/UCLA Towards Individualized Instruction With Technology-Enabled Tools and Methods Gregory K. W. K. Chung, Girlie C. Delacruz, Gary B. Dionne, Eva L. Baker, John Lee, Ellen Osmundson American Educational Research Association Annual Meeting Chicago, IL - April 9-13, 2007 Symposium: Rebooting the past: Leveraging advances in assessment, instruction, and technology to individualize instruction and learning UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing

2/∞ CRESST/UCLA Structure of Talk Problem Pre-algebra/Algebra Classroom constraints Research Domain analysis, assessment design, instructional design, screen shots Results Did it work?

3/∞ CRESST/UCLA Problem

4/∞ CRESST/UCLA Study Context Why pre-algebra? Pre-algebra provides students with the fundamental skills and knowledge that underlie algebra Pre-algebra  Algebra  STEM LAUSD 9th grade cohort: Only 65% of 9th graders (~28,000) progress to 10th grade (23% retained in 9th grade [~10,000]; 12% leave district) Algebra identified as gatekeeper In fall 2006, 38% of CSU first-time freshmen needed remediation in mathematics (17,300) CSU 5-year graduation rate (STEM): 34%

5/∞ CRESST/UCLA Proportion Wrong 8th (1st semester) 9th (entering) th/7th Grade Standards

6/∞ CRESST/UCLA Research

7/∞ CRESST/UCLA Research Questions To what extent can students learn from “instructional parcels”—brief slices of instruction and practice? To what extent can automated reasoning (i.e., Bayesian networks) be used for automated diagnosis of pre-algebra knowledge gaps? What is the architecture for a diagnosis and remediation system?

8/∞ CRESST/UCLA General Idea Automated Reasoning (Bayes net) Pre-algebra pretest adding fractions distributive property multiplicative identity Individualized instruction and practice Individualized posttest

9/∞ CRESST/UCLA Research Study Overview Domain analysis and assessment design Identify the key concepts that underlie pre- algebra and the relations among those concepts Instruction and practice Design based on best-of-breed (worked examples, schema-based instruction, multimedia learning, effective feedback)

10/∞ CRESST/UCLA Domain Analysis

11/∞ CRESST/UCLA Domain Analysis

12/∞ CRESST/UCLA Domain Analysis

13/∞ CRESST/UCLA Sample Assessment Items DP (OPER) CP-ADD (CE) AP-ADD (CE) DP (CE) DP (OPER) DP = distributive property CP-ADD = commutative property of addition AP-ADD = associative property of addition CE = common error OPER = operation

14/∞ CRESST/UCLA Instructional Design

15/∞ CRESST/UCLA Design Assumptions To maximize the chance of learning with only brief exposure to content, instruction should: Direct the learner’s attention to important content Highlight and explain the importance of the content Use lay language and 1st/2nd person voice Use worked examples with visual annotations, coordinated and complementary narration Provide varied examples with different surface features but same underlying concept Provide practice on applying the concept Provide tailored and explanatory feedback

16/∞ CRESST/UCLA Multiplicative Identity - 4

17/∞ CRESST/UCLA Multiplicative Identity - 5

18/∞ CRESST/UCLA Multiplicative Identity - 6

19/∞ CRESST/UCLA Multiplicative Identity - 7

20/∞ CRESST/UCLA Multiplicative Identity - 8

21/∞ CRESST/UCLA Multiplicative Identity - 9

22/∞ CRESST/UCLA Multiplicative Identity - 10

23/∞ CRESST/UCLA Multiplicative Identity - 11

24/∞ CRESST/UCLA Multiplicative Identity – 12/12

25/∞ CRESST/UCLA Multiple Examples Stage 1: What is the next step in solving the problem? Stage 2: What is the result of carrying out the step? Stage 3: What is the underlying math concept?

26/∞ CRESST/UCLA Feedback is tailored to the specific option selected Knowledge of results (correct/incorrect) Explanation—why correct or incorrect Knowledge of results (correct/incorrect) Explanation—why correct or incorrect What to think about to solve problem Animation/video—goal, why, how, and common errors

27/∞ CRESST/UCLA Method 2 group pretest, posttest design 113 middle school students Instruction vs. no instruction (stratified by concept and high, medium, low knowledge) Procedure Pretest (84 items,  =.89) Instruction on concepts 4-6 concepts per student (out of 10) Individualized posttest

28/∞ CRESST/UCLA Method Analysis Examined 6 scales (pretest  = ) Adding fractions, distributive property, transformations, multiplicative identity Multiplying and adding fractions, rational number equivalence Participants dropped an analysis if Diagnosed as high knowledge Non-compliant (20-50%, depending on scale)

29/∞ CRESST/UCLA Results Adding fractions Distributive property Transfor- mations Multiplicative identity Multiply, add fractions Rationale number equivalence max = 8 n = 9/22 p =.50 d = -- max = 8 n= 14/18 p =.04 d =.76 max = 6 n= 13/21 p =.04 d =.77 max = 7 n= 7/21 p =.06 d =.91 max = 6 n= 13/12 p =.03 d =.91 max = 6 n= 13/12 p =.01 d =.50 Instruction No instruction

30/∞ CRESST/UCLA Student Perceptions 46% reported tool the was very useful, 53% reported they were very willing to use Tool not for everyone “I really understood the way they took step by step to show the problem.” “I thought that the video and the practice problems were very easy to understand, but if they were a little more ‘exciting’ it would help make the process more fun.” “No, it just made me confused. I like seeing everything on a board, not computer.”

31/∞ CRESST/UCLA Closing Remarks Preliminary results promising Instruction appears effective, even if brief Technical aspects of individualization of instruction and assessment tractable DITSy – Dumb Intelligent Tutoring System Limitations and next steps One instructional day between pretest and posttest; unknown retention effect; low  on some scales; novel assessment format Refine measures, replicate, validate diagnosis, examine more complex outcomes, examine instructional variables

32/∞ CRESST/UCLA Backup

33/∞ CRESST/UCLA Content Ten concepts tested Properties (e.g., distributive property) Problem solving (e.g., multiplying fractions) Time per parcel Mean time (mm:ss): 3:45 Range (mm:ss): 2:05 to 5:19, 10:58

34/∞ CRESST/UCLA Task Sequence Instruction (worked examples, narration, visual annotations) Practice (stage 1) Identify the next step Practice (stage 2) Identify the result Feedback (tailored and explanatory) Practice (stage 3) Identify the math concept Feedback (tailored and explanatory)

35/∞ CRESST/UCLA Incorrect response feedback: (a) confirmation that the response is incorrect, and (b) a brief explanation of why the response is wrong Hint on what to think about to solve problem Feedback is tailored to the specific option selected.

36/∞ CRESST/UCLA Correct response feedback: (a) confirmation that the response is correct, and (b) a brief explanation of why the response is correct

37/∞ CRESST/UCLA Don’t know response feedback: guidance on what the student should be considering

38/∞ CRESST/UCLA animation/video feedback: goal, why, how, and common errors

39/∞ CRESST/UCLA Timing ConceptDefault Time 10:58Adding fractions 5:19Reducing fractions 4:12Associative property of multiplication 4:09Distributive property 3:27Multiplicative inverse 3:19Multiplying fractions 3:04Multiplicative identity 2:27Associative property of addition 2:14Commutative property of addition 2:05Commutative property of multiplication

40/∞ CRESST/UCLA Some Observations Classroom / group instruction inefficient A lot of time spent on non-instructional activities Teacher telling jokes Students settling down (pulling out notebook from backpack, opening textbook, getting up to sharpen pencil) Teacher writing the equation on the whiteboard

41/∞ CRESST/UCLA Pretest-Posttest (Instruction) Adding fractions Distributive property Transfor- mations Multiplicative identity Multiply, add fractions Rational number equivalence max = 8 n = 22 p =.02 d =.50 max = 8 n= 18 p =.09 d =.54 max = 6 n= 21 p <.001 d =.61 max = 7 n= 21 p <.001 d =1.36 max = 6 n= 11 p =.14 d =.37 max = 6 n= 12 p =.10 d =.43

42/∞ CRESST/UCLA Pre-algebra Bayes Net definitions operations transformations common errors

43/∞ CRESST/UCLA Diagnosis and Remediation Model the domain of pre-algebra with a Bayesian network Treats test items as evidence of understanding Computes the probability of a student understanding a concept Short slices of instruction that are focused on a single concept NOT intended to replace classroom teaching Intended to support homework, review, wrap- around activities

44/∞ CRESST/UCLA

45/∞ CRESST/UCLA Multiplicative Identity - 1

46/∞ CRESST/UCLA Multiplicative Identity - 2

47/∞ CRESST/UCLA Multiplicative Identity - 3