The Effects of Feedback During Exploratory Math Practice SREE Fall 2011 The Effects of Feedback During Exploratory Math Practice Emily R. Fyfe & Bethany Rittle-Johnson Vanderbilt University Marci S. DeCaro University of Louisville

How do children learn best? Two schools of thought have emerged Advocates of Direct Instruction (Kirschner, Sweller, & Clark, 2006) Advocates of Discovery Learning (Bruner, 1961; Kuhn, 1989; Piaget, 1973)

An Integrated Perspective No need for this strict dichotomy “Proponents of constructivism and direct instruction…each have something to learn from the other” (Rieber, 1992) “A mixture of guidance and exploration is needed” (Mayer, 2004) “There’s a place for both direct instruction and student-directed inquiry” (Kuhn, 2007) “Characterizing discovery and direct instruction as diametrically opposed…has done a disservice to both approaches” (Wilson, et al, 2010)

Combining Instruction and Discovery Exploration prior to instruction facilitates learning (DeCaro & Rittle-Johnson, 2011; Schwartz & Bransford, 1998) Hints or coaching during problem solving is better than pure problem solving alone (Mayer, 2004) Recent meta-analysis indicates that guided discovery is better than unassisted discovery or direct instruction (Alfieri et al, 2010)

Questions remain… Which elements of instruction are most suitable to incorporate within exploration? For whom is guided exploration most advantageous and why?

Feedback During Exploration Feedback represents one element of instruction that may be particularly effective in combination with exploration What is Feedback? Any information that the learner can use to confirm, reject, or modify prior knowledge Accuracy information, hints, etc.

(Ahmad, 1988; Kluger & DeNisi, 1996; Luwel et al., 2011) Types of Feedback Outcome Feedback Strategy Feedback Provides information about learner’s answer Used extensively Related to strong, positive effects in past work compared to no feedback Provides information about how answer was obtained Only examined in a few previous studies Better than outcome feedback in terms of strategy selection (Ahmad, 1988; Kluger & DeNisi, 1996; Luwel et al., 2011)

Why feedback? May reduce disadvantages of discovery by guiding the learner’s search for information Helps identify errors and encourages search for plausible alternatives (e.g., new strategies) (Hattie & Timperley, 2007; Mory, 2004) But… Past research indicates variable efficacy (Kluger & DeNisi, 1996) May only benefit a subset of learners

What about prior knowledge? Expertise reversal effect Instructional technique is effective for novices, but loses its benefits for high-knowledge learners (Kalyuga, Ayres, Chandler, & Sweller, 2003) Example: worked examples vs. problem solving Low knowledge learners benefit from more external guidance; high knowledge learners benefit from less Perhaps feedback during exploration only helps children who have low prior knowledge

Why the reversal? Due to existing cognitive resources (Paas, Renkl, & Sweller, 2003) Novices lack schemas; need external guidance to reduce cognitive load High-knowledge learners have schemas; additional guidance creates more cognitive load

Goals of this study Examine the effects of feedback during exploratory math practice for children with varying levels of prior knowledge Specifically Compare feedback vs. no feedback Compare outcome vs. strategy feedback Look at effects of prior knowledge

Hypotheses Ho 1: Feedback > No Feedback Ho 2: Strategy Feedback > Outcome Feedback One sentence about why strat feedback is better Ho 3: Feedback better for children with low prior knowledge

Domain: Mathematical Equivalence Concept that two sides of an equation represent the same amount and are interchangeable Commonly represented by equal sign (=) Important for algebra: hammer this point 3 + 7 + 8 = 3 + _ 6 + 4 = _ + 8

Why Study Math Equivalence? Fundamental concept in arithmetic and algebra Very difficult for children in U.S. Interpret equal sign as an operator symbol that means “the total” as opposed to relational symbol (McNeil, 2008; Rittle-Johnson & Alibali, 1999) In one study, only 24% of U.S. children in 3rd and 4th grade solved math equivalence problems correctly (McNeil & Alibali, 2000)

Participants Worked with 91 children (2nd & 3rd grade) M age = 8 yrs, 7 mo 53 females, 38 males 45% white, 40% black, 15% other 47% receive free or reduced lunch

Design and Procedure Session 1: Pretest (~25 minutes) Excluded if score >80% on pretest measures Session 2: Intervention & Posttest (~50 minutes) Session 3: Two-week Retention Test (~25 minutes)

Tutoring Intervention Exploratory Practice Attempt to solve 12 math equivalence problems Randomly assigned to 1 of 3 conditions No Feedback (n = 31) Outcome Feedback (n = 32) Strategy Feedback (n = 28) Midtest Brief conceptual instruction (DeCaro & Rittle-Johnson, 2011) Be 7:30 in

Exploratory Practice Find the number that goes in the blank. 3 + 4 + 8 = 3 + ☐ How did you solve that problem? No Feedback: “OK, let’s move on to the next problem.” Outcome Feedback: “Good try, but that’s not the correct answer. The correct answer is 12.” Strategy Feedback: “Good try, but that’s not a correct way to solve that problem.”

Assessment of Math Equivalence Procedural Knowledge Use correct strategy to solve problems Conceptual Knowledge Understand concept of equivalence Used at Pretest, Midtest, Posttest, & Retention Test 7 + 6 + 4 = 7 + _ 6 - 4 + 3 = _ + 3 Emphasize how we measure procedural knowledge What does the equal sign mean? 4 + 8 = 8 + 4 True or False? (Rittle-Johnson, Matthews, Taylor, & McEldoon, 2011)

Analysis & Results Contrast-based ANCOVA (West, Aiken, & Krull, 1996) Two contrast-coded condition variables Feedback (no feedback vs. two feedback conditions combined) Feedback Type (outcome feedback vs. strategy feedback) Two condition x prior knowledge interactions Three covariates Interaction follow-up Categorize as low vs. high knowledge (median split) Simple main effects of condition Also note the results I’ll be presenting. First I’ll discuss children’s procedural knowledge on the posttest assessments. Then I’ll show some results from the intervention activities. Contrast-based ANCOVA Two contrast-coded condition variables Feedback (No feedback vs. Feedback conditions combined) Feedback type (Outcome feedback vs. Strategy feedback) Two condition x prior knowledge interactions Three covariates

Procedural Knowledge Repeated Measures ANCOVA: Midtest, Posttest, and Retention Test. Less time.

Procedural Knowledge Overall feedback x prior knowledge interaction, F(1, 83) = 7.05, p = .01 Low Knowledge: Feedback vs. No Feedback, F(1, 83) = 3.84, p = .05 High Knowledge: Feedback vs. No Feedback, F(1, 83) = 4.61, p = .04 * Less time. *

Procedural Knowledge Results Expertise reversal effect Feedback during exploratory math practice is more beneficial than no feedback, but only for children with low knowledge For children with high prior knowledge, the reverse is true; they benefit more from no feedback

Intervention Activities Subjective Cognitive Load (NASA-TLX, Hart & Staveland, 1988) I had to work hard to solve those problems. I was stressed and irritated when I had to solve those problems. Mean rating on agreement scale from 1 to 5. Problem-Solving Strategy Use Variability of correct & incorrect strategies

Cognitive Load * Low Knowledge: Feedback vs. No Feedback, F(1, 83) = 1.15, p = .29 High Knowledge: Feedback vs. No Feedback, F(1, 83) = 6.05, p = .02

Cognitive Load Supports expertise reversal effect explanation Feedback may have hurt high-knowledge learners’ performance because of increased cognitive load An effect not found for low-knowledge learners

Strategy Coding Scheme Sample explanation (4 + 5 + 8 = __ + 8) Correct Strategies Equalize I added 4, 5, and 8 and got 17. And 9 plus 8 is 17. Add-Subtract I added 4, 5, and 8 and got 17. And 17 minus 8 is 9. Grouping I took out the 8’s and I added 4 plus 5. Incorrect Strategies Add-All I added the 4, 5, 8 and 8. Add-to-Equal I just added the first three, the 4, 5, and 8. Carry I saw a 4 here, so I wrote a 4 in the blank. Ambiguous I used 8 plus 8, and then 5. Spend less time on this slide

Perseveration Perseveration = Using the same incorrect strategy on all the problems. * * P = .01

Incorrect Strategy Variability * * Ranged from 0 to 5. Low Knowledge: Feedback vs. No Feedback, F(1, 83) = 5.66, p = .02 High Knowledge: Feedback vs. No Feedback, F(1, 83) = 4.56, p = .04

Correct Strategy Variability * Ranged from 0 to 3. Low Knowledge: Feedback vs. No Feedback, F(1, 83) = 4.76, p = .03 High Knowledge: Feedback vs. No Feedback, F(1, 83) = 0.16, p = .90

Strategy Variability For children with low prior knowledge, feedback prevented perseveration and led to the generation of diverse strategies May explain why these children learned more when they received feedback than when they did not For children with higher prior knowledge, feedback led to the generation of more incorrect, but not correct strategies May help explain why feedback had negative impact Feedback facilitated the generation of incorrect, but not correct strategies for children with high prior knowledge May help explain why feedback had negative impact

Summary Feedback led to higher procedural knowledge of math equivalence than no feedback, but only for children with low prior knowledge For children with high prior knowledge, no feedback was better No differences between outcome feedback and strategy feedback

Implications Theoretical Practical Extends expertise reversal effect to feedback Clarifies research on discovery learning Practical Pay more attention to when you give feedback during tutoring and teaching

Thank You Children’s Learning Lab Bethany Rittle-Johnson Marci DeCaro Laura McLean Maryphyllis Crean Lucy Rice Funding Sources ExpERT Training Grant, through IES to Fyfe NSF CAREER grant to Rittle-Johnson

(Kirschner, Sweller, & Clark, 2006; Mayer, 2004; Sweller, 1988) Instruct vs. Discover Direct Instruction Discovery Learning Told or shown how to solve the problems Provides structure and reduces task ambiguity But… Limits self-discovery Might limit learner engagement Explore the problems on your own with no guidance Find target information and new strategies independently But… Overwhelms working memory Might never locate target info or invent correct strategy (Kirschner, Sweller, & Clark, 2006; Mayer, 2004; Sweller, 1988)

Exploratory Practice Children’s performance on 12 problems during intervention (Percent Correct)

Let’s take a look at this problem. Instruction Let’s take a look at this problem. 3 + 4 = 3 + 4 There are two sides to this problem, one on the left side of the equal sign and one on the right side of the equal sign… The equal sign means that the left side of the equal side is the SAME AMOUNT AS the right side of the equal sign. That is, things on both sides of the equal sign are equal or the same.

Conceptual Knowledge Feedback type x prior knowledge interaction, F(1, 83) = 4.82, p = .03. Low knowledge: No effect of feedback type, F(1, 83) = 0.51, p = .48 High knowledge: Main effect of feedback type, F(1, 83) = 8.21, p = .005

Intervention – Strategy Use Incorrect Correct * * * Note. Differences are between no feedback condition and two feedback conditions combined: * p < .05

Strategy Variability * * Note. Difference is between no feedback condition and two feedback conditions combined: * p < .001