1. Conducting Rigorous Evaluations of Interventions That (May) Improve Student Learning: A Researcher/Teacher Reflects
Jon R. Star
Michigan State University
Harvard University (as of July 2007)

2. About me Former middle and high school mathematics teacher
PhD in Educational Psychology, with emphasis on middle/high school students’ learning of mathematics
Two grants from US Dept. of Ed (IES) exploring interventions designed to improve students’ learning of mathematics

3. Why I was asked to speak today
Perhaps useful for me to share my experiences as a researcher who designs and conducts rigorous (experimental) evaluations of interventions designed to improve students’ learning of mathematics
What are the challenges that I face in my work that I feel are endemic to the evaluation work that many of you are trying to do?

4. Research examples (1)
The benefits of comparison for learning mathematics
Better to solve problems by viewing worked examples sequentially or side by side?
Example 1
Example 2
p.1
p.2
Example 1
Example 2
p.1

5. Research examples (2)
Learning strategies for solving ratio and proportion problems using schema
Is using schema is better than not using schema? ? If
Then
Proportion
Problem
Schemata ? =
Ratio Problem Schemata

6. Two types of challenges
Tension between:
my role as a researcher, and
my heart and soul as a teacher
Balancing between:
the need to design the intervention to be very ‘clean’ to enable an experimental study, and
the need to provide experiences for students that are appreciative of good (and sometimes ‘messy’) instructional practices

7. Researcher vs. teacher

8. Flow of intervention Let intervention run its course
Give help in standardized way
Don’t deviate from the instructional protocol or script
If students don’t ‘get it’, spend extra time with them
Focus on mastery of the material
Spend extra time with struggling students

9. Scorecard =
Flow of intervention

10. Assessing learning
Materials need to discriminate and have room for learning
OK with pretest scores of 10%
OK with posttest scores of 50%
Materials should be challenging but students should be able to succeed
‘Failing’ tests is disheartening
Average posttest grade of a B?

11. Scorecard
Flow of intervention =
Assessing learning

12. Relations with schools
Keep professional distance from schools and teachers
Don’t want to ‘spill the beans’ so that we can use the school in future work
Don’t tell if study didn’t work
Establish deep and long-lasting relationships with schools, teachers, and students
Researchers shouldn’t zip in/out
Go back and visit; share results

13. Relations with schools
Scorecard
Flow of intervention
Assessing learning =
Relations with schools

14. Ethics of study design
Experimental design tells us the most about the intervention
Control group is necessary
There may be “winner” and “loser” conditions
All conditions should lead to student learning
Design should allow non-treatment students to experience treatment eventually (if successful)

15. Relations with schools
Scorecard
Flow of intervention
Assessing learning
Relations with schools =
Ethics of study design

16. Two types of challenges
Tension between:
my role as a researcher, and
my heart and soul as a teacher
Balancing between:
the need to design the intervention to be very ‘clean’ to enable an experimental study, and
the need to provide experiences for students that are appreciative of good (and sometimes ‘messy’) instructional practices

17. Clean vs. messy

18. Instructional format Students working alone is an easier design
Logistically
Statistically
Students working with partners is often better for learning
Also can help teachers get more comfortable with this instructional format

19. Scorecard =
Instructional format

20. Instructional delivery
Researchers providing instruction is easier:
no PD
consistency across schools
Letting students’ regular teachers do the teaching is better:
we are unfamiliar with norms
we don’t know students’ names

21. Instructional delivery
Scorecard
Instructional format =
Instructional delivery

22. Relations with teachers
Better if teachers don’t know too much about the intervention
might bias their interactions with students
The best way to have a long term impact on students is by working with teachers on instructional improvement

23. Scorecard = Instructional format Instructional delivery
Relations with teachers

24. Jon Star jonstar@msu.edu www.msu.edu/~jonstar
Thanks!
Jon Star