1. What Works in Undergraduate Science Education? 1 Heidi Iverson, Colorado State University OECD France Workshop Education for Innovation: the Role of Arts and STEM Education May 24th, 2011

2. Traditional Science Classes OECD France Workshop May 24 th, 2011 2

3. 3 Innovations OECD France Workshop May 24 th, 2011

4. Meta-Analysis Systematic approach to the synthesis of research findings Pooling existing evidence Summing up: Innovations implemented Evidence of success 4 OECD France Workshop May 24 th, 2011

5. What is needed? Gather empirical evidence Metric for computing effects of different studies? Effect Size 5 OECD France Workshop May 24 th, 2011

6. Gathering and Coding of Research Studies 6

7. What was included? Four criteria: 1.focus on undergraduate education in biology, chemistry, engineering or physics; 2.include one or more instructional strategies considered to be an innovation; 3.refer to actual classrooms, rather than controlled conditions; and 4.be reported in article published in 1990 or later. (Suter & Narayanan, 2006; Beichner, 2009) 7 OECD France Workshop May 24 th, 2011

8. Innovation Commonalities 8 Conceptually Oriented Tasks Collaborative Learning Technology Inquiry- Based Projects OECD France Workshop May 24 th, 2011

9. 9 All Innovation Studies n = 310 Biology n = 98 Chemistry n = 26 Engineering n = 38 Physics n = 148 OECD France Workshop May 24 th, 2011

10. Effect Size Distribution Included Studies n = 166 ES from -1.06 to 2.57 Mean = 0.47 SD = 0.54 83% Positive Effect Size 10 OECD France Workshop May 24 th, 2011

11. Factors Explaining Variability?

12. Innovation Type Innovation Combination Number of Studies Mean Effect Size (SD) Technology300.35 (0.39) Conceptually Oriented Tasks + Collaborative Learning 420.56 (0.41) Conceptually Oriented Tasks + Technology 180.41 (0.61) Conceptually Oriented Tasks + Collaborative Learning +Technology 380.46 (0.50) 12 OECD France Workshop May 24 th, 2011

13. 13 Biology Mean = 0.54 (0.66) Chemistry Mean = 0.27 (0.41) Engineering Mean = 0.08 (0.58) Physics Mean = 0.59 (0.37)

14. The Issue of Assignment 88% of studies do not have random assignment of individuals to treatment and control Pretest scores are essential to ensure group equivalence For those studies that do, when taken into account: Mean effect size is much larger 14 OECD France Workshop May 24 th, 2011

15. Importance of Outcome Measure Outcome measure item format Multiple-choice (mean ES=0.56) Open-ended (mean ES= 0.35) Alignment of the outcome measure to the study In one particular study: Force and Motion Conceptual Evaluation (ES = 1.36) Open-ended traditional problem solving (ES = -0.62) 15 OECD France Workshop May 24 th, 2011

16. 16 All Studies n = 79 n = 48 n = 29 n = 23 Study Design Pretest-Posttest 2 Group Control Group Traditional Outcome Measure Multiple Choice Conceptual Evaluation Subsequent Analysis in Physics OECD France Workshop May 24 th, 2011

17. Workshop/Studio Physics 17 1. Integrated Classroom Environment 4. Research-Based Activities 3. Students Gather Data Using Technology 5. Instructor is Facilitator 2. Structured Student Group Work OECD France Workshop May 24 th, 2011

18. Conclusions Innovations have a positive effect on student learning Things to consider: Providing sufficient empirical data Importance of outcome measure Pretest administration Workshop/Studio Physics has a particularly high effect on student learning 18 OECD France Workshop May 24 th, 2011

19. Special Thanks to: Ayita Ruiz-Primo, Derek Briggs, Lorrie Shepard, and Bud Talbot This research has been funded by NSF grant REC-REESE 0635491 Contact Email: Heidi.L.Iverson@Gmail.com

20. Study Search Methods Word of Mouth Journal Search Validate Keywords Expanded Search Classify Classify papers as either: Background Synthesis Descriptive Comparative Comparative Studies Biology = 82 Chemistry = 18 Engineering = 23 Physics = 74 20 SEI at CU Boulder End of Year Event 2011