1. High-Impact Research Design: Examples from the BioKIDS Project
Nancy Butler Songer
The University of Michigan
Abstract
How do the concepts of fidelity and sustainability shift when scaling is focused on EFFECTING CHANGE in urban schools through a focus on the URBAN DISTRICTS? Building from current multidimensional definitions of scaling (e.g. Coburn, 2003), this paper discusses major challenges to implementing interventions with fidelity in many simultaneous and diverse classroom contexts, and then presents revised definitions of key terms to emphasize a measure of sustained impact on individuals. Key points are illustrated with examples from our multi-year, coordinated curricular/assessment system designed to afford comprehensive evidence of longitudinal impact across time and topic.

2. Focus on High-Poverty Urban Students
~30,000 Detroit Public School middle school students
70% free/reduced lunch
95% African American, Hispanic, Mixed Race
Class sizes often over 30
~ No substitutes, outside recess
Extreme pressure to perform well on high-stakes tests
Virtually no experience with digital resources or data
little agreement exists between programs on what components should be brought to scale or what the profile of growth should be. In other words, does “bringing an intervention to scale” refer to documentation of widespread use among a thousand classrooms at any one time or are consideration of depth of implementation and sustainability important?
Regardless of focus, most researchers would agree that the current body of research on scaling curricular reforms includes an under emphasis of research approaches and sound experimental methods that might result in clear evidence of cognitive and epistemological outcomes to evaluate the effectiveness of the intervention in the many locations.
In our case, we are bringing to scale several standards-based middle school curricular programs shown to be successful as measured by student learning outcomes. Key ideas in our scaling efforts throughout the eight years include…

3. BioKIDS: Kids’ Inquiry of Diverse Species
Numerous, sequential replacement units in middle school science
Activities scaffold learning of science concepts and complex reasoning in concert
Learning technologies created; from digital resource to cognitive tool
Own data as basis for inquiry reasoning (building explanations, analyzing data)
little agreement exists between programs on what components should be brought to scale or what the profile of growth should be. In other words, does “bringing an intervention to scale” refer to documentation of widespread use among a thousand classrooms at any one time or are consideration of depth of implementation and sustainability important?
Regardless of focus, most researchers would agree that the current body of research on scaling curricular reforms includes an under emphasis of research approaches and sound experimental methods that might result in clear evidence of cognitive and epistemological outcomes to evaluate the effectiveness of the intervention in the many locations.
In our case, we are bringing to scale several standards-based middle school curricular programs shown to be successful as measured by student learning outcomes. Key ideas in our scaling efforts throughout the eight years include…

4. From Digital Resource to Cognitive Tool using CyberTracker

6. Research Foci
Development of assessment systems to systematically evaluate simple and complex reasoning in science
Urban students’ development of complex reasoning throughout multiple, coordinated science units
Scaling focusing on high-impact of intervention

7. Research Foci
Development of assessment systems to systematically evaluate simple and complex reasoning in science (with PADI IERI)
Urban students’ development of complex reasoning throughout multiple, coordinated science units (BioKIDS IERI)
Scaling focusing on high-impact of intervention

8. Treatment
6th grade: Three eight-week, inquiry-fostering science units using customized learning technologies
Biodiversity and Ecology
Weather
Simple Machines
7th grade: Up to four additional eight-week units
8th grade
One fall unit
Michigan Standardized Test in Science (MEAP)
___________________________________________
= 3-8 coordinated inquiry units in 6, 7, 8th grades

9. Quasi-Experimental Design
Pool of 25 teachers all given professional development support (~5000 students)
Assignment to treatment groups determined by extent of intervention activities implemented
Minimal Treatment (MT)
595 students in 6 classrooms
Performed 0-30% activities in one academic year
High Impact Treatment (HT)
1329 students in 15 classrooms
Performed % activities in one academic year

10. Development of Assessment System: Content-Inquiry Matrix
complexity of science content required to perform task
amount of
inquiry scaffold provided

11. Look for student gains at simple, moderate and complex levels
complexity of science content required to perform task
amount of
inquiry scaffold provided

12. If all the small fish in the pond ecosystem died one year from a disease that killed only the small fish, what would happen to the algae in the pond? Explain why you think so.
Example of step 3 complex task on making claim and building explanations

13. Student Performance on Inquiry Reasoning: Pre and Posttest (Biodiversity)

14. Student Performance on Inquiry Reasoning By Complexity Type

15. Passing Percentages by State, District, and Treatment Schools
On State Science Test, Eighth Grade (MEAP)
Comparative statistics on fish pond item
Not that these data represent only one item. While this early data is suggestive and encouraging, we need to get averages across multiple ides in order to have better generalizability of the results.
We are now collecting these data and have some to report.
(Numbers in parentheses indicate standard error)

16. Research Foci
Development of assessment systems to systematically evaluate simple and complex reasoning in science (with PADI)
Urban students’ development of complex reasoning throughout multiple, coordinated science units
Scaling focusing on high-impact of intervention

17. What is being “brought to scale”?
Numerous consecutive eight-week middle school science units
Coordination towards fostering complex reasoning
Emerging technologies for complex reasoning
Ongoing professional development
little agreement exists between programs on what components should be brought to scale or what the profile of growth should be. In other words, does “bringing an intervention to scale” refer to documentation of widespread use among a thousand classrooms at any one time or are consideration of depth of implementation and sustainability important?
Regardless of focus, most researchers would agree that the current body of research on scaling curricular reforms includes an under emphasis of research approaches and sound experimental methods that might result in clear evidence of cognitive and epistemological outcomes to evaluate the effectiveness of the intervention in the many locations.
In our case, we are bringing to scale several standards-based middle school curricular programs shown to be successful as measured by student learning outcomes. Key ideas in our scaling efforts throughout the eight years include…

18. Three Scaling Research Foci
1: Focus on Numbers
2: Focus on Complexities of Intervention
3: Focus on Persistence Within Individuals

19. Current IERI project
Persistence includes up to three curricular units for each teacher , and up to 8 units for each student over a three year period.

20. Scaling Focusing on Persistence
Persistence = sustained impact of intervention with individuals
Persistence works with science inquiry (e.g. developing complex reasoning skills takes time, multiple experiences)

21. Conclusions IERI presents a profound research challenge
Realize strong empirical outcomes and clear evidence to evaluate effectiveness of reform programs
Within challenging school contexts
While also considering scaling /impact tradeoffs
However, we know a fair amount about how people learn; now is the time to develop strengthen the pool of empirical data to further examine and challenge our learning theory-based understanding in real classroom settings
Our research scaling of successful, small-scale educational interventions with an explicit focus on maintaining program fidelity across a range of scaled locations. Early on, while our scaling efforts were regarded as successful on many criteria, we questioned our ability to make sound conclusions about the character of the reform in the distant locations. In addition, we questioned the ability of the reform to remain in place once we were no longer supporting it, A focus on longer term intervention impact within individuals motivated our shift to those more at-risk for school failure including children in inner city schools. Finally, we address reform challenges at the district level These shifts in research focus reflect an explicit concentration on the concepts of fidelity and sustainability relative to scaling research. In our work, we have adopted a multidimensional definition of scaling not dissimilar from that championed by Coburn . We adapted a fidelity and sustainability-driven definition of scaling because we felt that scaling research focusing primarily on quantitative measures of impact disallowed a sound examination of the depth, sustainability of program, sustainability within individuals, and ownership of the reform that we found were necessary for systematic and long-lasting change.
We also adopted and adapted this multidimensional definition of scaling so that it was well matched to a central goal of our reform, the longitudinal development of complex reasoning in science. curricular reform program concentrating on sustained, coordinated fostering of basic skills intertwined with gradual development of complex thinking across time and topic , . With an ambitious reform goal of fostering complex thinking in science among inner city middle school students, we focused on consistent curricular supports that could guide the development of complex skills over longer amounts of time. Without scaffolding that is consistent and that builds productively in subsequent curricular units, we believe our goal of fostering critical thinking and staying true to our reform ideals would not be possible.

22. For more information: www.biokids.umich.edu
Our notes from Detroit conclude with the idea that as scaling research grows and expands beyond studies focusing largely on numbers, attention should be given to work that is more theoretically-grounded, more methodologically-sound, and that challenges and deepens our understanding of key terms such as sustainability and fidelity. In our story, we were motivated by a desire to systematically study the nature of the impact and changes we were motivating, as we were encouraging the ability of our reforms to have longitudinal impact within individuals over time, particularly those individuals not well represented by many current studies. We encourage research that challenges existing systems of accountability towards concrete evidence and comprehensive assessment to more accurately document what is occurring and why. While work in this area is ongoing, our research approach of coordinated systems, e.g. the coordination of our curriculum units containing imbedded critical thinking scaffolds with our coordinated assessments built on sound theory in educational measurement, holds promise for detailed and comprehensive evidence of scaling research focused on ambitious learning goals.