1. Sharing and Communicating IERI:
Addressing the Invisible Challenge
Nancy Butler Songer
The University of Michigan
Abstract
How do the concepts of fidelity and sustainability shift when scaling is focused on EFFECTING CHANGE in urban schools through a focus on the URBAN DISTRICTS? Building from current multidimensional definitions of scaling (e.g. Coburn, 2003), this paper discusses major challenges to implementing interventions with fidelity in many simultaneous and diverse classroom contexts, and then presents revised definitions of key terms to emphasize a measure of sustained impact on individuals. Key points are illustrated with examples from our multi-year, coordinated curricular/assessment system designed to afford comprehensive evidence of longitudinal impact across time and topic.

2. Disseminating Research: Insights from Mature IERI Projects
What obstacles have you come across in trying to deliver and share your information? With what audience(s) do you have trouble?
How do you overcome these obstacles?
What outlets have you used (e.g. media, print) to communicate outside the academy?
What has been the result of your efforts?

3. BioKIDS: Kids’ Inquiry of Diverse Species
Nancy Songer, School of Education, Univ. of Michigan
Phil Myers, Department of Zoology, Univ. of Michigan
Michelle Astolfi
Cesar Delgado
Tanya Dewey
Karen Dvornich
Roger Espinosa
Mike Giordano
Amelia Gotwals
George Hammond
Pier Sun Ho
Anne Huber
Ben Kelcey
Hee-Sun Lee
Silvia Lee
Scott McDonald
Tom Vlajkov

4. PADI: Principled Assessment Design for Inquiry
Geneva Haertel, SRI International
Robert Mislevy, University of Maryland
Sub-Contract Principal Investigators:
Nancy Songer, University of Michigan
Mark Wilson, University of California, Berkeley
Kathy Long, Lawrence Hall of Science, Berkeley

6. Educational Challenge
Between fourth and eighth grade, American students’ achievement and understandings of complex science decline relative to their peers internationally.
Declines are pronounced for urban students, and in many cities standardized test scores remain among the nation’s lowest.

7. Educational Challenges Within High-Poverty Detroit Public Schools
Extreme pressure to perform well on high stakes tests
Extreme levels of student, teacher mobility
Nearly double national poverty rate: 70% free/reduced lunch
Class sizes often over 30
~ No substitutes, outside recess
Virtually no experience with complex reasoning in science or digital resources
Focus Audience: ~30,000 high-poverty Detroit middle school students; 95% African American, Hispanic or Mixed Race
little agreement exists between programs on what components should be brought to scale or what the profile of growth should be. In other words, does “bringing an intervention to scale” refer to documentation of widespread use among a thousand classrooms at any one time or are consideration of depth of implementation and sustainability important?
Regardless of focus, most researchers would agree that the current body of research on scaling curricular reforms includes an under emphasis of research approaches and sound experimental methods that might result in clear evidence of cognitive and epistemological outcomes to evaluate the effectiveness of the intervention in the many locations.
In our case, we are bringing to scale several standards-based middle school curricular programs shown to be successful as measured by student learning outcomes. Key ideas in our scaling efforts throughout the eight years include…

8. Addressing The Challenge
Research Areas
Guiding Complex Reasoning in Science
Assessment Systems for Science Inquiry (with PADI)
From Digital Resource to Cognitive Tool
Focus on Persistence Within Individuals (scaling)
Empirical Results on Longitudinal Learning

9. Which programs should Congress allocate money to, and why?”
“A tremendous amount of money has been allocated to educational research….
What have you learned?
Which programs work?
Which programs should Congress allocate money to, and why?”

10. Not my educational challenge!
“A tremendous amount of money has been allocated to educational research….
What have you learned?
Which programs work?
Which programs should Congress allocate money to, and why?
Not my educational challenge!

11. Concrete empirical evidence
Visible Challenges
Concrete empirical evidence
Concrete empirical evidence within the entire Detroit Public School District
Logistics
Recruitment
Barriers to randomization
little agreement exists between programs on what components should be brought to scale or what the profile of growth should be. In other words, does “bringing an intervention to scale” refer to documentation of widespread use among a thousand classrooms at any one time or are consideration of depth of implementation and sustainability important?
Regardless of focus, most researchers would agree that the current body of research on scaling curricular reforms includes an under emphasis of research approaches and sound experimental methods that might result in clear evidence of cognitive and epistemological outcomes to evaluate the effectiveness of the intervention in the many locations.
In our case, we are bringing to scale several standards-based middle school curricular programs shown to be successful as measured by student learning outcomes. Key ideas in our scaling efforts throughout the eight years include…

12. Additional Invisible Challenges
Concrete empirical evidence
Concrete empirical evidence within the entire Detroit Public School District
Logistics
Recruitment
Barriers to randomization
Language
Presentation
little agreement exists between programs on what components should be brought to scale or what the profile of growth should be. In other words, does “bringing an intervention to scale” refer to documentation of widespread use among a thousand classrooms at any one time or are consideration of depth of implementation and sustainability important?
Regardless of focus, most researchers would agree that the current body of research on scaling curricular reforms includes an under emphasis of research approaches and sound experimental methods that might result in clear evidence of cognitive and epistemological outcomes to evaluate the effectiveness of the intervention in the many locations.
In our case, we are bringing to scale several standards-based middle school curricular programs shown to be successful as measured by student learning outcomes. Key ideas in our scaling efforts throughout the eight years include…

13. Addressing The Challenge: Language
Research Areas
Guiding Complex Reasoning in Science
Assessment Systems for Science Inquiry
From Digital Resource to Cognitive Tool
Focus on Persistence Within Individuals
Empirical Results on Longitudinal Learning

14. Cognitive Tool CyberTracker for field-based data entry
Enter in the field via PDA
Visual Maps
Icon-based entry

16. Addressing The Challenge: Language
Research Areas
Guiding Complex Reasoning in Science
Assessment Systems for Science Inquiry
From Digital Resource to Cognitive Tool
Focus on Persistence Within Individuals
Empirical Results on Longitudinal Learning
Blah Blah
Blah Academic Blah….

17. Which programs should Congress allocate money to, and why?”
“A tremendous amount of money has been allocated to educational research….
What have you learned?
Which programs work?
Which programs should Congress allocate money to, and why?”

18. Coalition for National Science Funding Event
“The U.S. faces competition from new centers of innovation. For example, the U.S. share of world-wide high-tech exports has been in a 20-year decline. From 1980 until 2001, the U.S. share fell from 31 percent to 18 percent. At the same time, the global share for China, South Korea, and other emerging Asian countries increased from just 7 percent to 25 percent.”

19. “ Since 1980, the number of Science and Engineering positions in the U
“ Since 1980, the number of Science and Engineering positions in the U.S. has grown at almost five times the rate of the civilian workforce as a whole. However the number of Science and Engineering degrees earned by U.S. citizens is growing at a much smaller rate.”

20. What can your IERI project tell us about these issues?

21. Addressing the Invisible Challenge
Repackage your IERI findings into simple, persuasive language

22. Coalition for National Science Funding
“Examples of major innovations spawned by NSF research include…
In the mid-1990s, two NSF-supported graduate students discovered a better way of searching data on the Internet and used their idea to found Google, Inc., a $50-billion company that employs more than 2,000 people.
Fundamental research supported by NSF led to the development of Magnetic Resonance Imaging (MRI), which is used widely to detect cancer and internal tissue damage and is predicted to be a $110-billion industry by 2007.”

23. Addressing the Invisible Challenge
Our results suggest that understanding science and math involves:
increased time on topics
systematic guidance in developing more complex ideas, and
an ability to revisit and deepen understandings in a systematic manner.

24. Addressing the Invisible Challenge
Our results suggest that understanding science and math involves: increased time on topics
Both groups made significant improvement on declarative knowledge
However groups using our full program made much larger gains on higher order thinking
While less time on topics does not have a large impact on students’ learning of facts, it does make a large impact on students’ ability to reason about complex scientific problems
“Wide reading beats rereading”

25. Addressing the Invisible Challenge
Repackage your IERI findings into simple, persuasive language
Examples of major findings spawned by IERI research include:
1. [Yours]
2. [The person next to you]

26. Addressing The Challenge: Presentation
What can your IERI project tell us about these issues?
Who is your audience?

27. Conclusion: An Educational Researchers’ Perfect Day
It is the year 2008 and your have reams and reams of amazing, methodologically-sound research results
You write the next grant, and they fund it, providing you with MORE MONEY THAN YOU ASKED FOR
Who, besides your partner and your mom, can describe your findings?
Is this your educational challenge?
Our research scaling of successful, small-scale educational interventions with an explicit focus on maintaining program fidelity across a range of scaled locations. Early on, while our scaling efforts were regarded as successful on many criteria, we questioned our ability to make sound conclusions about the character of the reform in the distant locations. In addition, we questioned the ability of the reform to remain in place once we were no longer supporting it, A focus on longer term intervention impact within individuals motivated our shift to those more at-risk for school failure including children in inner city schools. Finally, we address reform challenges at the district level These shifts in research focus reflect an explicit concentration on the concepts of fidelity and sustainability relative to scaling research. In our work, we have adopted a multidimensional definition of scaling not dissimilar from that championed by Coburn . We adapted a fidelity and sustainability-driven definition of scaling because we felt that scaling research focusing primarily on quantitative measures of impact disallowed a sound examination of the depth, sustainability of program, sustainability within individuals, and ownership of the reform that we found were necessary for systematic and long-lasting change.
We also adopted and adapted this multidimensional definition of scaling so that it was well matched to a central goal of our reform, the longitudinal development of complex reasoning in science. curricular reform program concentrating on sustained, coordinated fostering of basic skills intertwined with gradual development of complex thinking across time and topic , . With an ambitious reform goal of fostering complex thinking in science among inner city middle school students, we focused on consistent curricular supports that could guide the development of complex skills over longer amounts of time. Without scaffolding that is consistent and that builds productively in subsequent curricular units, we believe our goal of fostering critical thinking and staying true to our reform ideals would not be possible.

28. For more information: www.biokids.umich.edu
Our notes from Detroit conclude with the idea that as scaling research grows and expands beyond studies focusing largely on numbers, attention should be given to work that is more theoretically-grounded, more methodologically-sound, and that challenges and deepens our understanding of key terms such as sustainability and fidelity. In our story, we were motivated by a desire to systematically study the nature of the impact and changes we were motivating, as we were encouraging the ability of our reforms to have longitudinal impact within individuals over time, particularly those individuals not well represented by many current studies. We encourage research that challenges existing systems of accountability towards concrete evidence and comprehensive assessment to more accurately document what is occurring and why. While work in this area is ongoing, our research approach of coordinated systems, e.g. the coordination of our curriculum units containing imbedded critical thinking scaffolds with our coordinated assessments built on sound theory in educational measurement, holds promise for detailed and comprehensive evidence of scaling research focused on ambitious learning goals.