1. Evaluation of NSF’s Research and Evaluation on Education in Science and Engineering (REESE) program Challenges and Strategies Joy Frechtling, Westat John Wells, Westat Dan Heck, Horizon Research Xiaodong Zhang, Westat AEA 2011

2. The Challenge of Evaluating a Complex Research Program Not an intervention Not a multi-site study Goals broader; standards or benchmarks ill-defined or incomplete

3. Goals (according to solicitation NSF 10-586) The Research and Evaluation on Education in Science and Engineering (REESE) program seeks to advance research at the frontiers of STEM learning, education, and evaluation, and to provide the foundational knowledge necessary to improve STEM teaching and learning at all educational levels and in all settings

4. Goals (continued) The goals of the REESE program are: 1. to catalyze discovery and innovation at the frontiers of STEM learning, education, and evaluation; 2. to stimulate the field to produce high quality and robust research results through the progress of theory, method, and human resources; and 3. to coordinate and transform advances in education, learning research, and evaluation.

5. Evaluation Questions from NSF To what extent is the REESE program catalyzing discovery and innovation in research in STEM learning, education, and evaluation? How effective is the REESE program in stimulating STEM fields to produce high- quality, rigorous research studies (with rigor being appropriately defined to reflect the phase of the Cycle of Research and Development that they address)?

6. Evaluation Questions from NSF (continued) How effective is the REESE program in supporting the accumulation of knowledge in STEM education at all levels and in all settings? How effective is the REESE program in promoting interdisciplinary research in pursuit of national questions of STEM teaching and learning? In the request for proposals they also—quite rightly—identified the importance of identifying alternative interpretations of findings that emerge.

7. Challenges Posed by the REESE Portfolio Defining a REESE award Heterogeneity of awards  Target population—Pre-K, Secondary, Informal Settings  Discipline—Science, Mathematics, Cognitive Sciences, Research Methods  Research design—Causal, Descriptive, Synthesis  Content of problem addressed

8. Challenges (cont) Varied status in Cycle of R & D

9. Challenges (continued) Short time since program inception Attribution of outcomes to REESE  Separating impacts of REESE funding from those of other funding sources  Lack of a comparison group, benchmarks in some areas Coordination with other data collection efforts

10. Multi-faceted Approach Using multiple methods—Surveys, Delphi, Case Studies, Bibliometrics, Social Network Analyses Exploring extensions of existing methodologies—KMD extension to syntheses Descriptive, with “fuzzy” RDD

11. For Example—for Question 1 Is the REESE program identifying and funding research believed by the field to be important with regard to discovery and innovation in STEM teaching and learning? Delphi panel—what are the hot issues? Review of solicitations—what is REESE asking for? Review of portfolio of funded projects—what is REESE funding? Review of a sample of unfunded projects— what is being proposed more broadly?

12. Question 1 Example (cont.) Survey of PIs (funded and unfunded proposals)—how is the REESE project situated in researchers’ broader research program?

13. Current status First advisory board meeting held Evaluation plan approved by NSF PI survey in cognitive testing Developing OMB package Review of funded portfolio completed; report being developed Delphi beginning Analyses of rigor of publications underway

14. Looking Ahead—New Challenges Practical—getting it all done Conceptual—Putting the pieces together