KATEWINTEREVALUATION.com Education Research 101 A Beginner’s Guide for S STEM Principal Investigators

KATEWINTEREVALUATION.com Welcome and Overview Structure of today’s webinar About 30 minutes of content and time for Q&A Quick overview of educational research Brief synopsis of the Common Guidelines Pulse checks (interaction points) to gauge engagement About me

KATEWINTEREVALUATION.com Objectives Increase awareness of the “Common Guidelines for Education Research and Development,” including: The 3 broad categories of research The 6 general types of research, with their: Purpose (Very) Abbreviated theoretical and empirical justifications Abbreviated guidelines for evidence produced Guidelines for external review and feedback

KATEWINTEREVALUATION.com What is Education Research? Research that uses various methodologies to address educational processes and outcomes Research that provides the evidence behind “evidence- based” practices/strategies/interventions in education Increasingly commonly by following established guidelines for rigor and significance set forth by IES and NSF, including the What Works Clearinghouse (WWC) standards

KATEWINTEREVALUATION.com Interaction #1 What was your background with the Common Guidelines for Education Research and Development PRIOR to this webinar?

KATEWINTEREVALUATION.com Disclosure In *many* places, this presentation draws text verbatim from the Common Guidelines. When you have time, I highly encourage you to find, bookmark, and review the full document from IES and NSF.

KATEWINTEREVALUATION.com Assumptions behind the Common Guidelines While the 6 research types generally follow a sequence of development: Knowledge development is not linear and can be informed from various places along the continuum Investigations can skip “phases” (MOOCs make large-scale testing possible without prior small-scale testing) Single studies may include aspects of more than one research type

KATEWINTEREVALUATION.com Foundational Research & Early-Stage or Exploratory Research Contribute to Core Knowledge in education, including basic understanding of teaching and learning (i.e., cognition); components and processes involved in learning and instruction; the operation of education systems; and, models of systems and processes Include: Research Type #1: Foundational Research Research Type #2: Early-Stage or Exploratory Research

KATEWINTEREVALUATION.com Design and Development Research Develops solutions to achieve a goal related to education or learning, such as improving student engagement or mastery of a set of skills Represents Research Type #3

KATEWINTEREVALUATION.com Impact Studies Contribute to evidence of impact, generating reliable estimates of the ability of a fully-developed intervention or strategy to achieve its intended outcomes Includes: Research Type #4: Efficacy Research Research Type #5: Effectiveness Research Research Type #6: Scale-Up Research

KATEWINTEREVALUATION.com Interaction #2 Which statements are TRUE?

KATEWINTEREVALUATION.com Research Type #1: Foundational Research Provides the fundamental knowledge that may contribute to improved learning and other relevant education outcomes Seeks to test, develop, or refine theories of teaching or learning and may develop innovations in methodologies and/or technologies that will influence and inform research and development in different contexts

KATEWINTEREVALUATION.com Foundational Research (cont.) Purpose: To advance the frontiers of education and learning; develop and refine theory and methodology; and provide fundamental knowledge about teaching and/or learning Justifications: Should address important research problems or questions related to education and learning; should outline the theoretical and empirical bases for the project and why it is necessary for core knowledge generation

KATEWINTEREVALUATION.com Foundational Research (cont.) Evidence to be produced: Advances in theory, methodology, and/or understanding of important constructs in education and learning Findings that could serve as the basis for future studies  May examine phenomenon without establishing explicit links to education outcomes

KATEWINTEREVALUATION.com Research Type #2: Early-Stage or Exploratory Research Examines relationships among important constructs in education and learning to establish logical connections that may form the basis for future interventions or strategies to improve education outcomes Connections are usually correlational rather than causal

KATEWINTEREVALUATION.com Early-Stage or Exploratory Research (cont.) Purpose: To investigate approaches to education problems to establish the basis for design and development of new interventions or strategies, and/or to provide evidence for whether an established intervention or strategy is ready to be tested in an efficacy study Justifications: Should address a practical education problem and provide a compelling case for how the study will inform the development, improvement, or evaluation of education programs, policies, or practices

KATEWINTEREVALUATION.com Early-Stage or Exploratory Research (cont.) Evidence to be produced: Empirical evidence regarding associations between malleable factors and student learning outcomes A well-specified framework behind these associations Justification for proceeding to a Design and Development project, an Efficacy study, or if additional foundational or early-stage or exploratory research is needed first

KATEWINTEREVALUATION.com Interaction #3 Who has engaged in a study or project that would be considered foundational research or early-stage or exploratory research?

KATEWINTEREVALUATION.com Research Type #3: Design and Development Research Draws on existing theory and evidence to design and iteratively develop interventions or strategies (solutions) May include pilot tests of fully developed interventions to determine if they achieve their intended outcomes under various conditions Results could lead to additional work to better understand the foundational theory behind the results or could indicate that the intervention or strategy is sufficiently promising to warrant more advanced testing

KATEWINTEREVALUATION.com Design and Development Research (cont.) Purpose: To develop new or improved interventions or strategies to achieve well-specified learning goals or objectives, including making refinements on the basis of small-scale testing. Typically entails four components Justifications: Explain why the proposed project has the potential to improve learning or education outcomes or increase efficiencies in the education system or institutional setting beyond what current practice provides

KATEWINTEREVALUATION.com Design and Development Research (cont.) Evidence to Be Produced: Fully developed version of the proposed design/research A well-specified theory of action Descriptions of the major iterations and the resulting evidence to support or question key assumptions about the theory of action Description and empirical evidence of the adjustments that resulted from design testing Measures with evidence of technical quality for assessing the intervention and data demonstrating the project’s success Pilot data on the intervention’s promise for generating the intended beneficial learner outcomes

KATEWINTEREVALUATION.com Interaction #3 Who has engaged in a study or project that would be considered design and development research?

KATEWINTEREVALUATION.com Research Type #4: Efficacy Research Allows for testing of a strategy or intervention under “ideal” circumstances May limit the investigation to a single population

KATEWINTEREVALUATION.com Efficacy Research (cont.) Purpose: To determine whether an intervention or strategy can improve outcomes under “ideal” conditions Justifications: Must compelling explain why testing the impact under “ideal” circumstances is warranted, with evidence from prior studies/implementation that supports the value of the intervention and the need for additional testing

KATEWINTEREVALUATION.com Efficacy Research (cont.) Evidence to Be Produced (same for all Impact Studies): Final report as outlined by the What Works Clearinghouse (WWC) Reliable estimates of the intervention’s average impact, with estimates by sample subgroups as appropriate Implementation details for the intervention and counterfactual condition, with connections to outcomes as appropriate Implications of the findings for theory of action: Replication details for projects demonstrating favorable impacts Otherwise, possible reasons why no favorable impact was demonstrated

KATEWINTEREVALUATION.com Research Type #5: Effectiveness Research Examines the effectiveness of a strategy or intervention under circumstances that would typically occur

KATEWINTEREVALUATION.com Effectiveness Research (cont.) Purpose: To estimate the impacts of an intervention or strategy when implemented under conditions of routine practice Justifications: Must compelling explain why testing the impact under routine circumstances is warranted, with evidence from prior studies/implementation that supports the value of the intervention and the need for additional testing

KATEWINTEREVALUATION.com Research Type #6: Scale-Up Research Examines effectiveness in a wide range of populations, contexts, and circumstances, without substantial developer involvement in implementation or evaluation (i.e., under “normal” conditions)

KATEWINTEREVALUATION.com Scale-Up Research (cont.) Purpose: To estimate the impacts of an intervention or strategy when implemented under conditions of routine practice AND across a broad spectrum of populations and settings Justifications: Must compelling explain why testing the impact under routine circumstances and with a broad sample is warranted, with evidence from prior studies/implementation that supports the value of the intervention and the need for additional testing

KATEWINTEREVALUATION.com Guidelines for External Review and Feedback All research types should be subject to a series of external, critical reviews of design, implementation, analysis and reporting External reviews should be independent and rigorous Appropriate review activities include: Peer review of proposed projects Ongoing monitoring and review by grant-making personnel External review panels or advisory boards Third-party evaluator Peer review of publications and/or conference presentations

KATEWINTEREVALUATION.com Interaction #4 Who has engaged in a study or project that would be considered an impact study?

KATEWINTEREVALUATION.com Recap Three categories: 1.Foundational research and early-stage or exploratory research 2.Design and development research 3.Impact studies: Efficacy research Effectiveness research Scale-up research OK to mix different approaches in a single project, if it makes sense to do so

KATEWINTEREVALUATION.com Questions or Comments? Raise your hand to be un-muted, or Type your question in the question box Please let us know if you would be interested in additional webinars on the specific research types, including aspects related to design plans and WWC standards