Goal 2/ Goal 3 In 2016, no Goal 2s accepted; 2017?

Goal 2: Development & Innovation Goal Development process must be iterative! Develop an innovative intervention (e.g., curriculum, instructional approach, program, or policy) OR improve existing education interventions AND collect data on its usability, feasibility, and fidelity of implementation in actual education settings AND collect pilot data on student outcomes Development & Innovation 4 years, $1,500,000

Goal 3: Efficacy & Replication Goal Evaluate whether or not a fully developed intervention is efficacious under limited or ideal conditions OR Gather follow-up data examining the longer-term effects of an intervention with demonstrated efficacy Replicate an efficacious intervention varying the original conditions Conduct retrospective analysis of secondary data collected in the past Efficacy & Replication Follow-up study 4 years, $3,500,000 3 years, $1,200,000

Intervention Research Proposal Components: IES Goal 2 or 3 Formatted Abstract (1 page) Major Sections of Research Narrative (25 pages) Project Significance Research Aims Intervention Theory of Change Rationale Research Plan Sample and setting Research design Timeline and procedures Measures proximal and distal outcomes fidelity of intervention implementation & control group practices key moderators or mediators Data analytic plan Detailed power analysis Cost analysis Resources (to conduct research AND disseminate intervention) Personnel Appendices Response to Reviews; Measures; Intervention Materials; Letters of Support; Data Management Plan

Goal 2 vs. Goal 3 Both Require: A plausible rationale that the intervention is needed; reason to believe it has advantages over what’s currently proven and available A well-specified intervention model basis in theory and prior research identified target population specification of intended outcomes/effects “theory of change” explication of what it does and why it should have the intended effects for the intended population Clarity about the relevant counterfactual– what it is supposed to be better than

Goal 2 vs. Goal 3 The Goal 3 also requires (and the Goal 2 can help provide): Evidence that the intervention is ready to deliver: Complete manual, with detailed content and instructions for implementation Finalized and ready-to-go materials and training procedures Evidence that the intervention can be delivered as intended and implemented well enough in practice to plausibly have effects Completed implementation guides, procedures, measures Some evidence that it can produce the intended effects and evidence of anticipated effect size

Common Aims for Goal 2 Develop the intervention curriculum and associated materials. Develop standardized training procedures and implementation guidelines. Design and test a supervisory or professional development model to support high-fidelity implementation. Use iterative field tests to refine intervention & estimate effects. Assess acceptability to different audiences. Collect input/feedback to adapt program materials for diverse samples. Assess measures for their sensitivity to change. Evaluate the feasibility of the sampling, recruitment, screening, enrollment procedures, and program delivery

Common Aims for Goal 3 Aim 1: Test the efficacy of the intervention using a rigorous, randomized-controlled design. Aim 2: Test hypothesized mechanisms of change; explore change processes. [mediation] Aim 3: Examine variations in intervention implementation/engagement/acceptability or impact and assess moderators that may account for these variations. [moderation]

Significance Section: Content to Cover Describe the nature of the need addressed: what and for whom is this intervention intended Identify the proximal and distal targets of the intervention and why are they targeted Articulate the theory and evidence linking the intervention approach to change in these targeted outcomes Be sure to convey: The key factors or components most distinctive in this intervention and how they differ from the usual practice/control condition TIP: It is always a good idea to provide a graphic to summarize the logic model of the intervention

Research Plan: Start with an Overview and Timeline *Paragraph summarizing the targeted population and sampling frame and general design framework. Provide a table showing the time-line.

Research Plan: Sample, Setting, Design Sampling must be strategic rather than convenience, with evidence of: cooperative schools, teachers, parents, & administrators willing to participate student sample appropriate in terms of representativeness and size for showing educationally meaningful effects access to students (e.g., for testing), records, classrooms (e.g., for observations) Settings are evaluated In terms of feasibility, cost-effectiveness, and sustainability of the intervention Generalizability of findings from the targeted settings Randomization and design descriptions should specify: Level of randomization Use of hierarchical designs (accounting for classrooms/schools) Protections against threat of intervention contamination Control group comparison (what is “usual practice”)

Intervention Description Highlight key components that make the intervention distinctive Describe the mechanisms of action of the key components Illustrate how the components will be operationalized Use graphics and tables to make this information easily accessible

Domain of Social-Emotional Competencies   Domain of Social-Emotional Competencies Target Skill Domains Prosocial Skills Help, share, cooperate Emotion Knowledge Label feelings, empathy Self-control Follow rules, inhibit aggressive impulses Social Problem-Solving Identify problems, generate solutions. Curriculum Materials (Manuals) PATHS: Friendship lessons & modeling stories PATHS: Feeling lessons, feeling faces PATHS: Turtle technique (stop, calm down, say the problem & feelings) PATHS: Problem-solving sequence & modeling stories Teaching Strategies (BKC) Positive class management: rules, routines, praise Emotion coaching: modeling & reflective listening Prompting use of the turtle technique, using induction Using problem-solving dialogue; facilitating communication Domain of Language and Emergent Literacy Skills Oral Language Vocabulary Narrative Temporal sequence, cause and effect Phonological Sensitivity Sequencing sounds Print Awareness Letter knowledge and letter sounds Dialogic Reading: Targeted vocabulary Story review Sound Games: Listening, sequencing words & syllables Alphabet Center: Multi-media exposure to letters and letter sounds Intentional repeated exposure to target vocabulary Rich language use, sensitive responding, questions, expansions Sequence developmentally; pace for mastery Monitoring knowledge acquisition; pacing to support mastery

Levels of Measurement fidelity of intervention implementation & control group practices proximal outcomes distal outcomes key moderators/mediators control variables/confounders

Measuring Intervention Fidelity Adherence/compliance: Evidence that the intervention content was delivered as intended Quality of the delivery: Evidence that the key processes (theory-based characteristics) of intervention delivery occurred as intended Dose/exposure : Degree to which the desired amount of intervention was delivered to and received by participants; Participant responsiveness : Degree to which participants were engaged and responsive; acceptability of intervention to participants Dane & Schneider (1998)

Organize the Measures to Allow for Easy Categorization

Issues in Measure Selection Reviewers often express concern about: Feasibility– time and resources required Respondent burden Developmental appropriateness Sensitivity to change in the intervention Appropriateness for special populations (disabilities, English language learners) Too many measures – How will you reduce? What is the PRIMARY outcome – or how will you know if the intervention worked?

Issues Regarding Plan of Analyses Sophisticated and detailed analytic plans are expected General approaches to data cleaning, reduction, scoring, transforming Detailed plans for how each aim will be analyzed Pay special attention to hierarchical models Provide a very detailed and well-justified power analysis

Newer Elements Cost Assessment Intervention Fidelity Check after Year 1 (and remediation plan) Dissemination Plan/Resources Data Management Plan

Other Things to Think About Most review committees have mixed experts: Content area specialists (some with no intervention experience/knowledge) Methodological specialists Intervention specialists (may have a strong theoretical affiliation) Try to have your proposal read by a mix of folks who can represent these different perspectives

Other thoughts, questions, suggestions?