Preliminary Results – Not for Citation Investing in Innovation (i3) Fund Evidence & Evaluation Webinar April 25, 2012 Note: These slides are intended as guidance only. Please refer to the official documents published in the Federal Register.

Preliminary Results – Not for Citation Agenda  Evidence Standards—eligibility requirement Requirements Review process  Independent Evaluation—program requirement Guidance on qualities of high-quality evaluation plans  Questions & answers Live—submission via the Webinar chat function Post-webinar: to 2

Preliminary Results – Not for Citation Evidence Standards— Eligibility Requirement 3

Preliminary Results – Not for Citation Evidence Eligibility Requirements Are Specific to the Type of Grant 4 An application must meet the applicable evidence requirements before an award is made Any application failing to meet the applicable evidence requirements is not eligible for a grant award, regardless of its score on the Selection Criteria Any application failing to meet the applicable evidence requirements will not be considered for a different type of i3 grant award

Preliminary Results – Not for Citation Scale-up Grants: Require “Strong Evidence” of Effectiveness  High internal validity of the evidence Studies designed and implemented in ways that support causal inference  High external validity of the evidence Studies based on a sufficient representation of participants and settings that the findings support scaling  Minimum size of evidence base More than one well-designed and well-implemented experimental or quasi-experimental study OR One large, well-designed and well-implemented multi-site randomized controlled trial 5

Preliminary Results – Not for Citation Caution: Not All Associations Support Causal Inferences 6 (Mis-) Interpretive Statement: Based on the evidence, early care & education programs should reduce pupil teacher ratios in order to lower the expulsion rate. Problem: There are competing explanations for why expulsions increase with increases in pupil-teacher ratios

Preliminary Results – Not for Citation Validation Grants: Require “Moderate Evidence” of Effectiveness  Validity of the evidence High internal validity and moderate external validity or High external validity and moderate internal validity  Minimum size of evidence base At least one well-designed experimental or quasi-experimental study May have small sample sizes OR may fail to demonstrate equivalence between the intervention & comparison groups, but has no other major flaws or A correlational study with strong statistical controls for selection bias and for discerning the influence of other potential confounds 7

Preliminary Results – Not for Citation Development Grants: Require Evidence to Support the Proposed Intervention  Theoretical support for the proposed practice, strategy, or program that is based on research findings or reasonable hypotheses, including related research or theories in education and other sectors and  Some empirical evidence of the promise of the proposed practice, strategy or program based on prior implementation and evaluation of something similar, albeit potentially on a limited scale or in a limited setting 8

Preliminary Results – Not for Citation Evidence Standards— Eligibility Review Process 9

Preliminary Results – Not for Citation Responsibility for the Evidence Eligibility Reviews  The Institute of Education Sciences (IES) conducts the reviews and reports their findings to OII  IES uses What Works Clearinghouse (WWC) evidence standards to judge the causal validity of the evidence IES uses experienced WWC consultants  Reviews are limited to evidence specifically cited in the application as that supporting the eligibility requirement 10

Preliminary Results – Not for Citation Evidence Citation Tips  Provide a list of full citations for the evidence you want ED to include in the evidence eligibility review in Appendix D of your application  If possible, also append the evidence itself (if space allows)  Include in the list only those citations that are relevant to the intervention being proposed  For Scale-up and Validation applications, include in the list only those citations that are primary analyses of the effect of the intervention being proposed 11

Preliminary Results – Not for Citation Evidence Reviews: Scale-up Applications Two focal questions: 1. Does the evidence include a sufficient number and quality of studies? (1) More than one well-designed and well-implemented experimental study or well-designed and well-implemented quasi-experimental study? or (2) One large, well-designed and well-implemented multi-site randomized controlled trial? 2. Does evidence include a reasonable representation of the kinds of participants and settings proposed to receive the practice, strategy, or program under the Scale-up grant to expect the results would generalize? 12

Preliminary Results – Not for Citation Evidence Reviews: Validation Applications Question 1 of 3: What is the internal validity of the evidence of effectiveness? High = At least one well-designed and well-implemented experimental or quasi-experimental study supporting the effectiveness of the practice, strategy, or program Moderate = (a) At least one well-designed experimental or quasi- experimental study, with no major flaws except possibly failure to demonstrate equivalence between the intervention & comparison groups or (b) At least one correlational study with strong statistical controls for possible selection bias Low = No study with high or moderate internal validity 13

Preliminary Results – Not for Citation Evidence Reviews: Validation (Cont’d) Question 2 of 3: What is the external validity of the evidence? High = Evidence pertains to the kinds of participants and settings that are the focus of the Validation Grant application Moderate = Evidence pertains to participants and settings that overlap with but may be more limited than those that are the focus of the application Low = No study with high or moderate external validity 14

Preliminary Results – Not for Citation Evidence Reviews: Validation (Cont’d) Question 3 of 3: Does the evidence of effectiveness provided meet one of the following conditions: 1. High internal validity and at least moderate external validity (as defined above)? or 2. High external validity and at least moderate internal validity (as defined above)? 15

Preliminary Results – Not for Citation Evidence Review: Development Applications Two focal questions: 1. Is there a well-reasoned, theoretically supported rationale for the proposed practice, strategy, or program based on research findings or reasonable hypotheses? 2. Is there empirical evidence of the promise that the proposed practice, strategy, or program (or one similar to it) will improve student outcomes (albeit, possibly, based on implementation on a limited scale or in a limited setting compared with the participants and settings proposed to receive the treatment under the Development grant)? 16

Preliminary Results – Not for Citation What Works Clearinghouse Evidence Standards Used to Judge Internal Validity of Evidence 17

Preliminary Results – Not for Citation How Does the WWC Assess Research Evidence?  Type of design: does the study design allow us to draw causal conclusions?  Strength of data: does the study focus on relevant outcomes and measure them appropriately?  Adequacy of statistical procedures: are the data analyzed properly? 18

Preliminary Results – Not for Citation WWC Standards Apply to Causal Designs Eligible Designs Randomized controlled trials (RCTs) Quasi-experimental designs (QEDs) Potentially Eligible Designs Regression discontinuity (RDD) Single case (SCD) 19 Ineligible Designs Anecdotes and testimonials Case studies Descriptive Correlational

Preliminary Results – Not for Citation Key Elements of the WWC RCT/QED Standards Randomization Attrition Equivalence Meets Evidence Standards Meets Evidence Standards with Reservations Does Not Meet Evidence Standards Low No Yes High Yes No 20

Preliminary Results – Not for Citation Caution 1: RCTs with high sample attrition must demonstrate baseline equivalence 21 Randomization? Attrition? Baseline Equivalence? Meets Evidence Standards Meets Evidence Standards with Reservations Does Not Meet Evidence Standards Low No Yes High Yes No

Preliminary Results – Not for Citation Standards Account for Overall and Differential Attrition Rates Overall AttritionTreatment-Control Differential Attrition 15%10.7% 20%10.0% 30%8.2% 22 Sample maximum attrition thresholds for meeting WWC evidence standards

Preliminary Results – Not for Citation Caution 2: All QEDs must demonstrate baseline between the treatment and control groups 23 Randomization? Attrition? Baseline Equivalence? Meets Evidence Standards Meets Evidence Standards with Reservations Does Not Meet Evidence Standards Low No Yes No

Preliminary Results – Not for Citation Baseline Equivalence Standard Equivalence must be demonstrated on the analytic sample 24 Baseline Difference: Treatment v. Control No Adjustment Needed Adjustment Needed Does not Meet Standards <.05 sd√ sd√ >.25 sd√

Preliminary Results – Not for Citation Other Criteria  Studies must measure impacts on relevant outcomes  Outcome measures must be reliable  Outcomes must not be over-aligned with the intervention  There is not a confound between the treatment condition and one or more other factors that also could be the cause of differences in outcomes between the treatment and comparison groups 25

Preliminary Results – Not for Citation Questions & Answers Please submit your questions on evidence eligibility requirements via the Webinar chat function now 26

Preliminary Results – Not for Citation Independent Evaluation-- Program Requirement and Guidance 27

Preliminary Results – Not for Citation Evaluation Requirements 28 MUST All i3 Grantees MUST 1.Conduct an independent project evaluation 2.Cooperate with technical assistance provided by the Department or its contractors 3.Share the results of any evaluation broadly 4.Share the data sets for Validation and Scale-up evaluations 1.Conduct an independent project evaluation 2.Cooperate with technical assistance provided by the Department or its contractors 3.Share the results of any evaluation broadly 4.Share the data sets for Validation and Scale-up evaluations MUST * Note: The quality of an applicant’s project evaluation is also a selection criterion.

Preliminary Results – Not for Citation Selection Criterion D: Quality of Project Evaluation 29 “ The extent to which the methods of evaluation will provide high-quality implementation data and performance feedback, and permit periodic assessment of progress toward achieving intended outcomes.” Scale-up, Validation, and Development “ The extent to which the proposed project plan includes sufficient resources to carry out the project evaluation effectively.” Scale-up, Validation, and Development Sufficient Funding to Carry Out Evaluation Understanding of Implementation and Intermediate Outcomes

Preliminary Results – Not for Citation Selection Criterion D: Quality of Project Evaluation (Cont’d) 30 “The extent to which the evaluation will provide sufficient information about the key elements and approach of the project so as to facilitate replication or testing in other settings.” Scale-up and Validation “The extent to which the proposed evaluation will provide sufficient information about the key elements and approach of the project to facilitate further development, replication, or testing in other settings.” Development Generates Information to Support Follow- on Scaling or Other Activities

Preliminary Results – Not for Citation Notes on Selection Criterion D. Quality of Project Evaluation 31 “ The extent to which the methods of evaluation will include a well-designed experimental study, or if a well-designed experimental study of the project is not possible, the extent to which the methods of evaluation will include a well-designed quasi-experimental design.” Scale-up “The extent to which the methods of evaluation will include a well-designed experimental study or a well-designed quasi-experimental study.” Validation Design & Methodology for the Evaluation

Preliminary Results – Not for Citation Guidance on Qualities of High-quality Evaluation Plans 32

Preliminary Results – Not for Citation Evaluation Goals  Aligned with i3 performance measures  Increase strength of evidence on the impact or promise of i3-supported interventions  i3 performance measures set the expectation for all i3 independent evaluations to provide high-quality implementation data and performance feedback  Other expectations vary by grant type 33

Preliminary Results – Not for Citation Evaluation Goals: Scale-up & Validation Projects  Estimate the impacts of the i3-supported intervention as implemented at scale Expectation they will be “well-designed and well-implemented,” meaning designed, implemented and reported in a manner to meet WWC evidence standards  Provide information on key elements of the intervention for replication or testing in other settings  Reflect changes to the intervention or delivery model  Reflect the nature of sites served, implementation settings, and participants  Document costs/resource use 34

Preliminary Results – Not for Citation Evaluation Goals: Development Projects  Provide evidence of the promise of the practice, strategy, or program for improving student outcomes : Using a research design that provides a comparison for the outcomes of the intervention group (e.g., a randomized controlled trial or strong matched comparison group design) Using valid outcome measures (e.g., following the WWC standards)  Provide information on key elements of the intervention for further development, replication, or testing in other settings 35

Preliminary Results – Not for Citation Evaluation Plan Tips (1 of 6)  Provide a fully developed plan in the application Maximizes the ability to adequately address key questions Minimizes the time spent at the beginning grant specifying details and negotiating revisions with ED Key components of a high-quality evaluation plan  Logic model (What is the intervention? Who is it intended to serve? What outcomes is it expected to produce? How?)  Research questions (What do you want to learn? Who will the results pertain to?)  Proposed methods (What is the sampling frame? How will treatment and comparison groups be formed? What is the minimum detectable effect size? What data collection measures and methods will be used? How will the data be analyzed and reported?)  Coherent links among the above components 36

Preliminary Results – Not for Citation Evaluation Plan Tips (2 of 6)  Provide a well-developed logic model to guide the evaluation plan: Clear description of the i3-supported intervention/scaling mechanism Key elements of the intervention in sufficient detail to guide replication Target population for the innovation Expected pathways through which the innovation is expected to affect intermediate and ultimate outcomes of interest Expected mechanisms of change—e.g., shows how factors such as content, organization, duration, training, and other procedures are expected to lead to the intended outcomes 37

Preliminary Results – Not for Citation Evaluation Plan Tips (3 of 6)  Clarity on key research questions in the evaluation plan Creates shared expectations with partners and ED about what will be learned  Clarity on aspects of the logic model the evaluation will examine  Design (methods, sample, data collection, analysis) appropriate to address the key questions 38

Preliminary Results – Not for Citation Evaluation Plan Tips (4 of 6)  Specify proposed methods For questions about program effectiveness (i.e., causal inference questions):  Rely on experimental methods, when possible NOTE: Challenging to meet WWC evidence standards with quasi- experimental methods (as noted above)  Describe how treatment and control groups will be formed and monitored 39

Preliminary Results – Not for Citation Evaluation Plan Tips (5 of 6)  Summarize data collection and analysis procedures Describe data sources, samples, data collection procedures and timeline, and analysis methods for each research question Ensure critical components of the logic model are represented in the data collection plan  NOTE: ensure measurement of implementation fidelity  NOTE: comparative studies should document the experiences of comparison participants 40

Preliminary Results – Not for Citation Evaluation Plan Tips (6 of 6)  Ensure good match among logic model, research questions, proposed methods, and i3 goals: Sample design should yield a sample representative of the relevant population group for the i3 project & support strong impact analyses (relevant to the intervention as implemented under i3) Analysis plan should be appropriate for addressing the study questions and well-matched to the sample design Plan for reporting the findings should be consistent with the evaluation goals and design 41

Preliminary Results – Not for Citation Questions & Answers Please submit your questions on evaluation via the Webinar chat function now 42

Preliminary Results – Not for Citation Other Important Resources 43 Note: These slides are intended as guidance only. Please refer to the official Notices in the Federal Register. Investing in Innovation Fund Website: (  Notices of Final Revisions to Priorities, Requirements, and Selection Criteria  Application Packages for each competition (includes the respective Notice Inviting Applications)  Frequently Asked Questions What Works Clearinghouse Website: (  Reference Resources, Procedures and Standards Handbook  Quick Review Protocol All questions about i3 may be sent to