1. Characteristics of Studies that might Meet the What Works Clearinghouse Standards: Tips on What to Look For 1

2. Two levels of evidence use WWC standards 2 Evidence Factors STRONG THEORY EVIDENCE OF PROMISE MODERATE EVIDENCE OF EFFECTIVENESS STRONG EVIDENCE OF EFFECTIVENESS Number of Studies n/a (logic model only) At least one At least two Study Findings on a Relevant Outcome n/a Statistically significant, substantively important (0.25 standard deviation or larger) positive association Statistically significant positive impact with no unfavorable and overriding impacts What Works Clearinghouse Standards n/a Meets without reservations (RCT) Meets with reservations (RCT or QED) Meets without reservations (RCT) Meets with reservations (RCT or QED) Study Sample Size n/a Large sample n/a Number of Study Sites n/a Multi-site sample n/a Similarity of Population n/a Overlaps with proposed populations or settings Overlaps with proposed populations and settings n/a Official definitions for each term are available in 34 CFR 77.1.(c)

3. The WWC and its review process 3

4. For each focus “topic,” the WWC searches for all studies, retrieves the actual reports or articles, and reads all of them The WWC assesses the quality of the evidence from each study The WWC summarizes findings succinctly The What Works Clearinghouse (WWC) reviews evidence to help educators make research-informed choices 4

5. WWC reviews every study against written standards Link to the Handbook. Handbook. 5

6. WWC ratings of effectiveness studies Meets WWC evidence standards without reservations Meets WWC evidence standards with reservations Does not meet WWC standards Each relevant outcome in the study receives one of three ratings; the study as a whole receives the rating of the highest-rated outcome. 6

7. WWC ratings and study designs Meets WWC evidence standards without reservations Meets WWC evidence standards with reservations Does not meet WWC standards Randomized controlled trials (RCTs) with little sample attrition Non-randomized studies with comparison groups similar at baseline Comparison groups are not similar at baseline 7

8. General characteristics of studies that meet WWC standards 8

9. “Studies” need to address effectiveness of an intervention, include technical detail Publicly-available original research studies Peer-reviewed or not Full written technical document (report, article), final version of the results Studies by developers of their own product or independent evaluators IN Newspaper articles Literature reviews, briefs, summaries Secondary research (re-analysis) Implementation studies Correlational studies (unless EOP) Descriptive studies of baselines and trend lines OUT 9

10. Characteristics that pertain to the way the study was designed and carried out 10

11. Intervention group College Freshmen SAT/ACT Score, Pell Fall 2015 Enhanced Supports Persistence through Spring 2016 Comparison group College Freshmen SAT/ACT Score, Pell Fall 2015 Enhanced Supports Persistence through Spring 2016 Study parameters needed to meet WWC standards Distinct groups Baseline equivalence No confounding factors Appropriate outcomes 11

12. Distinct Groups  At least two groups (one intervention and one comparison)  Not a group compared to itself (same students as freshmen and sophomore)  Not sequential/historical cohorts (freshmen in 2015 vs freshmen in 2014)  Clear definition of intervention and eligibility criteria – what services and who can get them?  Intervention group College Freshmen Comparison group College Freshmen Distinct groups 12 Created randomlyorNon-randomly  Identify eligible sample  Convenience sample  Lottery or flip of coin  Matching  Prospective  Retrospective

13. Baseline Equivalence  Necessary for QEDs to show because differences before intervention could carry through to outcomes, good practice (but not necessary) for RCTs to show  Must demonstrate groups were similar prior to intervention (at baseline)  WWC looks at the size of the difference  Effect size (ES) units – see Handbook for calculation SAT/ACT Score, Pell Fall 2015 SAT/ACT Score, Pell Fall 2015 Baseline equivalence 0.00 ≤ ES Difference ≤ 0.050.05 < ES Difference ≤ 0.25ES Difference > 0.25 Satisfies baseline equivalence Statistical adjustment required to satisfy baseline equivalence Does not satisfy baseline equivalence 13

14. No Confounding Factors  A component completely aligned with only one study condition  Impossible to separate effect of intervention and confounding factor  Cannot attribute impact solely to intervention  Study Does Not Meet WWC Standards  Common confounds  Single unit (school, classroom, teacher, mentor) in one or both conditions  Intervention is bundled with other services not being studied  Treatment and comparison occur at different points in time (e.g., historical cohort) Enhanced Supports No enhanced Supports No confounding factors 14

15. Appropriate Outcomes  Used to determine the impact of the intervention  Eligible outcomes (and data) have the following characteristics:  Face validity – measures what it is supposed to measure  Reliability – measures things consistently  Not over aligned with the intervention  Collected in a similar manner across groups Persistence through Spring 2016 Eligible outcomes 15

16. Beyond the WWC standards: other study features needed to meet Moderate or Strong Evidence of Effectiveness 16

17. Reminder: WWC standards only one factor 17 Evidence Factors STRONG THEORY EVIDENCE OF PROMISE MODERATE EVIDENCE OF EFFECTIVENESS STRONG EVIDENCE OF EFFECTIVENESS Number of Studies n/a (logic model only) At least one At least two Study Findings on a Relevant Outcome n/a Statistically significant, substantively important (0.25 standard deviation or larger) positive association Statistically significant positive impact with no unfavorable and overriding impacts What Works Clearinghouse Standards n/a Meets without reservations (RCT) Meets with reservations (RCT or QED) Meets without reservations (RCT) Meets with reservations (RCT or QED) Study Sample Size n/a Large sample Number of Study Sites n/a Multi-site sample Similarity of Population n/a Overlaps with proposed populations or settings Overlaps with proposed populations and settings Official definitions for each term are available in 34 CFR 77.1.(c)

18. Key factors: number of studies, findings 18 Evidence Factors STRONG THEORY EVIDENCE OF PROMISE MODERATE EVIDENCE OF EFFECTIVENESS STRONG EVIDENCE OF EFFECTIVENESS Number of Studies n/a (logic model only) At least one At least two Study Findings on a Relevant Outcome n/a Statistically significant, substantively important (0.25 standard deviation or larger) positive association Statistically significant positive impact with no unfavorable and overriding impacts What Works Clearinghouse Standards n/a Meets without reservations (RCT) Meets with reservations (RCT or QED) Meets without reservations (RCT) Meets with reservations (RCT or QED) Study Sample Size n/a n/a/a Large sample Number of Study Sites n/a Multi-site sample Similarity of Population n/a Overlaps with proposed populations or settings Overlaps with proposed populations and settings Official definitions for each term are available in 34 CFR 77.1.(c)

19. Key factors: study sample, sites, population 19 Evidence Factors STRONG THEORY EVIDENCE OF PROMISE MODERATE EVIDENCE OF EFFECTIVENESS STRONG EVIDENCE OF EFFECTIVENESS Number of Studies n/a (logic model only) At least one At least two Study Findings on a Relevant Outcome n/a Statistically significant, substantively important (0.25 standard deviation or larger) positive association Statistically significant positive impact with no unfavorable and overriding impacts What Works Clearinghouse Standards n/a Meets without reservations (RCT) Meets with reservations (RCT or QED) Meets without reservations (RCT) Meets with reservations (RCT or QED) Study Sample Size n/a Large sample Number of Study Sites n/a Multi-site sample Similarity of Population n/a Overlaps with proposed populations or settings Overlaps with proposed populations and settings Official definitions for each term are available in 34 CFR 77.1.(c)

20. Where to learn more WWC Web site (http://ies.ed.gov/ncee/wwc/webinar.aspx), go to “Tools and References“ under the Resources tab to find:http://ies.ed.gov/ncee/wwc/webinar.aspx ► Webinars about strong studies ► Review protocols ► Standards Briefs – more on specific WWC standards FAQ on designing strong QEDs (http://www.dir-online.com/wp- content/uploads/2015/11/Designing-and-Conducting-Strong- Quasi-Experiments-in-Education-Version-2.pdf http://www.dir-online.com/wp- content/uploads/2015/11/Designing-and-Conducting-Strong- Quasi-Experiments-in-Education-Version-2.pdf 20

21. THANK YOU! 21