Making a Difference Update on the National i3 Evaluation of the Implementation and Impact of Diplomas Now

2 Diplomas Now i3 Validation Study Goal: Validate effectiveness of the DN model Research Questions - Phase I (current): 1.What is the impact of Diplomas Now on students’ outcomes, particularly with regard to attendance, in- school behavior, and course performance? 2.What lessons can be learned about implementation of the model during national expansion?

3 Notable about this Evaluation It’s rigor: Random assignment design: uses a lottery-like process to assign schools to two equivalent groups (“DN schools” and “Non-DN schools”) Gold standard for evaluation research: Diplomas Now caused any differences in school and student outcomes Fairest way to allocate scarce resources It’s size: 11 school districts, 62 secondary schools 32 DN, 30 non-DN

4 Notable about this Study It’s generalizability: The study districts are broadly representative of urban districts across the country The schools represent some of the most challenged in the nation The students represent those traditionally dealing with the most educational and life challenges It’s duration: US DOE Office of Innovation and Improvement awarded extension grant for the evaluation: follow 2 student cohorts for 4 years Importance of continuity of implementation of the program model

5 Diplomas Now i3 Validation Study Second report (May 2015) on the first and second years of implementation: “fidelity” and “service contrast” The third report (early 2016) will include the impact of Diplomas Now on students’ ABCs: –Attendance –Behavior –Course Performance

6 Fidelity of Implementation How closely the model as implemented in the Diplomas Now schools matched the original design. Fidelity measures created with support from DNIST to identify the components of the model and to set thresholds for successful implementation of each component. Data from surveys of Diplomas Now field staff and partner organizations’ data systems.

7 Fidelity of Implementation In Year 1 and Year 2, the majority of the model was implemented as designed. –Fidelity measurement based on 111 components clustered in 9 “inputs” and reported on a 0-1 scale –Year 1 = 0.61 and Year 2 = 0.62 While there is still room for growth, the level of fidelity necessary to achieve impacts on student outcomes is still unknown.

8 Fidelity of Implementation: Year 2 All DN Schools Top Quartile Bottom Quartile Pillar I. Teacher Teams/Small Learning Communities Strong Learning Environments Pillar II. Curriculum and Instruction with PD Professional Development and Peer Coaching Curriculum for College Readiness Pillar III. Tiered Student Supports Tiered Intervention Model Student Supports Student Case Management Pillar IV. Can-Do Culture and Climate Integrated On-Site Support Family and Community Involvement Program Staff Training and Development Overall

9 Service Contrast The extent to which the Diplomas Now model as implemented created differences between practices in Diplomas Now and Non-Diplomas Now schools. Created survey items that focused on the kinds of activities implemented in DN schools, but used language that any teacher or administrator could understand. Teachers and administrators were surveyed in both DN and Non-DN schools.

10 Service Contrast Implementation of Diplomas Now made schools different in organization and staff practices than what would have been expected. In Year 2, across 29 measures: –26 indicated positive change made by Diplomas Now –16 of these differences were statistically significant From Year 1 to Year 2, differences increased, often driven by stability in DN schools and declines in Non- DN schools.

11 Outcomes and Impacts by Implementation Quality (Career Academies example) Program A: Strong Graduation Rates (%) Program C: Weak Graduation Rate (%) Program B: Moderate Graduation Rates (%) Career Academy GroupControl Group

12 Diplomas Now i3 Validation Study Research Questions – Phase II (extension): 1.What is the impact of Diplomas Now on four-year student outcomes: the 9 th grade transition of middle grades students and the graduation of high school students 2.What additional lessons can be learned about implementation of the model during national expansion, particularly about relationships between implementation and outcomes?

13 For More Information… See the second evaluation report: See the evaluation project page: Contacts: William Corrin, MDRC, Susan Sepanik, MDRC,