Layers of Evaluation & Meaningfulness of Results 3CSN and ELI Collaborate: Data Collection Workshop September 6, 2013 Mehmet Dali Öztürk, Ph.D. mehmeto@cos.edu

Meaningfulness of Results Layers of Evaluation Meaningfulness of Results Impact Assessment Most Meaningful Outcome Analysis More Meaningful Least Meaningful Output Analysis

Layers of Evaluation Impact assessment focuses on determination of changes in outcomes that can be attributed to polices, programs, initiatives and projects for schools, individuals, families, or communities. For example, the change in school graduation rates due to a specific program in comparison to schools who did not receive the same program.   Outcomes focus on intended and expected results of polices programs, initiatives and projects. For example, the percent increase in schools meeting state standards. Outputs are the types and quantities of engagements and efforts. For example, the number of mentoring hours produced by program personnel.

Output Outcome Impact Program X Number of students in program Number of students completing program Percent gain in standard test scores in math Significant changes in percent gain in standard math test scores at the focus schools where program is implemented Percent gain in standard test scores in reading Significant changes in percent gain in standard test scores in reading at the focus schools where program is implemented Percent of students demonstrating a grade increase in math on report cards Significant changes in percent of students demonstrating a grade increase in math on report cards at the focus schools where program is implemented Percent of students demonstrating a grade increase in reading on report cards Significant changes in percent of students demonstrating a grade increase in reading on report cards at the focus schools where program is implemented

Providing a Context for the Evaluation Process Are there differences in outcome variables? Are the differences significant? Are the differences meaningful? What do these differences mean with regard to the overall community?

Program Effectiveness Scale/Rating System Program Effectiveness/Scale Rating System, which was developed to be a tool to help policy-makers better understand the impact of programs, strategies, and innovations. By placing evaluation results into a format that is easy to understand, the Program Effectiveness Scale can help policy-makers make objective judgments on the worth of a program or strategy.