1. An Introduction to Evaluating Federal Title Funding
Kendra Winner, Research and Evaluation Coordinator MA DESE

2. Lower anxiety about evaluation Begin with the end in mind
Today
Lower anxiety about evaluation
Begin with the end in mind
Plan evaluation and programming at the same time
Identify measures
Conduct simple analysis

3. Lowering Anxiety
Lowering Anxiety

4. Begin with the end in mind: Title I example

5. Purpose of Title I: ESSA Text
SEC STATEMENT OF PURPOSE: 
The purpose of this title is to provide all children significant opportunity to receive a fair, equitable, and high-quality education, and to close educational achievement gaps.

6. Purpose of Title I: ESSA Text
SEC STATEMENT OF PURPOSE: 
The purpose of this title is to provide all children significant opportunity to receive a fair, equitable, and high-quality education, and to close educational achievement gaps.

7. Moving Backward from the Impacts
Resources
Federal Title Funds
Activities
Outputs
Outcomes
Impact
Close Educational Achievement Gaps

8. Plan program implementation and evaluation strategy at the same time

9. Before you even decide how to spend your Title Funds …
Align the program planning and evaluation planning
Identify lead or leads on evaluation
Include key evaluation staff in program meetings and key program staff in evaluation meetings
Cover program implementation and program evaluation in the same meetings
Include evaluation as an agenda item for all program implementation meetings

10. During Implementation and Evaluation Planning
Documentation
Write down your hypothesis: What do you anticipate will happen? How do you believe your program activities will facilitate the achievement of what you hope will happen?
Create a program logic model: what is the relationship between the resources you have available, the activities you will engage in, the outputs produced, and the impacts you desire?
Identify what measures will provide you with the information you need: How will you know if the gap between low performing students and high performing students is being reduced?

11. Title I Literacy Coach Example
Hypothesis: Hiring full time literacy coaches in each low- performing school who will provide bi-weekly training to and conduct individual coaching sessions with all teachers will lead to teachers applying literacy strategies that accelerate reading achievement in their classrooms. Implementation of these literacy strategies will result in low performing students making more than one year’s progress in literacy.

12. Literacy Coach Example Logic Model
Resources
Federal Title Funds
Salaries for four full-time literacy coaches
Activities
Literacy coaches hired
Literacy coaches trained
Literacy coaches provide bi-weekly coaching to all teachers
Teachers assess student progress monthly
Outputs
Number of coaching sessions conducted
Number of group trainings conducted
Number of coaching sessions received by individual teacher
Number, frequency, interval between student assessments
Outcomes
Teachers apply strategies in their classrooms
Teachers and coaches use student assessment data to design instructional interventions
Impact
Low performing students make more than one year of academic progress
The gap between low performing students and high performing students decreases

13. Select Measures
Select Measures

14. Selecting Outcome Measures
What do you need to know?
What measures will capture what you need to know?
Which students need to be measured?
Who will administer the measures?
When will the measures be collected?
How will the measures be documented?

15. Identify Measures and Data Collection Strategies
Resources
Federal Title Funds
Salaries for four full-time literacy coaches
Activities
Literacy coaches hired
Literacy coaches trained
Literacy coaches provide bi-weekly coaching to all teachers
Teachers assess student progress monthly
Outputs
Number of coaching sessions conducted
Number of group trainings conducted
Number of coaching sessions conducted by teacher
Outcomes
Teachers apply strategies in their classrooms
Teachers and coaches use student data to design instructional interventions
Impact
Low performing students make accelerated academic progress
The gap between low performing students and high performing students decreases

16. What Makes a Measure Good?
Valid: Does it measure what it intends to measure? What’s it supposed to measure?
Reliable: Is the measure consistent and accurate? If the same person takes the same test, is the result the same?
Usable: Can you use the test with the available time, money and effort? Is use of the test practicable?
Relevant: Is what you are measuring relevant to the outcomes you expect?

17. Measures Will Vary Based on Intervention Goals
Title I
Beginning of year and year-end reading proficiency
Title II
Assignment of low-income and minority students to effective teachers compared to their wealthier white peers
Title III
EL student achievement, progress in English Language Proficiency
Title IV:
Teacher and student perceptions of school climate

18. Conduct Analysis
Conduct Analysis

19. NOT an Expectation Causality?
What about …
Causality?
Separating out impacts of one intervention?
NOT an Expectation

20. BUT – You have to go beyond pre and post measurement
Not enough to document that students made progress over the school year
Must look at how students who receive the “treatment” progressed over the school year compared to something

21. Which students need to be measured? Compared to what?
Comparison to other students:
Similar students not receiving the intervention
Similar students identified as high performing (whole school programs) 
Comparison to internal standards:
Progress against a district performance standard
Comparison to external benchmarks:
Normed grade-level assessments

22. Analysis Strategy – Targeted Programs
Compare means
How does the end of year average “measure” of students receiving the intervention compare to the average “measure” of students who did not receive the intervention?
How does the average change over the year in the “measure” of students compare to the average change in the measure of students who did not receive the intervention?
Practical implications? Plan to assess all of the students you need to at the same time with the same assessment

23. Analysis Strategy – Whole School Programs
Compare means
How does the end of year average “measure” of low achieving students compare to that of high achieving students?
How does the end of year average “measure” of low achieving students compare to valid year end expectations?
How does the average change in “measure” of low achieving students compare to that of high achieving students?
Practical Implications? Method to identify low and high achieving students. Assess all students at the same times with the same assessment.

24. Title I example

25. A picture is worth a thousand words

26. Sample graph

27. NAEP 4th grade reading scores
Show trends over time. Trends over one year. Trends over multiple years. Implications – keeping the same measures over time.

28. Hiring an Evaluation Technical Assistance Provider
Future Opportunity
Hiring an Evaluation Technical Assistance Provider
Soliciting up to 10 district and/or school teams
Piloting support and tools for district and school evaluations
Including small districts and schools
Developing a set of evaluation tools and guidance
Please contact me if you are interested; keep a look out in the Commissioner’s Weekly

29. Thank you! Thank you page
Kendra Winner, Research and Evaluation Coordinator
Office of Planning and Research, DESE