1. So What?
How Do We Know States Are Making an Impact, and How Do We Report It?
Patricia Mueller and Jocelyn Cooledge,
Evergreen Evaluation & Consulting, Inc.
Pat do the intro…

2. Identifying Impact Early to show impact Poll #1 results
22/46 grantees are in their second year.
65% of grantees are in Years 2 or 3.
Poll #1 results
Jocelyn – do the intro.
Pat: Poll #1 – what year of the grant are you in? Point out where they are with worksheet, etc.

3. Demonstrating Impact
Discussion about impact and points to keep in mind when working with states.
Knight’s work on research to practice (KU-CRL)
Drew parallels with SPDG evaluation
Please share your experiences!
Jocelyn - Jim Knight’s research to practice was with the KU Center for Research on Learning.
Pat – please share experiences.

4. Concepts for Demonstrating Impact
1. Explicit
Can you describe an impact?
How do you take an “I know there is something different” to measurable and explicit data?
2. Take Time
Impacts of scale take time
Reporting on small-scale impacts qualitatively
Reporting qualitative on the 524B
Jocelyn take this slide

5. Concepts for Demonstrating Impact
3. Generalized Impact
Widespread impact vs. impact for a few
Depends on the scale of the initiative
Ex: 3 students out of 200 showed enormous gains on assessment—good, but an initiative impact?
Low incidence initiatives
Does it make a difference from a state’s perspective for public relations?
How do you address impact that is not generalized?
Jocelyn will start the slide.
Pat’s example – use SLP Master’s Program in a rural state. 11 graduates total, not sure if they will stay in the state…no service requirement. Sustainability – IHE is picking up the Program.
E.g., improvement in 2 – 3 schools/districts, but not statewide.

6. Concepts for Demonstrating Impact
4. Routine
Do you see reliable changes in the practices?
What if you’re not—is there an impact?
May not see reliable changes in Indicator data with changes in practice. Do you focus on Indicator data for your outcomes?
Jocelyn do the intro.
Pat – is there an impact? If you see changes and they aren’t consistent, is there an impact? How do interpret the data? What should we be looking at? Talk about the story behind the data…explanation of progress. Maybe dependent on the Project Officer.
Do projects focus on SPP/APR Indicators?

7. Concepts for Demonstrating Impact
5. Barrier-Free Implementation
Reducing barriers increases likelihood of positive outcomes
We encourage states to mitigate barriers
Do you talk to states about barriers to implementation?
If so, how do you help them identify the barriers?
6. Changing the Ladder
From student to State Department
How do you address the issue with states if the changes need to be made at the state level?
Jocelyn - #5 Changing the ladder—implementation science triangle
#6 Pat – creating impact…need to change all the way up the cascade: student – teacher- school – district – region- state
How do you address issues if changes need to be made at the state level?

8. Demonstrating an Impact
What if you’re not able to demonstrate any impacts?
Initiative stalled or slow
Availability of data
Data measures are weak
Data results are weak
Jocelyn’s slide

9. Demonstrating an Impact
Working with partners
PTI, IHEs, and early childhood initiatives
What kinds of strategies do you use for collecting and obtaining data from partners?
Activities for #######
Time
Activity
Purpose
Participants
4/2/2013 PD
PD for schools
4/22/2013
Planning
Planning with team 2
5/1/2013
Meeting 5
6/3/2013
Office
Office Work
Pat – this slide. PD logs, getting data from them. How to focus the data collection; strategies for working with partners.

10. Reporting on Impact
How do you allocate your time for evaluating the impact when the performance measures emphasize implementation?
Reporting outcome data on the 524B
Reporting outcome data for states
Poll #2
Jocelyn’s slide. Poll – how many are in NCE?

11. Reporting on Impact PM #1 worksheets No-cost extensions
Who completes them for your states?
No-cost extensions
How do they affect impact, if at all?
Qualitative measures
Pat’s slide.

12. Thank you! Please contact Pat or Jocelyn at
Adios! See you in November.