So What? How Do We Know States Are Making an Impact, and How Do We Report It? Patricia Mueller and Jocelyn Cooledge, Evergreen Evaluation & Consulting, Inc. Pat do the intro…

Identifying Impact Early to show impact Poll #1 results 22/46 grantees are in their second year. 65% of grantees are in Years 2 or 3. http://signetwork.org/content_pages/5 Poll #1 results Jocelyn – do the intro. Pat: Poll #1 – what year of the grant are you in? Point out where they are with worksheet, etc.

Demonstrating Impact Discussion about impact and points to keep in mind when working with states. Knight’s work on research to practice (KU-CRL) Drew parallels with SPDG evaluation Please share your experiences! Jocelyn - Jim Knight’s research to practice was with the KU Center for Research on Learning. Pat – please share experiences.

Concepts for Demonstrating Impact 1. Explicit Can you describe an impact? How do you take an “I know there is something different” to measurable and explicit data? 2. Take Time Impacts of scale take time Reporting on small-scale impacts qualitatively Reporting qualitative on the 524B Jocelyn take this slide

Concepts for Demonstrating Impact 3. Generalized Impact Widespread impact vs. impact for a few Depends on the scale of the initiative Ex: 3 students out of 200 showed enormous gains on assessment—good, but an initiative impact? Low incidence initiatives Does it make a difference from a state’s perspective for public relations? How do you address impact that is not generalized? Jocelyn will start the slide. Pat’s example – use SLP Master’s Program in a rural state. 11 graduates total, not sure if they will stay in the state…no service requirement. Sustainability – IHE is picking up the Program. E.g., improvement in 2 – 3 schools/districts, but not statewide.

Concepts for Demonstrating Impact 4. Routine Do you see reliable changes in the practices? What if you’re not—is there an impact? May not see reliable changes in Indicator data with changes in practice. Do you focus on Indicator data for your outcomes? Jocelyn do the intro. Pat – is there an impact? If you see changes and they aren’t consistent, is there an impact? How do interpret the data? What should we be looking at? Talk about the story behind the data…explanation of progress. Maybe dependent on the Project Officer. Do projects focus on SPP/APR Indicators?

Concepts for Demonstrating Impact 5. Barrier-Free Implementation Reducing barriers increases likelihood of positive outcomes We encourage states to mitigate barriers Do you talk to states about barriers to implementation? If so, how do you help them identify the barriers? 6. Changing the Ladder From student to State Department How do you address the issue with states if the changes need to be made at the state level? Jocelyn - #5 Changing the ladder—implementation science triangle #6 Pat – creating impact…need to change all the way up the cascade: student – teacher- school – district – region- state How do you address issues if changes need to be made at the state level?

Demonstrating an Impact What if you’re not able to demonstrate any impacts? Initiative stalled or slow Availability of data Data measures are weak Data results are weak Jocelyn’s slide

Demonstrating an Impact Working with partners PTI, IHEs, and early childhood initiatives What kinds of strategies do you use for collecting and obtaining data from partners? Activities for ####### Time Activity Purpose Participants 4/2/2013 PD PD for schools   4/22/2013 Planning Planning with team 2 5/1/2013 Meeting 5 6/3/2013 Office Office Work Pat – this slide. PD logs, getting data from them. How to focus the data collection; strategies for working with partners.

Reporting on Impact How do you allocate your time for evaluating the impact when the performance measures emphasize implementation? Reporting outcome data on the 524B Reporting outcome data for states Poll #2 Jocelyn’s slide. Poll – how many are in NCE?

Reporting on Impact PM #1 worksheets No-cost extensions Who completes them for your states? No-cost extensions How do they affect impact, if at all? Qualitative measures Pat’s slide.

Thank you! Please contact Pat or Jocelyn at pat@evergreenevaluation.net jcooledge@evergreenevaluation.net Adios! See you in November.