SSIP Evaluation Workshops: Introductory Webinar DaSy, ECTA and OSEP December 13, 2017

Evaluation Workshop Series, by DaSy and ECTA, in collaboration with We are excited about this collaborative TA opportunity, with these four centers.

Welcome and Introduction Kathy Hebbeler DaSy and ECTA

SSIP Evaluation Online Workshop Series Evaluation of: Infrastructure Practices This collaborative TA activity involves two sets, or series, of online workshops. Through these working sessions, states will have the opportunity to improve or refine their SSIP evaluation plans to ensure meaningful and feasible evaluation of: infrastructure improvements; and practice change and practice fidelity.

Just a few reasons these workshop series may be for you . . . You have responsibilities related to the SSIP and want to increase your evaluation knowledge and skills; Your state’s SSIP might be overly ambitious and you’re rethinking some of your evaluation questions;  Your state has developed some great evaluation questions but is still struggling with how to get them answered; You have collected a lot of data and are wondering what to do with it . . . Who should attend these workshop series Just a few reasons these workshop series may be for you . . .

Agenda Introduction OSEP Perspective on SSIP Evaluation of: Practices Infrastructure Overview of SSIP Evaluation Workshop Series Registration Information Questions We are fortunate to have Leslie Fox from OSEP with us today. Leslie leads the Performance Accountability Team in OSEP, the group responsible, in part, for reviewing the State Systemic Improvement Plans. Leslie will help to reinforce the importance and value of evaluation in both of these areas: infrastructure and practices; and highlight some important aspects of each. And through the remainder of the webinar we will provide information that we hope will help you determine whether you’d like to participate in either or both of these workshop series, including information on how to register.

Webinar Objectives Participants will: Increase their understanding of important aspects of evaluating infrastructure and practice change/fidelity. Gain sufficient information to determine if they would like to participate in one or both of these workshop series. From that, here are our objectives for today. Leslie will remind us of some important considerations in the evaluation of both of these areas, tying them to expectations for the Phase III Year 2 SSIPs, and she will talk a bit about the relationship between evaluation of infrastructure and practices. And then, as I mentioned, we’ll provide some information on what to expect should you decide to participate in either or both of these series.

Poll Questions Who’s on the webinar?

OSEP Perspective and Considerations Leslie C. Fox, PhD Office of Special Education Programs, US Department of Education

SSIP Phase III, Year 2 http://freshspectrum.com/what-is-evaluation-anyway/

Nice to Know or Need to Know? Defining milestones Alignment with theory of action Data sources Timelines Comparison data/baseline, trend, criteria or benchmark

SSIP: State Systemic Improvement Plan Clearly connected to the system analysis conducted in Phase I Activities and strategies if implemented will lead to system changes that can be described and measured (e.g. “what is different about your system in 2020 compared to 2014?)

Progress toward the SiMR FFY 2013 FFY 2014 FFY 2015 FFY 2016 FFY 2017 FFY 2018 71.00 % 71.00% 71.20% 71.40% 71.60% 72.00% Phase I 2015 Phase II 2016 Phase III – 1 2017 Phase III-2 2018 Phase III-3 2019 Phase III-4 2020 SiMR: Substantially increase the rate of growth in infants and toddlers demonstrating positive social-emotional skills by 1 percent by the final year (FFY 2018)

If only this work really was linear… Provider Knowledge Provider Practices Parent/Caregiver Knowledge, Behavior, Practices Family and/or Child Outcomes

Provider Knowledge Training and professional development activities Pre-post knowledge assessments Evaluating more than satisfaction or perceptions of knowledge gain Making sure the attendees are the ones who most need the training Follow-up and support activities

Provider Practices What is the practice? How will it be demonstrated? (e.g. IFSP, checklists) What is the necessary frequency? How will fidelity decisions be made?

Parent/Caregiver Practices A prickly issue but one that has to be eventually acknowledged Whatever your program expects the provider to do, there is a strong likelihood the parent is expected to carry-over the activity/behavior into day to day activities and routines How will you know?

Child/Family Outcomes May be measured through the process used for APR data reporting May be measured by additional tools or instruments – particularly if the EBP is intended to support specific skills or constructs (e.g. DECA for attachment)

But what about infrastructure evaluation?? Progress implementing the SSIP helps to answer the question: What is descriptively and operationally different about your system at the end of the SSIP cycle? Not simply describing the activities that were implemented but relates them to the initial analysis It is reporting on benchmarks or other indicators of system change

An example:

So what? Measuring and evaluating system changes helps to communicate why the system-level work matters Can often be important for sustainability of EBPs, scale-up and/or generalization to other outcome areas Contingent on the initial system analysis and an on-going Plan-Do-Study-Act approach

SSIP Phase III, Year 2 In summary, as OSEP reviews the SSIP submission we look for incremental changes in provider practices and how system improvements are supporting high quality intervention services We would expect to see “progress” each year and be able to relate the reported activities and data to the State’s theory of action

Overview of SSIP Evaluation Workshops Robin Nelson, DaSy

Intended Outcomes of Workshops Increase understanding of evaluation of: 1. Infrastructure improvement and impacts 2. Practice change and practice fidelity Identify appropriate measures/tools Develop/refine outcomes and related performance indicators Develop/refine strategies for data analysis and use of data Identify strategies for improving evaluation plan We have a common set of intended outcomes for each of the two workshop series. The first outcome, the box on the left, is geared toward gaining a better understanding of what is involved in evaluations of infrastructure and practices. Increase understanding of the importance of evaluating these two, and some key concepts and considerations, including: the difference between infrastructure change and the impact of those changes; and between practice change and fidelity of practices. We focus on each of the other outcomes individually, and also look at the alignment of some of those areas/constructs with one another. For example, can you use the same tool for measuring practice change and practice fidelity, and if so, how would your performance indicators, data collection and/or analysis be different for each? By the end of the 3rd workshop, you will have clear strategies for improving your evaluation plan.

Structure/Format of Workshop Series Two workshop series: infrastructure and practices Each series is a set of 3 90-minute working sessions Expectation that state will participate in all 3 sessions States may participate in one or both series More than one person per state may participate Expect some, minimal prep work between sessions (1 to 2 hours) Individualized TA will be available between sessions Because the online sessions build on one another, we expect participants to attend each of the three sessions in the series. Ideally, for the sake of continuity, we encourage at least one state representative to participate in all three sessions. We welcome multiple people from a state’s SSIP team to participate and work together during the sessions. We will ask participants to do about 1 to 2 hours of work in preparation for the next working session. And we will make individualized TA available in between sessions to support this and related work.

Content of Working Sessions Session 1: What are we measuring and why? Shoring up your foundation: evaluation questions and outcomes Session 2: How are we measuring? Delving deeper into measurement strategies and data sources Session 3: What do we do with all these data? Pulling everything together and preparing for data analysis and use Workshops will build on one another. There will be some flexibility with the content in order to meet your needs as a state. 1. Looking at measurement in the context of your evaluation questions? What are infrastructure and practice outcomes? Concrete, practical approaches. Infrastructure outputs vs. change vs. impacts of that change; practice change vs. practice fidelity. Do you have good performance indicators, and importantly, do they align with your evaluation questions? 2. Options for measurement strategies. Criteria for a good fidelity tool, and the challenges; what’s feasible? Considerations for data sources and data collection – using self-report vs. observation by a peer or coach, or a hybrid model. Alignment of measurement strategies, data collection and outcomes/PIs. You developed your PI before having a good sense of the measurement strategy and data. You have a PI on practice fidelity but you’re actually measuring practice change. 3. Thinking through the data you’re collecting, whether it aligns with your outcomes and performance indicators; How you’re going to analyze, share and use that data; Approaches for data reduction – how do you take a bunch of survey items and reduce them to one performance indicator.

Common Structure for Each Working Session Introduction of topic Large group discussion/activity Reflection question(s) Small groups/hands-on activities Wrap up, next steps, prep work What worked/could be improved We’ll have a mix of different kinds of activities in a given session. Small group breakouts will include hands-on activities related to your SSIP evaluation. We’ll make decisions on the breakouts once we know who’s registered – it could be based on area of infrastructure, type of evidence-based practice, or how far along states are in the process of evaluating these things. We’ll wrap up at the end of each session, talk about the next session and some related prep work for that session. Formative evaluation at the end of each session.

Dates and Times of Working Sessions Infrastructure Practices Wednesday, Jan. 24, 2018 3:30 – 5:00 pm ET Wednesday, Feb. 7, 2018 3:30 – 5:00 pm ET Wednesday, Feb. 21, 2018 3:30 – 5:00 pm ET Thursday, Feb. 1, 2018 3:30 – 5:00 pm ET Thursday, Feb. 15, 2018 3:30 – 5:00 pm ET Thursday, March 1, 2018 3:30 – 5:00 pm ET

Registration Information Betsy Davies-Mercier

How to Register After this webinar, watch for an email with a link to register Sign up for either/both series using the same link Use one registration form to register multiple state participants for the workshops Sign up anytime from now through January 5

Information Needed to Register Participants’ role and contact information For Infrastructure series Areas(s) of infrastructure being evaluated Tools using to evaluate infrastructure, if identified What you hope to gain from the series For Practice series Evidence-based practice, if identified Tools using to measure practice, if identified In order to tailor the workshops to the needs of participants, TA providers will review the SSIPs of all participating states prior to the workshops beginning. During registration, we’ll ask a few basic questions to assist us with our preparation. We want to emphasize that it’s a simple form with checklists, not very burdensome, so you won’t have to provide a lot of narrative information.

Questions

Thank you! I’d like to thank you for attending today to learn more about SSIP evaluation and this exciting TA opportunity. We hope you’ll hop on board and come along for the ride! Reminder: OSEP TA call tomorrow will answer some frequently asked questions about the upcoming SSIP submission and the Performance Accountability Team’s review process, and provide tips for submitting a concise but thorough SSIP report.

Thank you The contents of this tool and guidance were developed under grants from the U.S. Department of Education, #H326P120002 and #H373Z120002. However, those contents do not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government. Project Officers: Meredith Miceli, Richelle Davis, and Julia Martin Eile.