1. Phase III: Evaluating Your First Year of Implementation
NCSI Cross-State Learning Collaboratives Part B Meeting
November 29 and December 1, 2016
Dallas, Texas

2. Session Overview Introduction to Phase III Evaluation
Evaluating State Systemic Improvement Plan (SSIP) Implementation
Evaluating Progress Toward the State- identified Measurable Result (SiMR)
Collecting and Using Data

3. Evaluation: Moving From Phase II to Phase III
You planned your evaluation in Phase II; is it giving you the information you need for Phase III?
What will your evaluation allow you to say about what you do and achieve in your first year of SSIP implementation?
Demonstrate progress
Identify needed changes (continuous improvement)

4. What Will You Need to Report in Phase III?
What are your key measures?
What activities and outcomes are most important for tracking progress toward the SiMR?
For each key measure, describe:
Alignment to your theory of action
Data sources
Data collection procedures and timelines
Baseline data
When appropriate, sampling procedures and planned comparisons (analytic methods)
Theory of action – intended activity and the activity’s intended outcome?

5. Prioritizing Early Evaluation Activities
Implementation
Early Outcomes
Think back to your theory of action or logic model. Are you on track to implement your intended activities and strategies? What short-term outcomes do you expect to achieve as a result?
This is intended to be a thought exercise to get participants thinking forward about what they want to be able to share about implementation.

6. Developing Data Analysis Plan Details
Acknowledge this work already under way.
Sample SSIP Action Plan Template:

7. Engaging Stakeholders in All Aspects of Evaluation
Prioritizing evaluation questions
Selecting data sources and measures
Preparing for data collection (e.g., getting buy-in, providing professional development [PD] on measures)
Collecting data
Analyzing data
Communicating findings
Using findings to inform decision making

8. Evaluating SSIP Implementation
We need data that answer the following questions:
Are we doing what we said we would do?
On time?
How well? (meeting a standard?)
Are activities achieving intended outputs or outcomes?
If not, why?
Prioritizing activities to evaluate the following:
How is the activity related to the theory of action?
Does it lead to short-term outcomes or benchmarks that will inform decision making?
Does it support long-term outcomes needed for sustainability, scaling up, and achieving the SiMR?

9. Possible Tool: Implementation Evaluation Matrix
Activity to Evaluate
Data Collection Plan
Evaluation of Activity Implementation
SSIP Activity
Level of System
Sources/Methodology
Schedule
Scoring Criteria
Data/Score
Notes
Implemented activity from logic model or action plan
Agency or system level at which activity is implemented
Describe data source/measurement tool, collection and analysis methods, and parties responsible.
Specify data collection and analysis schedule.
Specify criteria for scoring/rating implementation.
Provide data used to determine score. Mark score.
Additional information on implementation or rationale for score
☐ State
☐ Regional (e.g., professional development providers)
☐ District
☐ School
☐ Provider
☐ Other (describe below):
Data source/ measurement tool:
Data collection and analysis methods:
Parties responsible:
Collection schedule:
Analysis schedule:
0 =
1 =
2 =
3 =
Data:
Date:
Score:
☐ 0
☐ 1
☐ 2
☐ 3
The Implementation Evaluation Matrix helps facilitate thinking pertaining to the following questions: What data will you collect to track, document, and report progress in accomplishing these activities? Data could include surveys, focus groups, counts of participation, or observations. What else? Once you decide WHAT data you are going to collect, you need a plan for who is going to collect the data, where the data are going to be stored, aggregating the data, cleaning the data, and finally analyzing/interpreting the results of the data (How often are you planning to review data results, and how will these results be used to inform your activities moving forward?).

10. Practice: Implementation Analysis
Activity to Evaluate
Data Collection Plan
Evaluation of Activity Implementation
SSIP Activity
Level of System
Sources/Methodology
Schedule
Scoring Criteria
Data/Score
Notes
Pilot schools will implement universal literacy screening in Grades K–3.
☐ State
☐ Regional (e.g., professional development providers)
☐ District
☐ School
☐ Provider
☐ Other (describe below):
Data source/measurement tool:
Data collection and analysis methods:
Parties responsible:
Collection schedule:
Analysis schedule:
Let’s work through an example together. How could we measure and analyze implementation of this activity? “Pilot schools will implement universal literacy screening in Grades K–3.”
[click] Here’s a possible performance measure aligned to this activity. When we get to scoring criteria, we’ll set a target for full implementation.
First, who, or what level of the system, is implementing the activity?

11. Practice: Implementation Analysis
Activity to Evaluate
Data Collection Plan
Evaluation of Activity Implementation
SSIP Activity
Level of System
Sources/Methodology
Schedule
Scoring Criteria
Data/Score
Notes
Pilot schools will implement universal literacy screening in Grades K–3.
☐ State
☐ Regional (e.g., professional development providers)
☐ District
☒ School
☐ Provider
☐ Other (describe below):
Data source/measurement tool:
Data collection and analysis methods:
Parties responsible:
Collection schedule:
Analysis schedule:
This activity will be implemented by schools. Next, what data source can tell us about implementation?

12. Practice: Implementation Analysis
Activity to Evaluate
Data Collection Plan
Evaluation of Activity Implementation
SSIP Activity
Level of System
Sources/Methodology
Schedule
Scoring Criteria
Data/Score
Notes
Pilot schools will implement universal literacy screening in Grades K–3.
☐ State
☐ Regional (e.g., professional development providers)
☐ District
☒ School
☐ Provider
☐ Other (describe below):
Data source/measurement tool:
Screening score summaries by school and grade
Data collection and analysis methods:
Parties responsible:
Collection schedule:
Analysis schedule:
Now that we know what data we’ll collect, how will we collect them and how will we analyze them?

13. Practice: Implementation Analysis
Activity to Evaluate
Data Collection Plan
Evaluation of Activity Implementation
SSIP Activity
Level of System
Sources/Methodology
Schedule
Scoring Criteria
Data/Score
Notes
Pilot schools will implement universal literacy screening in Grades K–3.
☐ State
☐ Regional (e.g., professional development providers)
☐ District
☒ School
☐ Provider
☐ Other (describe below):
Data source/measurement tool:
Screening score summaries by school and grade
Data collection and analysis methods:
Summaries submitted via state data system portal (collection); state verifies all relevant schools and grades submitted (analysis)
Parties responsible:
School data managers (collection); Lily Snyder (analysis)
Collection schedule:
Analysis schedule:
Next, how often will we collect and analyze data?

14. Practice: Implementation Analysis
Activity to Evaluate
Data Collection Plan
Evaluation of Activity Implementation
SSIP Activity
Level of System
Sources/Methodology
Schedule
Scoring Criteria
Data/Score
Notes
Pilot schools will implement universal literacy screening in Grades K–3.
☐ State
☐ Regional (e.g., professional development providers)
☐ District
☒ School
☐ Provider
☐ Other (describe below):
Data source/ measurement tool:
Screening score summaries by school and grade
Data collection and analysis methods:
Summaries submitted via state data system portal (collection); state verifies all relevant schools and grades submitted (analysis)
Parties responsible:
School data managers (collection); Lily Snyder (analysis)
Collection schedule:
Three times per year, within 1 month of fall, winter, and spring screening dates
Analysis schedule:
Initial review 1 month after last school’s screening date; follow-up as needed
Next, how will we score or rate the level of implementation?

15. Practice: Implementation Analysis
Activity to Evaluate
Data Collection Plan
Evaluation of Activity Implementation
SSIP Activity
Level of System
Sources/Methodology
Schedule
Scoring Criteria
Data/Score
Notes
Pilot schools will implement universal literacy screening in Grades K–3.
☐ State
☐ Regional (e.g., professional development providers)
☐ District
☒ School
☐ Provider
☐ Other (describe below):
Data source/measurement tool:
Screening score summaries by school and grade
Data collection and analysis methods:
Summaries submitted via state data system portal (collection); state verifies all relevant schools and grades submitted (analysis)
Parties responsible:
School data managers (collection); Lily Snyder (analysis)
Collection schedule:
Three times per year, within 1 month of fall, winter, and spring screening dates
Analysis schedule:
Initial review 1 month after last school’s screening date; follow-up as needed
0 = 0–29% of pilot schools
1 = 30–69% of schools
2 = 70–90% of schools
3 = 91–100% of schools
Now let’s add some possible data and score them using these criteria.

16. Practice: Implementation Analysis
Activity to Evaluate
Data Collection Plan
Evaluation of Activity Implementation
SSIP Activity
Level of System
Sources/Methodology
Schedule
Scoring Criteria
Data/Score
Notes
Pilot schools will implement universal literacy screening in Grades K–3.
☐ State
☐ Regional (e.g., professional development providers)
☐ District
☒ School
☐ Provider
☐ Other (describe below):
Data source/measurement tool:
Screening score summaries by school and grade
Data collection and analysis methods:
Summaries submitted via state data system portal (collection); state verifies all relevant schools and grades submitted (analysis)
Parties responsible:
School data managers (collection); Lily Snyder (analysis)
Collection schedule:
Three times per year, within 1 month of fall, winter, and spring screening dates
Analysis schedule:
Initial review 1 month after last school’s screening date; follow-up as needed
0 = 0–29% of pilot schools
1 = 30–69% of schools
2 = 70–90% of schools
3 = 91–100% of schools
Data: 75% of pilot schools
Date: 10/28/2016
Score:
☐ 0
☐ 1
☒ 2
☐ 3
Here’s an example of a 4-point (0–3 scale).
[click] These criteria require transforming our school summary data into percentage of schools.
In this case, implementation is not complete. Do we know which schools are not implementing screening and why?

17. Practice: Implementation Analysis
Activity to Evaluate
Data Collection Plan
Evaluation of Activity Implementation
SSIP Activity
Level of System
Sources/Methodology
Schedule
Scoring Criteria
Data/Score
Notes
Pilot schools will implement universal literacy screening in Grades K–3.
☐ State
☐ Regional (e.g., professional development providers)
☐ District
☒ School
☐ Provider
☐ Other (describe below):
Data source/measurement tool:
Screening score summaries by school and grade
Data collection and analysis methods:
Summaries submitted via state data system portal (collection); state verifies all relevant schools and grades submitted (analysis)
Parties responsible:
School data managers (collection); Lily Snyder (analysis)
Collection schedule:
Three times per year, within 1 month of fall, winter, and spring screening dates
Analysis schedule:
Initial review 1 month after last school’s screening date; follow-up as needed
0 = 0–29% of pilot schools
1 = 30–69% of schools
2 = 70–90% of schools
3 = 91–100% of schools
Data: 75% of pilot schools
Date: 10/28/2016
Score:
☐ 0
☐ 1
☒ 2
☐ 3
All schools from one district reported that their staff were not yet adequately trained in the new screening tool and that no fall screening occurred. Other late schools reported that the screening was completed but they were facing delays in data entry.
Dig deeper into why implementation is not to standard.

18. Using Implementation Findings
One district was not adequately trained.
Provide training before winter screening.
But why not adequately trained for fall?
Same PD provider as other districts?
Low staff buy-in for screening?
Low administration support (e.g., staff time, scheduling)?
Some schools were late with entry.
Update matrix when data are entered to more accurately assess extent of screening.
Explore ways to improve data entry efficiency (issues with training, staff time, data system?).

19. Evaluating Infrastructure
What infrastructure changes are you making in the first year of implementation?
What data will demonstrate that these changes were made? And on time?
Are your infrastructure changes achieving intended outcomes? How do they support…
Implementation of your improvement strategies?
Progress toward the SiMR?
SSIP sustainability and scale-up?
Office of Special Education Programs guidance: What aspect of the state system is different or how has it changed as a result of the SSIP’s coherent improvement strategies?

20. Infrastructure Example: Data System
Phase II
Phase III
Planned data system improvements that allow you to better link different data sets (e.g., demographic information, attendance data, assessment data)
Tracked implementation milestones for making improvements using the Implementation Evaluation Matrix
Documented intended outcomes by measuring accuracy and time for data transfers to merged database
See the Topical Breakout Session Evaluating Infrastructure Changes and the Impact of Infrastructure Changes.

21. Evaluating PD
PD to support evidence-based practice (EBP) implementation
Increased practitioner EBP knowledge and skill
Improved practitioner EBP implementation
Improved student outcomes
Similar questions: Did you do it? Is it achieving results?
Evaluate at any of these stages, starting earlier.
PD cannot improve student outcomes without changes in adult behavior.

22. PD Example: Math Intervention
Evaluation Question(s)
Possible Data Sources
Was PD delivered as intended?
Reach
Dosage
Timelines
Did PD vary by site?
Training calendar
Attendance sheets
Participant surveys
Provider course descriptions or materials
Did PD result in increased participant…
Knowledge?
Implementation fidelity?
Pretest/posttest
Intervention logs
Intervention fidelity data
Did student outcomes improve?
State mathematics test
Other mathematics assessments (e.g., screening or progress monitoring)
You may want to look for outcome differences according to how the training was provided.
Digging deeper: If PD is not implemented well, consider infrastructure.
Emphasize importance of implementation fidelity.
See the Topical Breakout Session Evaluating Professional Development Implementation and Impact.

23. Evaluating EBP Implementation at the District and School Levels
Possible Evaluation Question
Example Data Sources
Possible Analyses
To what extent are SSIP schools implementing the EBP?
District/school survey regarding material availability and usage
Number and percentage of SSIP schools implementing the EBP
To what extent are eligible students participating in the EBP?
Records of eligibility (e.g., assessment scores)
Records of participation (e.g., intervention logs)
Number and percentage of eligible students who are participating
See the Rotating Session Evidence-Based Practices: Moving Beyond Identification Toward Implementation With Fidelity.

24. Evaluating EBP Implementation at the Practitioner and Student Levels
Possible Evaluation Question
Example Data Sources
Possible Analyses
To what extent are participating students receiving the intended dosage?
Records of participation (e.g., intervention logs)
Average minutes/hours per participating student
To what extent are practitioners implementing the EBP with fidelity?
Teacher self-report (e.g., using checklist created by program developer)
Observation data (e.g., using rubric aligned to the EBP)
Average percentage of steps/components reported or observed

25. What If Implementation Does Not Meet Your Criteria?
Concern
Possible Reasons
Some schools not implementing EBP PD
Buy-in/ leadership support
Materials
Staffing
Not provided to all eligible students
Assessment/screening (PD?)
Scheduling
Low fidelity
Buy-in
Dosage
Attendance
Consider context

26. Progress Toward SiMR
Unlikely to see student-level changes in SiMR in the first year
SiMR data may not be available in time
Likely need longer implementation before the needle moves
Can measure other outcomes “that are necessary steps toward achieving the SiMR”
More frequent, sensitive measures of student outcomes (still may be too early)
Nonstudent measures, such as changes in school or educator practices
Quote from OSEP Phase II Report Outline

27. Examples of Other Measures
Student
Other Outcomes
Screening or progress monitoring data on measures related to academic SiMR
For graduation SiMR, student scores on risk screeners or specific related variables (e.g., attendance, behavior, grades)
Intervention data (e.g., participation, progress toward goals, exit rates)
District- or school-level outcomes
System changes (e.g., new assessments, data systems, problem-solving processes and meetings, PD infrastructure)
Implementing EBP with students
Practitioner-level outcomes
EBP knowledge following PD
EBP fidelity data

28. Data Analysis Considerations
How will you report outcomes?
Data may need to be converted to another type before analysis.
In order to disaggregate results, you may need to link outcome data to “filter” variables, such as implementation site or student characteristics.
What comparisons will you make?
Have you established a baseline?
Comparing treatment and nontreatment conditions
Cross-sectional versus longitudinal analyses
Are your sites ready to report all the information you need?
Does your data system easily link different data sets?

29. Sampling Considerations
Establish samples early!
Is your sample large enough for analyses?
Is your sample representative of the broader population?
Are pilot sample students representative of all students in the state?
If you are collecting some measures with only a subset of participants (e.g., fidelity observations), are they representative of the broader sample?
Sample size – reporting, attrition concerns
Frame – you’ve probably already selected your sample...

30. Consider a Tracking Sample
Longitudinally examine performance for a group of students.
What about students who enter or exit the program or target group?
Consider tracking progress for a broader sample of students who are at risk.
See the NCSI Brief: Advantages of Assessing SiMR Progress by Tracking a Sample Over Time ( advantages-of-assessing-simr-progress-by-tracking-a- sample-over-time/).
Look at the effects of the EBP in single cohort rather than only comparing different groups of students (cross-sectional).
Especially relevant for Grade K-3 academics

31. What Is Data Visualization?
There are many different ways to present a finding (e.g., 80% fidelity or completion).
Source:
Adapted from Belodoff, K., Derrington, T., Reid, K., & Ridgeway A. (2016, August). See what I mean? Creating engaging and effective data products, presentations, and reports. Session presented at the Improving Data, Improving Outcomes Conference, New Orleans, LA.
Adapted from Belodoff, Derrington, Reid, & Ridgeway (2016).

32. Good Data Visualization…
Promotes data use
Communicates results
Portrays data accurately and clearly
Facilitates understanding
Is engaging and attractive
Is accessible
Good data visualization:
Portrays data accurately and clearly
We need to remember that data carry authority, so we want to portray the data accurately and clearly so that our audience trusts us and is able to make correct interpretations.
Is engaging and attractive
Makes the data interesting and even exciting.
Facilitates understanding
Good visuals show relationships and patterns quickly.
Drives home your key messages.
Promotes data use
Makes the data meaningful to the audience.
Source:
Adapted from Belodoff, K., Derrington, T., Reid, K., & Ridgeway A. (2016, August). See what I mean? Creating engaging and effective data products, presentations, and reports. Session presented at the Improving Data, Improving Outcomes Conference, New Orleans, LA.
Adapted from Belodoff, Derrington, Reid, and Ridgeway (2016).

33. Data Visualization Toolkit

34. Making It All Happen—Data Steps/Roles and Quality
Quality Question(s)
Collection
Measures administered correctly? At all sites?
Entry
Data entered accurately?
Storage
Accurate transmission?
Linked to other needed data sets?
Security/confidentiality?
Backups?
Analysis
Does the analysis method match your evaluation question?
Who will verify formulas and codes?
Reporting
Are findings understandable for the intended audience(s)?
Use
Who participates in reviewing findings and making decisions?
What are your decision rules for when changes are needed?
Who is working on each of these steps?
Emphasize importance of buy-in.
Some measures may require PD and administration or corater checks. Observation protocols, self-report logs or checklists, student assessments, and so on.

35. Using Evaluation Findings
Focus data collection and analysis efforts on what you will use!
Are we on track?
If so, how do we know?
If not, what should we change?
Present findings in a way that stakeholders understand so they can be meaningfully involved in decision making.

36. Discussion and Questions

37. Laura Kuchle (lkuchle@air.org) and Gena Nelson (gnelson@air.org)
For additional information, support, and technical assistance: • Contact your NCSI TA facilitator • Submit your question on Ask the NCSI • Contact Kristin Ruedel, NCSI Data Use  Service Area Lead, at
Presenters:
Lou Danielson and Kristin Ruedel
Contributors:
Laura Kuchle and Gena Nelson

38. THANK YOU! http://ncsi.wested.org | @TheNCSI