1. Big Data, Education, and Society
April 4, 2018

2. Assignment 3
Let’s go over assignment 3

3. Implementation Fidelity
NIH definition
“Implementation fidelity is the degree to which an intervention is delivered as intended…”

4. Questions to ask during implementation (Feng et al., 2014, p. 564)
“Is this the quality of implementation we expected as creators of the intervention?”
“What actions can we take that might bring implementation up to our desired levels?”

5. The peril of implementation fidelity
You design a brilliant innovation

6. The peril of implementation fidelity
You design a brilliant innovation
You refine it in carefully-controlled settings

7. The peril of implementation fidelity
You design a brilliant innovation
You refine it in carefully-controlled settings
Well-designed teacher professional development

8. The peril of implementation fidelity
You design a brilliant innovation
You refine it in carefully-controlled settings
Well-designed teacher professional development
Buy-in from school administrators

9. The peril of implementation fidelity
You design a brilliant innovation
You refine it in carefully-controlled settings
Well-designed teacher professional development
Buy-in from school administrators
Monitoring of ongoing implementation

10. The peril of implementation fidelity
You design a brilliant innovation
You refine it in carefully-controlled settings
Well-designed teacher professional development
Buy-in from school administrators
Monitoring of ongoing implementation
You then throw it out into the cold, cruel world

11. Failure to take medicine

12. Carnegie Learning RAND study

13. Chartiers Valley Substitute Teacher

14. Does anyone have any personal examples of implementation fidelity failures to share?

15. Categories of implementation fidelity (Dane & Schneider, 1998)
Adherence – are program components delivered as prescribed?
Exposure/dosage – are program components delivered as much as intended? (and in the right proportions?)
Quality – are program components delivered in the “theoretical ideal” intended fashion?

16. What attributes influence implementation fidelity (Carroll et al
Intervention complexity – and prescriptiveness
Support – by developer, school, district
Participant responsiveness – how interested/willing are teachers, and how difficult is intervention for them to adopt

17. How can we improve odds of good implementation fidelity?
Your thoughts?

18. Reasoning Mind approach (Khachatryan et al., 2014)
Regional implementation coordinators (1 per 37.7 teachers)
Look at data on student engagement and performance to identify problem spots
Visit classrooms periodically and conduct observations according to a detailed rubric (average = 6 visits/year)
Meet with teachers after observations to provide professional development on classroom issues
This was rated as teachers’ favorite part of adopting the curriculum

19. Reasoning Mind approach (Khachatryan et al., 2014)
Rubric
Does teacher use analytics reports to make instructional decisions
Does teacher plan lesson activities and student interventions before class
Does teacher conduct varied interventions with students in need
Proportion of class time students spend using system
Do students use all system features
Does teacher engage with students during class

20. Reasoning Mind approach (Khachatryan et al., 2014)
Rubric (continued)
Does teacher use effective classroom management procedures
Does teacher establish clear goals and rewards for individual students and entire class
Do students have well-organized notebooks that show student work
Do students use recommended learning strategies
Are students on-task

21. Reasoning Mind approach (Khachatryan et al., 2014)
Curriculum modification
Use data from regional coordinator visits to identify areas where implementation fidelity is generally low
Encouraging students to take notes and show written work
Checking student notes and written work
Modify automated curriculum to better scaffold these areas for both teachers and students
Modifying teacher professional development to emphasize these areas (2 days before school year, six half-day workshops during school year)

22. Thoughts What do you like about this approach?
What do you dislike about this approach?

23. Feng et al. (2014) ASSISTments approach
12-hour (2-3 day) “Best practices” workshop with teachers at beginning of year
Count as state professional development credit
Beginning-of-year interviews with principals
Beginning-of-year teacher survey
Analyze system log data during year
Classroom observations of teacher practices
End-of-year teacher interviews

24. Feng et al. (2014) ASSISTments log data analysis
How often do teachers assign homework in ASSISTments?
What are homework completion rates?
How long do students spend on homework?
Which teachers are not opening homework reports?

25. Feng et al. (2014) actions taken
Visit teachers who did not use system as intended, with targeted plans for which behaviors to coach
Change agenda of “best practices” workshop to match general issues
Modify design of reports given to teachers

26. Thoughts What do you like about this approach?
What do you dislike about this approach?

27. Carnegie Learning Implementation Fidelity Approach (Pane et al
Carnegie Learning Implementation Fidelity Approach (Pane et al., 2014, p. 129)
3 days of professional development/training during summer
1 visit from PD staff to a school during year
PD staff “observe classrooms, offer recommendations, and help teachers address any problems they are having with implementations”
“Teachers also receive a set of training materials, an implementation guide, and a book of resources and assessments.”

28. Thoughts What do you like about this approach?
What do you dislike about this approach?

29. Karam et al., 2017
Studied implementation fidelity of Cognitive Tutor in real-world use (self-report surveys)
Only 45% of HS teachers reported using software for prescribed amount of time
Only 14% of HS teachers reported working with non-software recommended practices for prescribed amount of time
Only 30% of HS teachers reported spending as much time during software use working with students as prescribed

30. Effectiveness of PD
Correlation between more PD and prescribed pedagogical practices in Reasoning Mind (Miller et al., 2015)
No correlation between more PD and prescribed pedagogical practices in Cognitive Tutor (Karam et al., 2017)

31. Questions? Comments?

32. How Does Adoption Occur?
Makes a lot of difference to how good implementation fidelity will be
State-level curriculum approval processes
Top-down decisions based on sales
Curriculum reviews by groups of teachers
Individual teachers’ decision-making
Individual learners’ decision-making

33. Questions? Comments?

34. Activity
You are the implementation director for Bob’s Discount Math Curriculum (BDMC)
1000 teachers use your system
Each teacher’s classes produce $3000 of profit after all expenses except implementation/PD/support

35. Costs
Assuming
Salary is $50,000 year plus double for benefits, equipment, office space
In other words
Daily cost of implementation team member is $300
Daily site visit cost of implementation team member is $500

36. Sample Costs "trouble-spotting data analysis" 1 day 350
specific teacher data analysis
0.5 days
150
specific teacher classroom observation
500
single-teacher PD
250
full-day multi-teacher PD
2 days
1000
half-day multi-teacher PD
multi-teacher PD prep
3 days
900
books and materials
100

37. Create a plan for implementation
In groups of 3-4
What forms of support will you use?
How much profit is left over after support?

38. Each group read out your final profit

39. Each group read out your final profit
Highest-profit group and lowest-profit group
Please read out what implementation support you offered, and why you chose this combination

40. Should this calculation change…
Between a non-profit like Reasoning Mind
And a for-profit like Carnegie Learning
What are the risks of too low a profit for each type of organization?

41. Questions? Comments?

42. Upcoming office hours
April am-1030am or by appointment