1. Illuminating the Black Box: Assessing and Monitoring Impact
Joe Schroeder, PhD
Associate Executive Director, AWSA
@joeschroeder23

3. Black-Box Evaluation Teachers Student achievement in reading increases
receive 25 hours of training.
Student achievement in reading increases

4. Phases of the Change Process
Initiation → Implementation → Institutionalization

5. Professional Learning
Student Impact

6. Developing Deep Skill Within Your Areas of Focus
Adult Learning Mechanisms
Professional Development
Teacher Teams
Coaching/Mentoring
Practice
of New Knowledge
with Feedback
Improved Instructional Practices
Improved Student Learning
This has proven to be a very helpful model to many SAIL participants to date in thinking about how they assist their people develop deep learning and skill.
Emphasize where the problems break down and why we often don’t get the level of effect that Hattie describes from his research. According to Joyce and Showers (2002), it takes between 20 and 50 trials (depending on the complexity of the skill) before learners feel relatively confident in using a new practice. This means you have to provide people with extended opportunities for practice, paired with feedback, if you really expect them to effectively use new practices.
Brian McNulty of the Learning and Leadership Center (p. 184 of Activate): “The single biggest gap that we see in schools and districts is the lack of monitoring and feedback.”
Developing “deep skill” is really about PROVIDE DELIBERATE PRACTICE, which is chronicled well in in Chap. 4 of Activate: (p. 70)
“Developing talent in our teaching force is not about all-inclusive checklists that administrators use for compliance. Rather, teachers develop skill by focusing on a few strategies that they practice again and again with feedback on their progress by instructional masters. Reeves (2010) states that the components of deliberate practice include performance that is focused on a particular element of the task, expert coaching, feedback, careful and accurate self-assessment, and the opportunity to apply feedback immediately for improved performance.
P. 181 of Activate: The biggest difference between teachers – not what they know, but what they do.” The same could be said for administrators.
P. 190 of Activate: The principal must lead the learning in instruction, and for our purposes this means that he or she must actively lead the learning of the two instructional improvement strategies (research-based instructional practices and the IDTs, teacher-based Instructional Data Teams). Principals need to know, or learn, what the instructional Practices and Data Teams look like when they are being implemented well. They will need to be able to provide guidance to both the BDT (Building-level Data Team) and the IDTs. 6

7. Monitoring AND Diagnosing Your Strategy Implementation
Leading Indicators
Lagging Indicator
Adult Learning Mechanisms
Professional Development
Teacher Teams
Coaching/Mentoring
Improved Instructional Practices
Improved Student Learning
Practice of New Knowledge with Feedback 7

8. Glass-Box Evaluation
Actions
Results
Killion, Assessing Impact, 2008

9. SAMPLE GLASS-BOX EVALUATION
Resources
Trained coaches
Time for coaching
Actions
Teachers engage in professional development.
A skilled coach provides teachers with classroom support.
Classroom schedules are altered.
Initial Changes
Teachers’ knowledge about teaching reading increases.
Teachers gain understanding about students’ reading difficulties.
Students’ learning needs are identified
Intermediate Changes
Teachers’ capacity to provide differentiated, high-quality reading instruction increases.
Students receive differentiated, high-quality reading instruction.
Results
Achievement in reading increases for all students.
Gains for underperforming students are significant.

10. K A S B
nowledge
ttitude
kill
spiration
ehavior

11. 1 A 2 B 3 Type of Change Teachers Students Knowledge
Conceptual understanding of information, theories, principles, and research
Teachers understand mathematical concepts they are responsible for teaching.
Students have a deeper understanding of key mathematical concepts measured on criterion-referenced tests.
Attitude
Beliefs about the value of particular information or strategies
Teachers believe students’ competence in mathematics is important to their success,
both within and beyond school.
Students enjoy mathematics.
Skill
The ability to use strategies and processes to apply knowledge
Teacher know how to employ a variety of instructional strategies to help students
visualize mathematical concepts.
Students demonstrate their understanding of math on classroom tasks.
Aspiration
Desires, or internal motivation, to engage in a particular practice
Teachers have a genuine desire for their students to understand and perform well in mathematics.
Students want to advance their understanding of mathematics and aspire to advanced work in mathematics.
Behavior
Consistent application of knowledge and skills
Teachers consistently employ inquiry-based instructional practices in mathematics to help students acquire a deep understanding of
math concepts.
Students regularly apply mathematical reasoning to solve problems in math and other areas of their curriculum. 1 A 2 B 3
Indications of successful progress

12. Consolidation Time
What KASABs (for teachers and/or students) would best indicate that your school is making progress with your biggest improvement effort?

13. Monitoring AND Diagnosing Your Strategy Implementation
Leading Indicators
Lagging Indicator
Adult Learning Mechanisms
Professional Development
Teacher Teams
Coaching/Mentoring
Improved Instructional Practices
Improved Student Learning
Practice of New Knowledge with Feedback
Activate p. 179 – Doug Reeves has presented data on the critical importance of deep implementation. His studies in 2010 found that “half hearted implementation was actually worse than minimal or no implementation. His studies further reveal that unless you can implement at the 90 perent level, you should not expect to get outcomes that are promised in the research literature. It is probably fair to say that currently most schools and districts do not meet this 90 percent implementation standard”.
Moreover, in its admirable efforts of Continuous Improvement in Education, the Carnegie Foundation found this relevant, common theme: Data on lagging outcomes (i.e., student achievement) is generally used to inform and leverage changes in teacher practice, but without actually measuring the practices in which teachers engage.
So how well are we implementing instructional practice data? Let’s take out our cell phones and get a sense of where we currently stand with this throughout the cohort? 13

14. 1 A 2 B 3 Type of Change Teachers Students Knowledge
Conceptual understanding of information, theories, principles, and research
Attitude
Beliefs about the value of particular information or strategies
Skill
The ability to use strategies and processes to apply knowledge
Aspiration
Desires, or internal motivation, to engage in a particular practice
Behavior
Consistent application of knowledge and skills 1 A 2 B 3
Indications of successful progress

15. Type of Change
Teachers
Students
Knowledge
Conceptual understanding of information, theories, principles, and research
Attitude
Beliefs about the value of particular information or strategies
Skill
The ability to use strategies and processes to apply knowledge
Aspiration
Desires, or internal motivation, to engage in a particular practice
Behavior
Consistent application of knowledge and skills

16. Closure / Next Steps
What common problem of leadership can the use of KASABs help you manage?
What specific actions do you need to take in the next 100-days to secure ongoing progress per your KASABs?

17. Illuminating the Black Box: Assessing and Monitoring Impact
Joe Schroeder, PhD
Associate Executive Director, AWSA
@joeschroeder23