1. CCSA Summit R&D Team March 2018

2. Present a Problem, Get Commitment
Lesson: Get stakeholder buy-in. How urgent and important is this problem?
Learning: Make the problem compelling.
Growing Gap Over Time in Number of Assessments Behind
by *EL Versus Non-EL Students
*EL (blue) vs. non-EL (orange)
Late in the school year, our analyses had evolved in 3 ways:
We started using over-time representations to understand the changes in our system.
We were breaking down Incompletes by their 3 causes - overdue projects, below grade-level (70%) project (cognitive skill) scores, or being behind on “Focus Areas” (not passing a requisite number of assessments by a certain date)
We started disaggregating our data by subgroups.
We got a lot of signals like this that show some of the inequities built into our system. This graph shows the average # of assessments behind for our English Learners (blue) as compared to our non-English Learners (orange). We picked up this signal across a wide range of other measures (it took them more attempts to pass an assessment, they had more overdue projects, etc.). It showed us that as the year progressed, our English Learners were falling ever further behind and at a higher rate than their peers. Because we had seen some success with the approach, there was no formal work being done on EL across the organization (like SPED), and that this intersected with other historically underserved populations, we proposed an improvement project focused on addressing their needs and reducing these gaps (outcome gaps due to opportunity gaps).
When we presented the data to our Summit School leaders and asked them how urgent and important the problem was to solve, we got ratings seen in the photo above. We had a fortunate staffing consequence that had allowed for the creation of 25% “Teacher on Special Assignment” roles and 5 of them signed on to lead this improvement work at their respective sites. They were trained and we initiated lines of testing over the coming year.

3. Learn from “Positive Deviants”
Teachers successful with all students + EL students
Learn from “Positive Deviants”
Fewer EL Incompletes --->
Each bubble is a teacher. The diagonal line is where the subgroup performance matches the population performance.
We were able to identify the teachers that had strong outcomes. We talked to them to learn about their practices. We solicited their feedback on our working theory and the direction that we were taking. In many ways, it confirmed what we had hypothesized: these teachers were to a person explicitly teaching academic language (vocabulary, reading strategies, writing strategies). Seeing these representations also made us wonder how we could make more strategic movement “up and to the right” by identifying and spreading the core instructional practices.
This representation also told us something was going on with this particular population because the spread was much larger than it was across other demographic groups (SPED looked similar), more than race or SES.
Represents an example of systemization of what we started as a ‘quick and dirty’ approach.
Fewer Whole Population % Incompletes ---->

4. Relentlessly measure progress towards your aim
Make your graph less busy and easier to interpret if you can!!
Gap in Performance Between
ELs and Non-ELs
Size of *EL / non-EL gap year over year (shrinks over time)
More EL improvement than non-EL
School Year
Max Off Track
EL Max Off Track
53%
79%
40%
62%
33%
55%
Dark lines =
Light lines =
*Blue to blue is the reduction of EL incompletes from year to year, while orange to orange is the non-EL improvement.
The problem is illuminated when we look at subgroup data. ELs are the blue lines, orange are the non-ELs and gray/black are still the overall population. You can see that the same pattern exists, but the magnitude is much greater. You see a tightening of the gap over time (dark lines current year, lighter lines previous year). ELs saw more improvement (light blue to dark blue) than non-ELs). By the end of the year, EL performance was comparable with non-EL performance the previous year.
When looking at graphs of the “positive deviants”, and even broken down at the site level, you see deviations from this pattern that we are trying to learn from by further embedding researchers within sites. We have figured out how to “solve” this problem in localized cases but not yet at scale.
Our “next problem to solve” is how we can actually break this pattern - what is it going to take to keep Incompletes from ramping up and holding more or less constant?

5. Be “scrappy” with your data
Pro Tip: Clipboards are a teacher’s best friend!!
Make your best guess at useful measurements, try to make it easy, and useful. Let it evolve over time.
When the data you want doesn’t exist, go and get it.

6. Let your understanding of population and aim change with new data
% on grade % “on track” % incomplete
(>85%) (85%>70%) (<70%)
Low MAP
Mid MAP
High MAP
Note, this slide (or the one prior to it) could get dropped if we wanted to stick with the “evolving nature of our understanding of the problem through time series” or if we want to see another way. But this encapsulates the problem we are currently trying to solve.
This is the current problem to solve as we see it. Low and High are students 1 std dev beyond the grade-level mean, everyone within the Mid category is within 1 std dev of the grade-level mean.
Our goal is to move kids down and to the left on this representation. But this is the most significant % of incompletes for any subgroup (higher than EL, higher than SED, higher than race). We do notice that there are some differences by race (e.g. white students in the low MAP category are doing better than s.o.c. in the same category).

7. Timeline of Improvement
Fall 2015
Spring 2016
Summer 2016
Fall 2016
Spring 2016
Now
Small team of 3 teachers + CMO Academics and Data Team members
EL Data Analysis
Top-level buy-in, recruited 25% Teachers on Special Assign.
Developed EL Driver Diagram
Developed structures, trained on improvement
Established testing ramps
25%ers testing, leading site-level PD
Positive Deviant Analysis
Draft Change Package
Spread Change Package through org-wide PD
Demographic Analysis for Literacy
Revise Theory
Systemize EL in platform and curriculum. Hired EL coordinator
Recruit teachers across network around literacy / numeracy
EL Improvement
Transitions from EL to Literacy / Numeracy
Systemization of EL, Establishing Literacy / Numeracy