District of Columbia Public Schools | 1200 First Street, NE | Washington, DC | T | F | Lessons Learned in the DC Public Schools July 26, 2011 Student-Teacher Data Links

Goals, Relevance and Outline  Goals  Roster confirmation (RC): to verify student-teacher data links i.e., to connect the right students to the right teachers and to ensure we know who teaches whom what when.  Presentation: to assist other SEAs/LEAs with the successful and efficient implementation of roster confirmation by capturing those aspects of the project that were critical to project success and issues that require particular attention. Lessons Learned District of Columbia Public Schools, July 20112

Goals, Relevance and Outline  Relevance:  Mathematica Policy Research conducted analyses based on a preliminary roster validation process from : “We measure the effect of classification error on value-added estimates of teacher effectiveness by comparing results generated with the original administrative roster data to results using the validated roster data. (…) About one in seven teachers in our data are mismatched with entire classrooms of students because they did not teach math, reading, or both to their students.” (Hock and Isenberg, 2010 – emphasis added)  These results assumed that teachers with at least ten students would get an estimate. If that number were changed to something larger, an even higher teacher mismatch rate might be expected. Lessons Learned District of Columbia Public Schools, July 20113

Goals, Relevance and Outline  Outline  Overview of IMPACT  Overview of the RC process  Defining the success of RC  Key technical elements of RC  Keys to the success of RC  Threats to the success of RC Lessons Learned District of Columbia Public Schools, July 20114

Outline  Overview of IMPACT  Overview of RC  Defining the success of RC  Key technical elements of RC  Keys to the success of RC  Threats to the success of RC Lessons Learned District of Columbia Public Schools, July 20115

Where We Were In 2007 IMPACT OVERVIEW District of Columbia Public Schools | Summer % vs. 95% 8th Grade Reading Proficiency (2007 NAEP) Teachers Meeting or Exceeding Expectations

Simple Goal For Our Teacher Effectiveness Work IMPACT REFLECTION District of Columbia Public Schools | Summer Create a system in which every parent would be satisfied randomly assigning their children to any classroom in DCPS.

IMPACT Overview Lessons Learned District of Columbia Public Schools |  IMPACT is DCPS’ performance management system for teachers and other school-based employees  Goals:  Reward high-performing teachers  Provide clear feedback and guidance for growth  Transition out underperforming teachers

IMPACT Overview  Important because “…having three years of good teachers…in a row would overcome the average achievement deficit between low- income kids…and others. In other words, high-quality teachers can make up for the typical deficits that we see in the preparation of kids from disadvantaged backgrounds.” Eric Hanushek Stanford University (in Izumi and Evers, Teacher Quality, 2002)  Collaborative process with substantial input from the DCPS community, prominent researchers and assessment systems from all major school districts around the country  Measures of effectiveness tailored to 20 different staff categories and based on multiple components, each with a detailed scoring rubric Lessons Learned District of Columbia Public Schools |

IMPACT Overview Lessons Learned District of Columbia Public Schools |

IMPACT Overview Lessons Learned District of Columbia Public Schools |  Five observations per year:  Three by an Administrator (Principal or Assistant Principal)  Two by a Master Educator

IMPACT Overview  Overall IMPACT Scale  100 Points – 400 Points  Four Ratings: Highly Effective, Effective, Minimally Effective, and Ineffective Lessons Learned District of Columbia Public Schools |

IMPACT Overview  The Ratings and Their Implications  Highly Effective: additional compensation pending union agreement  Effective: normal salary advancement  Minimally Effective:  Additional professional development  Salary “hold”  Separation after two consecutive years  Ineffective: separation Lessons Learned District of Columbia Public Schools |

Outline  Overview of IMPACT  Overview of the RC process  Defining the success of RC  Key technical elements of RC  Keys to the success of RC  Threats to the success of RC Lessons Learned District of Columbia Public Schools, July

Overview of the RC Process  Pilot  Pilot at scale  Contract  Business process  Tool is built  Tool goes live  Initial data  Follow ups  Final data Lessons Learned District of Columbia Public Schools, July

Outline  Overview of IMPACT  Overview of the RC process  Defining the success of RC  Key technical elements of RC  Keys to the success of RC  Threats to the success of RC Lessons Learned District of Columbia Public Schools, July

 High educator participation  What percent of teachers are expected to complete roster confirmation?  Critical data (100%)  Subject, roster, dosage  Minimal time commitment  Minimal direct support necessary  Hotline, , in-person sessions, school sessions Lessons Learned District of Columbia Public Schools, July Defining the success of RC

Outline  Overview of IMPACT  Overview of the RC process  Defining the success of RC  Key technical elements of RC  Keys to the success of RC  Threats to the success of RC Lessons Learned District of Columbia Public Schools, July

RC was designed with the following User Profiles in mind:  Administrator / central office role  What: design, development, testing, and operationalization.  Who: developers and business owners (central office personnel who verified the data entered by teachers and conducted follow ups).  Access: full.  Teacher role  What: target audience, confirmed their rosters and assigned dosages to students.  Who: teachers.  Access: data entry, access to guidelines, instructions, FAQs and ability to view the data they entered by logging in after submitting their roster. Lessons Learned District of Columbia Public Schools, July Key Technical Elements of RC

A custom front-end to the Quickbase Roster Confirmation online application was built to provide the following features to users:  Ability to perform RC in a user friendly interface that would allow teachers to add students to their roster from a dropdown containing all students at their school.  Ability to read the guidelines before doing RC and to refer to the guidelines, FAQs and instructions at any given point while using the application.  Automated quality-control checks: the system recognizes common mistakes and stops the teacher from submitting results until mistakes are fixed. Lessons Learned District of Columbia Public Schools, July Key Technical Elements of RC

 Underlying code that runs the data manipulation for the application retrieves the student roster for each teacher, then loads it on to the user interface and captures the dosage data entered by the teachers.  The application was designed and tested on different browsers. Lessons Learned District of Columbia Public Schools, July Key Technical Elements of RC

Identification and manipulation of seed data:  Obtain list of schools with the contact information of the School/Principal.  Classify schools as Elementary, Middle, Bilingual, and Departmentalized.  Obtain list of students from the student record system.  Identify teachers in the tested grades and subjects.  Design rules to flag potential errors and a process of follow up to correct the information.  Identify appropriate teacher ID for best match across databases, and s.  Identify Homerooms, and Math and Reading courses.  Classify teachers as Math, Reading or both (Homeroom) - may require calls to schools).  Assign appropriate dosages. Lessons Learned District of Columbia Public Schools, July Key Technical Elements of RC

 A relational Quickbase database was used, consisting of four major tables:  A school table with a school code.  A teacher table with a teacher ID as well as a “related” school ID.  A student table with a student ID as well as an “attended” school ID.  A student-to-teacher assignment (STA) table with all three IDs.  After follow-ups, a teacher table and an STA table with the relevant elements were provided to the value-added contractor, including input data, and initial and final output data. Lessons Learned District of Columbia Public Schools, July Key Technical Elements of RC

Outline  Overview of IMPACT  Overview of the RC process  Defining the success of RC  Key technical elements of RC  Keys to the success of RC  Threats to the success of RC Lessons Learned District of Columbia Public Schools, July

 What are the priorities?  What are teachers asked to verify?  The subject(s) they teach? AND/OR  Their roster? AND/OR  The dosage (the estimate of their contribution to each student’s instruction)?  What system/process is appropriate? e.g., role for schools/Principals? Lessons Learned District of Columbia Public Schools, July Keys to the success of RC

 When does the tool go live? When should application development begin? When must exact requirements be ready?  Which teachers will complete roster confirmation? How soon do we need to know?  What is a reading course? What is a math course? What is a homeroom? Which schools are departmentalized/bilingual and how does it work? How much of this does the administrative data capture? Can scheduling data be leveraged?  What are the must-haves for the first year? What are the nice-to-haves to add once the first year has rolled out successfully? Lessons Learned District of Columbia Public Schools, July Keys to the success of RC

 Who should be involved?  E.g., teacher human capital, accountability, legal, data, technical, teaching and learning  How many people from each team? Who? Role? Who provides input? Who decides?  How to ensure ongoing alignment? Meet? How often?  Is the language in the application both accurate and clear? Is it consistent with other communications materials? Are there legal/Union issues? Lessons Learned District of Columbia Public Schools, July Keys to the success of RC

 How many rounds?  What’s best? What’s feasible?  How many rounds of follow-ups?  To what extent can issues be identified in advance?  What is the capacity for last-minute issues?  What is the role of the value-added contractor?  Are there legal/Union issues here? Lessons Learned District of Columbia Public Schools, July Keys to the success of RC

Outline  Overview of IMPACT  Overview of the RC process  Defining the success of RC  Keys to the success of RC  Key technical elements of RC  Threats to the success of RC Lessons Learned District of Columbia Public Schools, July

 Unclear understanding of the importance of data accuracy  Lack of clarity in messaging and instructions  Highly complex concepts: one explanation does not fit all audiences and impractical to present all explanations to all audiences  Difficulty in getting everyone’s attention to a sometimes high level of detail  What are the options?  In person/in-school sessions Lessons Learned District of Columbia Public Schools, July Threats to the success of RC

 Process to verify teacher entries  Role of co-teachers, administrators  Legal/Union rules and teacher privacy  Teachers not even entering the application and central office not knowing why  Balancing of thorough follow ups with capacity  Very few changes in some follow-up categories Lessons Learned District of Columbia Public Schools, July Threats to the success of RC

Contact Information Anna Gregory Hella Bel Hadj Amor, Ph.D. Director, IMPACT Operations Senior Researcher District of Columbia Public Schools American Institutes for Research Office of Human Capital Education, Human Development and the Workforce 1200 First Street NE 10FL 1000 Thomas Jefferson Street, NW Washington, DC Washington, DC T T F F E E Lessons Learned District of Columbia Public Schools, July