1. OEA Leadership Academy 2011 Michele Winship, Ph.D. winshipm@ohea.org

2.  Evidence of growth in student learning and competency  Standardized tests, pre/post tests in untested subjects  Student performance (art, music, etc.)  Curriculum-based tests given in a standardized manner  Classroom-based tests such as DIBELS  Evidence of instructional quality  Classroom observations  Lesson plans, assignments, and student work  Student surveys such as Harvard’s Tripod  Evidence binder (next generation of portfolio)  Evidence of professional responsibility  Administrator/supervisor reports, parent surveys  Teacher reflection and self-reports, records of contributions

3.  Accountability measures under No Child Left Behind have greatly increased the amount of student performance data that is collected and reported.  In Ohio, student performance data generally refers to the data collected from required tests that measure students on what they know and are able to do in mathematics, reading, science, social studies and writing, with administration to students spread out from third to eighth grade 3

4.  Measuring student performance and then ranking schools, districts, states and countries is widely believed to be a sound method for improving student achievement and driving school and district improvement efforts.  However, constant “measuring” is not what drives improvement or achievement…  No matter how many times you weigh the cow, it doesn't get any fatter. 4

5. 5

6.  Used properly, student performance data DOES have a role in school and district improvement efforts, it CAN positively impact student performance.  Nationally, we have come to believe that the data itself—the “score”—is the end game instead of the starting point.  And…an overreliance on and faith in value- added metrics as accurate measures of TEACHER performance has entirely skewed the way we use student performance data. 6

7.  To be meaningful, student performance data should be used by educators to  Identify achievement gaps,  Inform instructional practice, and  Direct professional development.  To effectively use the data, teams of educators should  Be trained in the analysis and interpretation of student performance data,  Have real-time access to the data, and  Meet regularly in teams to analyze the data and plan intervention, instruction and professional development. 7

8. How do we create the conditions for educators to use student performance data effectively? 8

9.  Value-Added Modeling (VAM) has become the “gold” standard for measuring educator effectiveness.  One year’s growth in one year’s time is the benchmark = effective.  Teachers who exceed this growth rate have a positive value-added rating (+) = highly effective  Teachers who fail to meet this growth rate have a negative value-added rating (-)  HB 153 supplements districts rated excellent at $17 per student. 9

10.  BUT…VAM modeling is flawed.  The tests used to generate the scores were never designed to measure teacher effectiveness.  “Student test scores alone are not sufficiently reliable and valid indicators of teacher effectiveness to be used in high- stakes personnel decisions, even when the most sophisticated statistical applications such as value-added modeling are employed.” (EPI Briefing Paper--Problems with the Use of Student Test Scores to Evaluate Teachers) 10

11.  Given that students are not randomly assigned to classes, VAM can’t distinguish between teacher effects and the effects based on students’ needs.  VAM do not provide information to help “struggling” teachers.  Lack of properly scaled year-to-year tests makes it difficult to evaluate gains along the continuum.  Mobility of students (especially in high needs schools) impact the data  VAM cannot distinguish among teachers in the middle range of performance. 11

12.  About 69% of teachers can’t be accurately assessed with VAMs*  Teachers in subject areas that are not tested  Teachers in grade levels (lower elementary) where no prior test scores are available  Special education & ELL  VAM estimates vary with the tests used  If a teacher is in the bottom quintile based on one test there is a 43% chance she will be in the bottom quintile on a different test, but a 16% chance she will be in the top two quintiles.  If a teacher is in the top quintile based on one test there is a 43% chance she will be in the top quintile on a different test, but a 13% chance she will be in the bottom two quintiles. 12

13.  VAM estimates have proven to be unstable across statistical models, years, and classes that teachers teach.  Another (study) found that teachers’ effectiveness ratings in one year could only predict from 4% to 16% of the variation in such ratings in the following year. Thus, a teacher who appears to be very ineffective in one year might have a dramatically different result the following year. The same dramatic fluctuations were found for teachers ranked at the bottom in the first year of analysis.  This runs counter to most people’s notions that the true quality of a teacher is likely to change very little over time and raises questions about whether what is measured is largely a “teacher effect” or the effect of a wide variety of other factors.” (EPI Briefing Paper--Problems with the Use of Student Test Scores to Evaluate Teachers) 13

14. How’s My Performance? Y ijt − Y ijt−1 = β − δ + 2δt + β i + β ij + γ ′ ( x ijt − x ijt−1 ) + θ t(ij) +e ijt − e ijt−1 14

15.  HB153 requires student performance to factor in at 50% of teacher and principal evaluation.  Currently value-added data is not available for up to 80% of teachers, not to mention counselors, nurses and other educators.  It is imperative that we build an assessment system that gives us data to use effectively to improve our own practice, as well as provide evidence of student performance that will be required in evaluation. 15

16.  Sec. 3319.112.  (A) Not later than December 31, 2011, the state board of education shall develop a standards- based state framework for the evaluation of teachers. The framework shall establish an evaluation system that does the following:  (1) Provides for multiple evaluation factors, including student academic growth which shall account for fifty per cent of each evaluation;  (2) Is aligned with the standards for teachers adopted under section 3319.61 of the Revised Code;  (3) Requires observation of the teacher being evaluated, including at least two formal observations by the evaluator of at least thirty minutes each and classroom walkthroughs;  (4) Assigns a rating on each evaluation in accordance with division (B) of this section;  (5) Requires each teacher to be provided with a written report of the results of the teacher's evaluation;  (6) Identifies measures of student academic growth for grade levels and subjects for which the value-added progress dimension prescribed by section 3302.021 of the Revised Code does not apply;  (7) Implements a classroom-level, value-added program developed by a nonprofit organization described in division (B) of section 3302.021 of the Revised Code;  (8) Provides for professional development to accelerate and continue teacher growth and provide support to poorly performing teachers;  (9) Provides for the allocation of financial resources to support professional development. (HB 153 as signed by the Governor) 16

17. Best Case Scenario:  Student learning data used only in a formative assessment process.  Teacher practice and student learning data assessed.  Teachers rated or given a metric of effectiveness based on both practice and student learning data.  Teachers given useful feedback related to their practice, skills, and knowledge.  Teachers given individualized professional development and support based on this feedback.  Teachers later evaluated based on the growth and/or improvement of their practice, skills, and knowledge – not student learning data. 17

18. Worst Case Scenario:  Student performance is limited to value-added data.  Teacher practice is determined effective or ineffective based on a single metric.  Teachers without value-added data are “assigned” a score.  Teachers given no feedback related to their practice, skills, and knowledge.  Evaluation is not connected to professional development.  Teachers are evaluated with student performance scores comprising 50% or more of the evaluation rating. 18

19. www.tqsource.org Questions to ask about measures of teacher effectiveness 1.Rigorous. Are measures “rigorous,” focused on appropriate subject/grade standards? Measuring students’ progress towards college and career readiness? 2.Comparable. Are measures “comparable across classrooms,” ensuring that students are being measured with the same instruments and processes? 3.Growth over time. Do the measures enable student learning growth to be assessed “between two points in time”? 19

20. www.tqsource.org Questions to ask about measures of teacher effectiveness (cont’d) 4.Standards-based. Are the measures focused on assessing growth on important high-quality grade level and subject standards for students? 5.Improve teaching. Does evidence from using the measures contribute to teachers’ understanding of their students’ needs/progress so that instruction can be planned/adapted in a timely manner to ensure success? 20

21. www.tqsource.org Questions to ask about teacher evaluation models* 1.Inclusive (all teachers, subjects, grades). Do evaluation models allow teachers from all subjects and grades (not just 4-8 math & reading) to be evaluated with evidence of student learning growth according to standards for that subject/grade? 2.Professional growth. Can results from the measures be aligned with professional growth opportunities? *Models in this case are the state or district systems of teacher evaluation including all of the inputs and decision points (measures, instruments, processes, training, and scoring, etc.) that result in determinations about individual teachers’ effectiveness. 21

22.  Multiple Measures of Student Learning  Evidence of growth in student learning and competency  Standardized assessments (state/district tests)  Evidence collected by teachers and scored by groups of educators  The 4 Ps: portfolios, projects, products, and performances  Essays, written responses to complex questions  Evidence collected and scored in classrooms  Classroom-based assessments such as DRA, DIBELS, curriculum-based tests, unit tests Laura Goe 22

23. www.tqsource.org Rhode Island DOE Model: Framework for Applying Multiple Measures of Student Learning Category 1: Student growth on state standardized tests (e.g., NECAP, PARCC) Student learning rating Professional practice rating Professional responsibilities rating + + Final evaluation rating Category 2: Student growth on standardized district-wide tests (e.g., NWEA, AP exams, Stanford- 10, ACCESS, etc.) Category 3: Other local school-, administrator-, or teacher- selected measures of student performance The student learning rating is determined by a combination of different sources of evidence of student learning. These sources fall into three categories: 23

24. www.tqsource.org Rhode Island Model: Student Learning Group Guiding Principles “Not all teachers’ impact on student learning will be measured by the same mix of assessments, and the mix of assessments used for any given teacher group may vary from year to year.” Teacher A (5 th grade English) Teacher B (11 th grade English) Teacher C (middle school art) Category 1 (growth on NECAP) Category 2 (e.g., growth on NWEA) Category 3 (e.g., principal review of student work over a six month span) Teacher A’s student learning rating + += Category 2 (e.g., AP English exam) Category 3 (e.g., joint review of critical essay portfolio) Teacher B’s student learning rating += 24 Category 3 (e.g., joint review of art portfolio) This teacher may use several category 3 assessments Teacher C’s student learning rating =

25. 25

26. 26

27.  Use the assessments you have first.  Determine what assessments you need to create a rigorous, comparable and inclusive assessment system that is designed to provide student performance data to be used for educator professional growth and also for inclusion in an evaluation system.  Chart a course of action with a timeline, persons responsible and deliverables. 27

28.  Requiring student performance in teacher evaluations means districts will need to: 1. Map current school-based and district-wide assessments in all grades and subjects 2. Determine where assessment “gaps” exist 3. Create groups of educators to select/develop appropriate assessments for “gaps” 4. Create an assessment timeline for all grades and subjects 5. Collect, analyze and store student performance data 6. Provide time and training for educators to work together with student data to improve their own instruction 28

29. 29

30.  Laura Goe--Webinar for Oregon School Coaches, April 20, 2011: http://www.lauragoe.com/LauraGoe/Oregon -April%202011.pptx http://www.lauragoe.com/LauraGoe/Oregon -April%202011.pptx  EPI Briefing Paper--Problems with the Use of Student Test Scores to Evaluate Teachers: http://www.epi.org/publications/entry/bp27 8 http://www.epi.org/publications/entry/bp27 8  Rand Education—Evaluating Value-Added Models for Teacher Accountability: http://www.rand.org/pubs/monographs/200 4/RAND_MG158.pdf http://www.rand.org/pubs/monographs/200 4/RAND_MG158.pdf 30

31.  Michele Winship  614-227-3001  winshipm@ohea.org winshipm@ohea.org 31