NORTH CAROLINA Educator Effectiveness Update to PD Leads August 27, 2012

Overview for Today Rationale Why we are doing this? Status Determining educator effectiveness status MET Research Key ideas from recent research Surveys Potential element in educator effectiveness model Standard 6 & 8 How we’ll include student growth in educator evaluations 6. MSLs Measuring growth in previously non-tested areas 11/14/2018 • page 2

Rationale Why this is the right time and the right work

Overview The central focus of READY is improving student learning ... by enabling and ensuring great teaching. 11/14/2018 • page 4

What is our goal? Before Teaching and Leading Develop effective teachers and leaders in preparation programs Great Teachers and Leaders An effective teacher in every classroom and leader in every school Student Readiness Achievement and growth for all students During Teaching and Leading Use meaningful evaluation and professional development to increase effectiveness of teachers and leaders 11/14/2018 • page 5

Measures of Effective Teaching Using research to build a strong educator effectiveness model

What is our goal? Big Question: What is the best approach to Educator Evaluation and how do we get NC there? 11/14/2018 • page 7

Measures of Effective Teaching 11/14/2018 • page 8

Measures of Effective Teaching This research suggests: Multiple measures are important. Including student growth improves objectivity of evaluation. 11/14/2018 • page 9

Months of Learning Gained or Lost Re-creation of chart from Gathering Feedback For Teaching, http://www.metproject.org/downloads/MET_Gathering_Feedback_Practioner_Brief.pdf Observation + Other Measures Months of Learning Gained or Lost State Math State ELA Observation Tool Top 25% +1.2 +.2 Bottom 25% -1.4 -.4 Observation Tool + Student Survey Top 25% +2.8 +.7 Bottom 25% -2 -.9 Observation Tool + Student Survey + Growth (Value-Add) Top 25% +4.5 +1.2 Bottom 25% -3.1 -1.3

Standards 6 and 8 Incorporating student growth into the NC Educator Evaluation System

  What We Have From the MET… Observation Tools Some Assessments to Measure Growth From the MET… Evaluation Tools  End of Grade End of Course VoCATs Observation Tools Assessments to Measure Growth Student Survey  Standards 1-7 Standard 8 End of Grade End of Course VoCATs Standards 1-5 Standard 6 Exploring Pilot in 47 LEAs in 2011-12

Observation + Other Measures What do we need? Standard 6 and 8 We need a state-adopted growth model and a fair 6 & 8 rating strategy Status We need an overall method to determine educator effectiveness status Measures of Student Learning (MSLs) For those grades and subjects that are currently non- tested, we need ways to measure growth 11/14/2018 • page 13

1 2 3 4 5 6 1 2 3 4 5 6 7 8 Standards 6 & 8 – The Basics Teachers Contribute to Academic Success Demonstrate Leadership Establish Environment Know Content Facilitate Learning Reflect on Practice Principals (and other Administrators) 1 2 3 4 5 6 7 8 Human Resource Leadership Strategic Leadership Instructional Leadership Cultural Leadership Managerial Leadership External Development Leadership Micro- political Leadership Academic Achievement Leadership 11/14/2018 • page 14

Standard 6 and 8 are measures of Growth Growth Model Teachers 6 Standard 6 and 8 are measures of Growth Contribute to Academic Success Principals 8 Academic Achievement Leadership Academic Achievement Leadership 11/14/2018 • page 15

Growth Model Teachers 6 We will use Educator Value-Added Assessment System EVAAS for standards 6 & 8 when possible Contribute to Academic Success Principals 8 Academic Achievement Leadership Academic Achievement Leadership 11/14/2018 • page 16

6 8 Growth Model Teachers Principals How do Value-Added models work? They measure growth by predicting how well a student will do on an assessment. How do they predict how well the student will do? They look at previous test scores and estimate how well the student should do at the end of the year. Every student must grow based on where they start. 6 Contribute to Academic Success Principals 8 Academic Achievement Leadership Academic Achievement Leadership 11/14/2018 • page 17

How will the ratings on Standards 6 & 8 work? Teachers 6 Contribute to Academic Success How will the ratings on Standards 6 & 8 work? Principals 8 Academic Achievement Leadership Academic Achievement Leadership 11/14/2018 • page 18

8 1 6 5 4 3 2 7 Principal Rating Categories Principals Instructional Leadership Cultural Leadership Human Resource Leadership Managerial Leadership External Development Leadership Micro- political Leadership Academic Achievement Leadership Strategic Leadership 5 Rating Categories 3 Rating Categories Not Demonstrated Developing Proficient Accomplished Distinguished Does Not Meet Expected Growth Meets Expected Growth Exceeds Expected Growth 11/14/2018 • page 19

1 2 3 4 5 6 Teacher Ratings Categories Teachers 5 Rating Categories Demonstrate Leadership Establish Environment Know Content Facilitate Learning Contribute to Academic Success Reflect on Practice 5 Rating Categories 3 Rating Categories Not Demonstrated Developing Proficient Accomplished Distinguished Does Not Meet Expected Growth Meets Expected Growth Exceeds Expected Growth 11/14/2018 • page 20

1 2 3 4 5 6 Ratings Why the difference? Teachers 5 Rating Categories Demonstrate Leadership Establish Environment Know Content Facilitate Learning Contribute to Academic Success Reflect on Practice 5 Rating Categories 3 Ratings Categories Why the difference? Identifying only three rating categories on standard 6 & 8 improves certainty of categorization.

Why is school-wide EVAAS growth included? Teacher Ratings in 2011-12 Rationale - MET Research - Standard 6 & 8 - Status - Support ▲ 6 Yearly Rating Does not Meet Expected Growth Meets Expected Growth Exceeds Expected Growth Teacher EVAAS Growth School-wide EVAAS Growth 70% 30% Weighted Average Why is school-wide EVAAS growth included? To encourage collaboration and collective ownership of overall outcomes. Note: In 2011-12, teachers without individual EVAAS growth will have school-wide growth for Standard 6.

Teacher Ratings in 2011-12 6 Yearly Rating Does not Meet Expected Growth Meets Expected Growth Exceeds Expected Growth Teacher-level EVAAS reports available on September 11 (estimated) Teacher sixth standard ratings available at end of September Sixth standard ratings available in two locations: Rating only will be back-populated into the McREL tool and available to see on the 2011 – 2012 summary rating forms Rating and component data available in EVAAS

Possible additional element Teacher Ratings in 2012-13 2012 – 2013 is the first year of data for all teachers and school administrators who have their own data Possible additional element 6 Yearly Rating Does not Expected Growth Meets Expected Growth Exceeds Expected Growth Teacher EVAAS Growth School-wide EVAAS Growth Student Surveys (?) Weighted Average 11/14/2018 • page 24

Principal Ratings Standards 8 rating will be determined using school-wide EVAAS growth 8 Yearly Rating Does not Meet Expectations Meets Expected Growth Exceeds Expected Growth School-wide EVAAS Growth 11/14/2018 • page 25

1 6 5 4 3 2 1 6 5 4 3 2 7 8 Ratings Key Note on Ratings Teachers Demonstrate Leadership Establish Environment Know Content Facilitate Learning Reflect on Practice Contribute to Academic Success Teachers Principals 1 6 5 4 3 2 7 8 Strategic Leadership Instructional Leadership Cultural Leadership Human Resource Leadership Managerial Leadership External Development Leadership Micro- political Leadership Academic Achievement Leadership Key Note on Ratings Every educator is evaluated every year Each standard and rating stands on its own (1 out of 6, not 1/6) Ratings are used to create professional development plans each year Ratings are used to determine status

Effectiveness Status Using multiple measures to gauge overall effectiveness of educators

What is the difference between Ratings and Status? 11/14/2018 • page 28

Status Ratings Status Teachers 6 separate ratings to help teachers grow each year Principals 8 separate ratings to help principals grow each year A single overall status that is determined once a principal or teacher has three years of growth data to populate 6 or 8 Categories for Status In Need of Improvement Effective Highly Effective 11/14/2018 • page 29

Status and Standard 6 & 8 An educator receives an effectiveness status only once she has 3 years of data on Standard 6 or 8 A 3-year rolling average of growth data from standard 6 or 8 is used as part of determining overall status 11/14/2018 • page 30

6 6 6 3-Year Rolling Average 1.9 + -2.5 + 1.2 3 = .2 Rating from 2012 - 2013 Rating from 2013 - 2014 Rating from 2014 - 2015 1.9 + -2.5 + 1.2 Standard 6 Standard 6 Standard 6 3 Contribute to Academic Success Contribute to Academic Success = .2 Met Expected Growth 3- year average rating on standard 6 for determining status 1.9 Met Expected Growth -2.5 Did not meet Expected Growth 1.2 Met Expected Growth Note: A similar methodology applies to principals as well. Note: The values above represent values from the MRM model in EVAAS. 11/14/2018 • page 31

Three Years of Data Any three years of data attributable to a teacher or principal will be combined and used: Any grades Any subjects Any schools Any districts The three years of data do not start until they are specific to that teacher and his or her students 11/14/2018 • page 32

Status So once a educator has a three-year average rating for Standard 6 or 8, how is status determined? 11/14/2018 • page 33

In Need of Improvement Effective Highly Effective Status The Three Status Categories are In Need of Improvement Effective Highly Effective 11/14/2018 • page 34

/ 1 5 4 3 2 ) ) 3 Teacher Status Standards 1-5 In the year In Need of Improvement Effective Highly Effective Standards 1-5 In the year Any rating lower than proficient Proficient or Higher on Standards 1-5 Accomplished or Higher on Standards 1-5 1 5 4 3 2 Demonstrate Leadership Establish Environment Know Content Facilitate Learning Reflect on Practice And/Or And And Standard 6 Three-year rolling average Does Not Meet Expected Growth Meets or Exceeds Expected Growth Exceeds Expected Growth ) / ) 3 2 years ago 6 1 year ago + 6 + This year 6

/ ) 3 Principal Status Standards 1-7 In the year 1 6 5 4 3 2 7 In Need of Improvement Effective Highly Effective Standards 1-7 In the year Any rating lower than proficient Proficient or Higher on Standards 1-7 Accomplished or Higher on Standards 1-7 1 6 5 4 3 2 7 Strategic Leadership Instructional Leadership Cultural Leadership Human Resource Leadership Managerial Leadership External Development Leadership Micro- political Leadership And/Or And And Standard 8 Three-year rolling average Does Not Meet Expected Growth Meets or Exceeds Expected Growth Exceeds Expected Growth 8 2 years ago 1 year ago This year + / 3 )

What Will Teachers See? Ratings on Standards 1 – 5 of the Educator Evaluation System (as recorded in online tool) Standard 6 rating (current year and 2 prior years) Three-year rolling average of student growth values and accompanying Standard 6 rating (for Status determination) Overall Effectiveness Status 11/14/2018 • page 37

Detail on the Sixth Standard Rating

Student Survey Giving students a voice on classroom effectiveness

The Role of Student Surveys Student surveys can play a role in professional development for teachers, teacher evaluation, and school improvement These surveys also capture specific teacher behaviors that lead to student success, which can be useful formative information for the professional development process 11/14/2018 • page 43

Evaluation Standards and Surveys NC’s Five Teacher Evaluation Standards. Teachers: Survey Constructs 1. Demonstrate leadership. 1. Control (Balanced classroom management) 2. Establish a respectful environment 2. Care 3. Know the content they teach 3. Clarify 4. Facilitate learning for their students 4. Challenge (effort & rigor) 5. Captivate 6. Confer 5. Reflect on their practice 7. Consolidate 11/14/2018 • page 44

Learning Gains and Surveys 25th Percentile 75th 1. CARE: My teacher in this class makes me feel that s/he really cares about me 40 73 2. CONTROL: Our class stays busy and doesn’t waste time. 36 69 3. CLARIFY: My teacher explains difficult things clearly. 50 79 4. CHALLENGE: My teacher wants me to explain my answers – why I think what I think. 59 83 5. CAPTIVATE: My teacher makes learning enjoyable. 33 72 6. CONFER: My teacher wants us to share our thoughts. 47 7. CONSOLIDATE: My teacher takes the time to summarize what we learn each day. 38 67 11/14/2018 • page 45

GPA and Surveys 11/14/2018 • page 46

NC’s Pilot Program By the Numbers: 147,000 students 3,300 teachers 900 schools 47 local education agencies 11/14/2018 • page 47

Survey Methodology Focus on teachers in subjects/grades with state assessments (to allow for correlation with student growth) All grades Survey stratified on school and LEA: Size Wealth Prior student growth patterns Geographic location 11/14/2018 • page 48

Survey Administration Paper and online versions (80% online/20% paper) Long and short versions (50% long/50% short) Administration window of April 16 to May 11 (with some exceptions) Website with administration details accessible to all at http://www.tripodproject.org/ncdpi/ Results available online in late summer 2012 11/14/2018 • page 49

Next Steps Research and Recommendation to SBE (October 2012) Ongoing statistical analysis of results 2. Release of Results and Training (Fall 2012) Memo to superintendents and district coordinators (week of 8/27) Users verify accounts (week of 8/27) Release of results (week of 9/3) In-person regional trainings (early October) 11/14/2018 • page 50

Measures of Student Learning Measuring student growth in all areas of the curriculum

Measures of Student Learning Measures of Student Learning are being designed for non-tested subjects for district use to populated Standard 6 11/14/2018 • page 52

Guiding Principles NC’s experienced teachers know their students and their content NC teachers are best-qualified to provide input on meaningful assessment of currently non-tested grades and subjects Valid measures of what students know and are able to do will likely exceed traditional multiple- choice assessment 11/14/2018 • page 53

What MSLs Are Measures of what students know and are able to do after completing a course or grade Tightly linked to the instruction that a teacher delivers One part of how North Carolina will evaluate the effectiveness of its teachers Similar to the common summative assessments that many districts already have in place 11/14/2018 • page 54

What MSLs Are Not Multiple-choice standardized exams for all areas of the Standard Course of Study Assessments that need to be delivered with the same level of security as EOCs and EOGs Designed without teacher input The only source of data used to make decisions about a teacher’s effectiveness Part of the school accountability model 11/14/2018 • page 55

Courses Focused on Performance Locally Developed Courses Four Buckets of Assessments for Growth A B C D EOCs,EOGs and VoCATS MSLs Courses Focused on Performance Locally Developed Courses Assessment Common Across Districts Growth using EVAAS Guidance from DPI with local implementation options Growth determined by evaluator

1 2 3 4 Three Phase Process October 2011 Teachers design item specifications for all currently non-tested grades and subjects Summer 2012 Teachers review items generated by TOPS at NC State University Fall 2012 Teachers review rubrics and guidance on MSL Administration 1 2 3 4 11/14/2018 • page 57

1 2 3 4 Three Phase Process October 2011 Teachers design item specifications for all currently non-tested grades and subjects Summer 2012 Teachers review items generated by TOPS at NC State University Fall 2012 Teachers review rubrics and guidance on MSL Administration 1 2 3 4 11/14/2018 • page 58

Phase I: Create Groups Developed list of all non-tested courses and grades in the Common Core State Standards and NC Essential Standards Grouped courses and grades together into like-content groups Groups range from Extended Content Standards to Chemistry to Elementary Theatre Arts to Social Studies Electives

Phase I: Select Members Designed an online application system for interested educators to apply to join one of the design groups Advertised for the design groups Selected educators from over 1,500 applications Responded to LEA concerns and notified teachers of final selection decisions

Phase I: Ensure Representation 101 Local Education Agencies 10 Charter Schools Department of Juvenile Justice and Delinquency Prevention University of North Carolina North Carolina Virtual Public School

Phase I: Provide Training The Measures of Student Learning design process and how the Measures fit into the State’s educator effectiveness work (DPI Leadership and RttT Project Management) Assessment design, including potential item types, reliability, and validity (Assessment Design and Development) Overview of the Common Core and NC Essential Standards (Curriculum & Instruction)

What does meaningful assessment in your content area look like? Phase I: Gather Feedback Through three feedback protocols, teachers provided answers to the following critical question: What does meaningful assessment in your content area look like?

Teacher Thoughts and Concerns Appreciation, pride, and even joy that their content areas are now being valued and that the State is recognizing that they impact the learning of their students Worry about finding a valid way to measure student learning in an art or PE class that meets once a month and one that meets every day Value the input of teachers into the process Doubt over the sustainability of these Measures of Student Learning after Race to the Top ends Worry about a “test-heavy” environment for students, especially young children

1 2 3 4 Three Phase Process October 2011 Teachers design item specifications for all currently non-tested grades and subjects Summer 2012 Teachers review items generated by TOPS at NC State University Fall 2012 Teachers review rubrics and guidance on MSL Administration 1 2 3 4 11/14/2018 • page 65

Phase II: Develop Framework Validity framework and psychometric plan detail: Theory of Action Score Generation Propositions and Claims for Use Assessment Development Process Administration Scoring Item Calibration, Equating, and Scaling Data Collection Processes Involvement of NC Technical Advisors 11/14/2018 • page 66

Phase II: Create Blueprints 11/14/2018 • page 67

Phase II: Generate Items Staff members at CUACS at NC State University are writing items to the specifications provided by teachers 11/14/2018 • page 68

Phase II: Review Items Teachers return on rolling schedule to review items First design group members returned at end of July 2012 High School Science design group members High School World History, Civics and Economics, and American History II/II design group members English I, English III and IV design group members High School Mathematics design group members OCS design group members 11/14/2018 • page 69

Administration – Fall Semester Earth/Environmental Science Physics Chemistry Physical Science English Language Arts I English Language Arts III English Language Arts IV Pre-Calculus Advanced Functions and Modeling Geometry* Algebra II/Integrated Math III World History Civics and Economics U.S. History (2003 standards) American History I American History II OCS English Language Arts I OCS English Language Arts III OCS English Language Arts IV OCS Financial Management OCS Applied Science OCS Introductory Math 11/14/2018 • page 70

Administration – Spring Semester Grade Four Science Grade Six Science Grade Seven Science Grade Four Social Studies Grade Five Social Studies Grade Six Social Studies Grade Seven Social Studies Grade Eight Social Studies 11/14/2018 • page 71

Coming in 2013 - 2014 Literacy Assessment (running record) for grades K-3 Pre- and Post-Assessments for Healthful Living Analysis of Student Work Protocols for the Arts, World Languages, and District Electives

MSL Implementation Guide Currently in DRAFT form Balances between LEA flexibility in implementation and the need to collect secure data to send to EVAAS Will be revised after feedback from testing coordinators and superintendents (updated version posted by September 15) Will also be revised after Fall 2012 administration prior to launch of Spring 2013 administration Outlines decisions that LEAs need to make about implementation of the MSLs

Part I - Context Provides a summary of the educator evaluation system Outlines the purpose of the MSLs Inclusion in the guide does not imply that testing coordinators are responsible for training on the evaluation system or Standard 6 Acknowledges that teachers are also an audience for the guide and need to understand how the MSLs connect with educator effectiveness Separate guide on how assessment data are used to measure growth is forthcoming

Part II - Timelines Timeline of administration for the MSLs Testing window Up to LEA discretion Data from Fall 2012 administration due February 15, 2013 Data from Spring 2013 administration due June 28, 2013 No retesting (unless misadministration declared) Administration should not extend testing window MSLs designed to be administered during normal class period or during exam week

Part II – Length and Population Length of the Fall 2012 MSLs 90 minutes MSLs broken into two 45-minute sections to allow for administration in non-block schedules Testing population All 4 – 12 students (with or without accommodations) LEP students who meet eligibility criteria Not required for students being instructed on the Extended Content Standards

Part II – Online Administration LEAs may administer through existing online assessment programs as long as: The items are uploaded through a method that preserves the integrity of any images The program can export data in the required form (required file format will be released soon) Plans are in place for security of the MSLs The NCDPI cannot support online administration Note: The same form of the MSL will be used in Fall 2012 and Spring 2013. That form will then be released, and a new form will be available for use in 2013 – 2014

Part II – Paper Administration Paper and pencil administration NCDPI will provide PDF files of all MSLs (~November 8) One PDF for regular administration One PDF that includes the common large-print and one item per page modifications LEAs are responsible for printing Elimination of school-level expenses for printing and Scantron sheets for teacher-designed final exams Answer sheets available for purchase from vendor with whom the NCDPI will establish a sole-source relationship NCDPI will approve Race to the Top Detailed Scope of Work amendments that move funds to pay for administration of the MSLs

Part II – Materials Needed Materials for administration MSL Implementation Guide (electronic or paper version) MSL Test Books MSL Answer Sheets Blank paper Number 2 pencils Calculators for some science and math MSLs (students may use their own as long as they are cleared by a teacher prior to testing) Timing device

Part II – Handling of Materials Ensure that access to the MSLs is limited by storing in locked location Take steps to prohibit reproduction of any part of the MSLs Distribute only immediately before administration Testing Code of Ethics applies Collect all materials and destroy any test books that students have written in Store clean test books and unused answer sheets from Fall 2012 to use in Spring 2013

Part II – Secure Environment The NCDPI strongly recommends one of the following policies regarding test administrators and proctors: If the test administrator is the teacher of record for the grade or course, a proctor should be present during the MSL administration Another teacher (not the teacher of record) serve as the test administrator Other methods as determined by LEAs; principals ultimately responsible for security LEAs are not required to use one of the above policies, but should consider security in an environment where student test results play a role in the teacher’s evaluation

Part II – Irregularities LEAs should determine what constitutes an irregularity or misadministration MSL testing irregularities should be investigated and handled at the local level; do not enter into OTISS When a misadministration is declared, the MSL should be administered again after no fewer than five days from the original administration date

Part II – Teacher Scoring Many of the MSLs include one or more performance-based tasks The NCDPI strongly recommends one of the following policies regarding scoring of these items: Two teachers with the appropriate content knowledge review and grade the performance-based items (one may be the teacher of record One teacher with the appropriate content knowledge reviews and grades the performance-based items (should not be the teacher of record) Partnerships between neighboring LEAs may be valuable

Part II – Scoring The teacher(s) who scored the MSL performance items bubble(s) in the number of points awarded on the student answer sheet Testing staff scores answer sheets in Winscan, which allows for simultaneous capture of points awarded for performance items, scoring of multiple-choice items, and generation of a raw score Percent correct (or curved version) can be used in student grade as a final exam

Part III Each LEA should develop an implementation plan for the MSLs Plans should include: Training for teachers and school administrators on the MSLs (in conjunction with Human Resources staff) How/if MSLs will be used for student accountability (i.e. as final exam grades) How/if parents and guardians will be notified of the MSLs Testing window

Part III Training for teachers on how to score the performance-based items (NCDPI module released in early Fall 2012) Administration mode and security Uniform procedures for administration Procedures for the distribution, collection, storage, destruction, or recycling of MSL materials Roles and responsibilities for LEA and school-level staff members

Part IV Sample scripts to use for paper and pencil administration

Local Plans Each district will need to develop a plan on educator effectiveness, including the implementation of the MSLs Key members of district planning teams: Human Resources Director Curriculum and Instruction Director Accountability/Testing Director These members should be in attendance at September 20 meeting in Greensboro

New Resources “Rapid Response” Email Address: educatoreffectiveness@dpi.nc.gov New Educator Effectiveness Website http://www.ncpublicschools.org/educatoreffect/