1. NORTH CAROLINA Educator Effectiveness Update to PD Leads
August 27, 2012

2. Overview for Today Rationale Why we are doing this?
Status Determining educator effectiveness status
MET Research Key ideas from recent research
Surveys Potential element in educator effectiveness model
Standard 6 & 8 How we’ll include student growth in educator evaluations
6. MSLs Measuring growth in previously non-tested areas
11/14/2018 • page 2

3. Rationale
Why this is the right time and the right work

4. Overview The central focus of READY is improving student learning ...
by enabling and ensuring great teaching.
11/14/2018 • page 4

5. What is our goal? Before Teaching and Leading
Develop effective teachers and leaders in preparation programs
Great Teachers and Leaders An effective teacher in every classroom and leader in every school
Student Readiness
Achievement and growth for all students
During Teaching and Leading
Use meaningful evaluation and professional development to increase effectiveness of teachers and leaders
11/14/2018 • page 5

6. Measures of Effective Teaching
Using research to build a strong educator effectiveness model

7. What is our goal?
Big Question: What is the best approach to Educator Evaluation and how do we get NC there?
11/14/2018 • page 7

8. Measures of Effective Teaching
11/14/2018 • page 8

9. Measures of Effective Teaching
This research suggests:
Multiple measures are important.
Including student growth improves objectivity of evaluation.
11/14/2018 • page 9

10. Months of Learning Gained or Lost
Re-creation of chart from Gathering Feedback For Teaching,
Observation + Other Measures
Months of Learning Gained or Lost
State Math
State ELA
Observation Tool
Top 25%
+1.2
+.2
Bottom 25%
-1.4
-.4
Observation Tool + Student Survey
Top 25%
+2.8
+.7
Bottom 25% -2
-.9
Observation Tool + Student Survey
+ Growth (Value-Add)
Top 25%
+4.5
+1.2
Bottom 25%
-3.1
-1.3

11. Standards 6 and 8
Incorporating student growth into the NC Educator Evaluation System

12.   What We Have From the MET… Observation Tools
Some Assessments to Measure Growth
From the MET…
Evaluation Tools 
End of Grade End of Course VoCATs
Observation Tools
Assessments to Measure Growth
Student Survey 
Standards 1-7
Standard 8
End of Grade End of Course VoCATs
Standards 1-5
Standard 6
Exploring
Pilot in 47 LEAs in

13. Observation + Other Measures
What do we need?
Standard 6 and 8 We need a state-adopted growth model and a fair 6 & 8 rating strategy
Status We need an overall method to determine educator effectiveness status
Measures of Student Learning (MSLs) For those grades and subjects that are currently non- tested, we need ways to measure growth
11/14/2018 • page 13

14. 1 2 3 4 5 6 1 2 3 4 5 6 7 8 Standards 6 & 8 – The Basics Teachers
Contribute to Academic Success
Demonstrate Leadership
Establish Environment
Know Content
Facilitate Learning
Reflect on Practice
Principals (and other Administrators) 1 2 3 4 5 6 7 8
Human Resource Leadership
Strategic Leadership
Instructional Leadership
Cultural Leadership
Managerial Leadership
External Development Leadership
Micro- political Leadership
Academic Achievement Leadership
11/14/2018 • page 14

15. Standard 6 and 8 are measures of Growth
Growth Model
Teachers 6
Standard 6 and 8 are measures of Growth
Contribute to Academic Success
Principals 8
Academic Achievement Leadership
Academic Achievement Leadership
11/14/2018 • page 15

16. Growth Model
Teachers 6
We will use Educator Value-Added Assessment System EVAAS for standards 6 & 8 when possible
Contribute to Academic Success
Principals 8
Academic Achievement Leadership
Academic Achievement Leadership
11/14/2018 • page 16

17. 6 8 Growth Model Teachers Principals How do Value-Added models work?
They measure growth by predicting how well a student will do on an assessment.
How do they predict how well the student will do?
They look at previous test scores and estimate how well the student should do at the end of the year. Every student must grow based on where they start. 6
Contribute to Academic Success
Principals 8
Academic Achievement Leadership
Academic Achievement Leadership
11/14/2018 • page 17

18. How will the ratings on Standards 6 & 8 work?
Teachers 6
Contribute to Academic Success
How will the ratings on Standards 6 & 8 work?
Principals 8
Academic Achievement Leadership
Academic Achievement Leadership
11/14/2018 • page 18

19. 8 1 6 5 4 3 2 7 Principal Rating Categories Principals
Instructional Leadership
Cultural Leadership
Human Resource Leadership
Managerial Leadership
External Development Leadership
Micro- political Leadership
Academic Achievement Leadership
Strategic Leadership
5 Rating Categories
3 Rating Categories
Not Demonstrated
Developing
Proficient
Accomplished
Distinguished
Does Not Meet Expected Growth
Meets Expected Growth
Exceeds Expected Growth
11/14/2018 • page 19

20. 1 2 3 4 5 6 Teacher Ratings Categories Teachers 5 Rating Categories
Demonstrate Leadership
Establish Environment
Know Content
Facilitate Learning
Contribute to Academic Success
Reflect on Practice
5 Rating Categories
3 Rating Categories
Not Demonstrated
Developing
Proficient
Accomplished
Distinguished
Does Not Meet Expected Growth
Meets Expected Growth
Exceeds Expected Growth
11/14/2018 • page 20

21. 1 2 3 4 5 6 Ratings Why the difference? Teachers 5 Rating Categories
Demonstrate Leadership
Establish Environment
Know Content
Facilitate Learning
Contribute to Academic Success
Reflect on Practice
5 Rating Categories
3 Ratings Categories
Why the difference?
Identifying only three rating categories on standard 6 & 8 improves certainty of categorization.

22. Why is school-wide EVAAS growth included?
Teacher Ratings in Rationale - MET Research - Standard 6 & 8 - Status - Support ▲ 6
Yearly Rating
Does not Meet Expected Growth
Meets Expected Growth
Exceeds Expected Growth
Teacher EVAAS Growth
School-wide
EVAAS Growth
70% %
Weighted Average
Why is school-wide EVAAS growth included?
To encourage collaboration and collective ownership of overall outcomes.
Note: In , teachers without individual EVAAS growth will have school-wide growth for Standard 6.

23. Teacher Ratings in 6
Yearly Rating
Does not Meet Expected Growth
Meets Expected Growth
Exceeds Expected Growth
Teacher-level EVAAS reports available on September 11 (estimated)
Teacher sixth standard ratings available at end of September
Sixth standard ratings available in two locations:
Rating only will be back-populated into the McREL tool and available to see on the 2011 – summary rating forms
Rating and component data available in EVAAS

24. Possible additional element
Teacher Ratings in
2012 – 2013 is the first year of data for all teachers and school administrators who have their own data
Possible additional element 6
Yearly Rating
Does not Expected Growth
Meets Expected Growth
Exceeds Expected Growth
Teacher EVAAS Growth
School-wide
EVAAS Growth
Student Surveys
(?)
Weighted Average
11/14/2018 • page 24

25. Principal Ratings
Standards 8 rating will be determined using school-wide EVAAS growth 8
Yearly Rating
Does not Meet Expectations
Meets Expected Growth
Exceeds Expected Growth
School-wide
EVAAS Growth
11/14/2018 • page 25

26. 1 6 5 4 3 2 1 6 5 4 3 2 7 8 Ratings Key Note on Ratings Teachers
Demonstrate Leadership
Establish Environment
Know Content
Facilitate Learning
Reflect on Practice
Contribute to Academic Success
Teachers
Principals 1 6 5 4 3 2 7 8
Strategic Leadership
Instructional Leadership
Cultural Leadership
Human Resource Leadership
Managerial Leadership
External Development Leadership
Micro- political Leadership
Academic Achievement Leadership
Key Note on Ratings
Every educator is evaluated every year
Each standard and rating stands on its own (1 out of 6, not 1/6)
Ratings are used to create professional development plans each year
Ratings are used to determine status

27. Effectiveness Status
Using multiple measures to gauge overall effectiveness of educators

28. What is the difference between Ratings and Status?
11/14/2018 • page 28

29. Status
Ratings
Status
Teachers 6 separate ratings to help teachers grow each year
Principals 8 separate ratings to help principals grow each year
A single overall status that is determined once a principal or teacher has three years of growth data to populate 6 or 8
Categories for Status
In Need of Improvement
Effective
Highly Effective
11/14/2018 • page 29

30. Status and Standard 6 & 8
An educator receives an effectiveness status only once she has 3 years of data on Standard 6 or 8
A 3-year rolling average of growth data from standard 6 or 8 is used as part of determining overall status
11/14/2018 • page 30

31. 6 6 6 3-Year Rolling Average 1.9 + -2.5 + 1.2 3 = .2
Rating from
Rating from
Rating from
Standard 6
Standard 6
Standard 6 3
Contribute to Academic Success
Contribute to Academic Success
= .2
Met Expected Growth 3- year average rating on standard 6 for determining status
1.9 Met Expected Growth
-2.5 Did not meet Expected Growth
1.2
Met Expected Growth
Note: A similar methodology applies to principals as well.
Note: The values above represent values from the MRM model in EVAAS.
11/14/2018 • page 31

32. Three Years of Data
Any three years of data attributable to a teacher or principal will be combined and used:
Any grades
Any subjects
Any schools
Any districts
The three years of data do not start until they are specific to that teacher and his or her students
11/14/2018 • page 32

33. Status
So once a educator has a three-year average rating for Standard 6 or 8, how is status determined?
11/14/2018 • page 33

34. In Need of Improvement Effective Highly Effective Status
The Three Status Categories are
In Need of Improvement
Effective
Highly Effective
11/14/2018 • page 34

35. / 1 5 4 3 2 ) ) 3 Teacher Status Standards 1-5 In the year
In Need of Improvement
Effective
Highly Effective
Standards 1-5 In the year
Any rating lower than proficient
Proficient or Higher on Standards 1-5
Accomplished or Higher on Standards 1-5 1 5 4 3 2
Demonstrate Leadership
Establish Environment
Know Content
Facilitate Learning
Reflect on Practice
And/Or
And
And
Standard 6 Three-year rolling average
Does Not Meet Expected Growth
Meets or Exceeds Expected Growth
Exceeds Expected Growth ) / ) 3
2 years ago 6
1 year ago + 6 +
This year 6

36. / ) 3 Principal Status Standards 1-7 In the year 1 6 5 4 3 2 7
In Need of Improvement
Effective
Highly Effective
Standards 1-7 In the year
Any rating lower than proficient
Proficient or Higher on Standards 1-7
Accomplished or Higher on Standards 1-7 1 6 5 4 3 2 7
Strategic Leadership
Instructional Leadership
Cultural Leadership
Human Resource Leadership
Managerial Leadership
External Development Leadership
Micro- political Leadership
And/Or
And
And
Standard 8 Three-year rolling average
Does Not Meet Expected Growth
Meets or Exceeds Expected Growth
Exceeds Expected Growth 8
2 years ago
1 year ago
This year + / 3 )

37. What Will Teachers See?
Ratings on Standards 1 – 5 of the Educator Evaluation System (as recorded in online tool)
Standard 6 rating (current year and 2 prior years)
Three-year rolling average of student growth values and accompanying Standard 6 rating (for Status determination)
Overall Effectiveness Status
11/14/2018 • page 37

41. Detail on the Sixth Standard Rating

42. Student Survey
Giving students a voice on classroom effectiveness

43. The Role of Student Surveys
Student surveys can play a role in professional development for teachers, teacher evaluation, and school improvement These surveys also capture specific teacher behaviors that lead to student success, which can be useful formative information for the professional development process
11/14/2018 • page 43

44. Evaluation Standards and Surveys
NC’s Five Teacher Evaluation Standards.
Teachers:
Survey Constructs
1. Demonstrate leadership.
1. Control (Balanced classroom management)
2. Establish a respectful environment
2. Care
3. Know the content they teach
3. Clarify
4. Facilitate learning for their students
4. Challenge (effort & rigor)
5. Captivate
6. Confer
5. Reflect on their practice
7. Consolidate
11/14/2018 • page 44

45. Learning Gains and Surveys
25th
Percentile
75th
1. CARE: My teacher in this class makes me feel that s/he really cares about me 40 73
2. CONTROL: Our class stays busy and doesn’t waste time. 36 69
3. CLARIFY: My teacher explains difficult things clearly. 50 79
4. CHALLENGE: My teacher wants me to explain my answers – why I think what I think. 59 83
5. CAPTIVATE: My teacher makes learning enjoyable. 33 72
6. CONFER: My teacher wants us to share our thoughts. 47
7. CONSOLIDATE: My teacher takes the time to summarize what we learn each day. 38 67
11/14/2018 • page 45

46. GPA and Surveys
11/14/2018 • page 46

47. NC’s Pilot Program
By the Numbers: 147,000 students ,300 teachers 900 schools local education agencies
11/14/2018 • page 47

48. Survey Methodology
Focus on teachers in subjects/grades with state assessments (to allow for correlation with student growth)
All grades
Survey stratified on school and LEA:
Size
Wealth
Prior student growth patterns
Geographic location
11/14/2018 • page 48

49. Survey Administration
Paper and online versions (80% online/20% paper)
Long and short versions (50% long/50% short)
Administration window of April 16 to May 11 (with some exceptions)
Website with administration details accessible to all at
Results available online in late summer 2012
11/14/2018 • page 49

50. Next Steps Research and Recommendation to SBE (October 2012)
Ongoing statistical analysis of results
2. Release of Results and Training (Fall 2012)
Memo to superintendents and district coordinators (week of 8/27)
Users verify accounts (week of 8/27)
Release of results (week of 9/3)
In-person regional trainings (early October)
11/14/2018 • page 50

51. Measures of Student Learning
Measuring student growth in all areas of the curriculum

52. Measures of Student Learning
Measures of Student Learning are being designed for non-tested subjects for district use to populated Standard 6
11/14/2018 • page 52

53. Guiding Principles
NC’s experienced teachers know their students and their content
NC teachers are best-qualified to provide input on meaningful assessment of currently non-tested grades and subjects
Valid measures of what students know and are able to do will likely exceed traditional multiple- choice assessment
11/14/2018 • page 53

54. What MSLs Are
Measures of what students know and are able to do after completing a course or grade
Tightly linked to the instruction that a teacher delivers
One part of how North Carolina will evaluate the effectiveness of its teachers
Similar to the common summative assessments that many districts already have in place
11/14/2018 • page 54

55. What MSLs Are Not
Multiple-choice standardized exams for all areas of the Standard Course of Study
Assessments that need to be delivered with the same level of security as EOCs and EOGs
Designed without teacher input
The only source of data used to make decisions about a teacher’s effectiveness
Part of the school accountability model
11/14/2018 • page 55

56. Courses Focused on Performance Locally Developed Courses
Four Buckets of Assessments for Growth A B C D
EOCs,EOGs and VoCATS
MSLs
Courses Focused on Performance
Locally Developed Courses
Assessment Common Across Districts
Growth using EVAAS
Guidance from DPI with local implementation options
Growth determined by evaluator

57. 1 2 3 4 Three Phase Process October 2011
Teachers design item specifications for all currently non-tested grades and subjects
Summer 2012
Teachers review items generated by TOPS at NC State University
Fall 2012
Teachers review rubrics and guidance on MSL Administration 1 2 3 4
11/14/2018 • page 57

58. 1 2 3 4 Three Phase Process October 2011
Teachers design item specifications for all currently non-tested grades and subjects
Summer 2012
Teachers review items generated by TOPS at NC State University
Fall 2012
Teachers review rubrics and guidance on MSL Administration 1 2 3 4
11/14/2018 • page 58

59. Phase I: Create Groups
Developed list of all non-tested courses and grades in the Common Core State Standards and NC Essential Standards
Grouped courses and grades together into like-content groups
Groups range from Extended Content Standards to Chemistry to Elementary Theatre Arts to Social Studies Electives

60. Phase I: Select Members
Designed an online application system for interested educators to apply to join one of the design groups
Advertised for the design groups
Selected educators from over 1,500 applications
Responded to LEA concerns and notified teachers of final selection decisions

61. Phase I: Ensure Representation
101 Local Education Agencies
10 Charter Schools
Department of Juvenile Justice and Delinquency Prevention
University of North Carolina
North Carolina Virtual Public School

62. Phase I: Provide Training
The Measures of Student Learning design process and how the Measures fit into the State’s educator effectiveness work (DPI Leadership and RttT Project Management)
Assessment design, including potential item types, reliability, and validity (Assessment Design and Development)
Overview of the Common Core and NC Essential Standards (Curriculum & Instruction)

63. What does meaningful assessment in your content area look like?
Phase I: Gather Feedback
Through three feedback protocols, teachers provided answers to the following critical question:
What does meaningful assessment in your content area look like?

64. Teacher Thoughts and Concerns
Appreciation, pride, and even joy that their content areas are now being valued and that the State is recognizing that they impact the learning of their students
Worry about finding a valid way to measure student learning in an art or PE class that meets once a month and one that meets every day
Value the input of teachers into the process
Doubt over the sustainability of these Measures of Student Learning after Race to the Top ends
Worry about a “test-heavy” environment for students, especially young children

65. 1 2 3 4 Three Phase Process October 2011
Teachers design item specifications for all currently non-tested grades and subjects
Summer 2012
Teachers review items generated by TOPS at NC State University
Fall 2012
Teachers review rubrics and guidance on MSL Administration 1 2 3 4
11/14/2018 • page 65

66. Phase II: Develop Framework
Validity framework and psychometric plan detail:
Theory of Action
Score Generation
Propositions and Claims for Use
Assessment Development Process
Administration
Scoring
Item Calibration, Equating, and Scaling
Data Collection Processes
Involvement of NC Technical Advisors
11/14/2018 • page 66

67. Phase II: Create Blueprints
11/14/2018 • page 67

68. Phase II: Generate Items
Staff members at CUACS at NC State University are writing items to the specifications provided by teachers
11/14/2018 • page 68

69. Phase II: Review Items
Teachers return on rolling schedule to review items
First design group members returned at end of July 2012
High School Science design group members
High School World History, Civics and Economics, and American History II/II design group members
English I, English III and IV design group members
High School Mathematics design group members
OCS design group members
11/14/2018 • page 69

70. Administration – Fall Semester
Earth/Environmental Science
Physics
Chemistry
Physical Science
English Language Arts I
English Language Arts III
English Language Arts IV
Pre-Calculus
Advanced Functions and Modeling
Geometry*
Algebra II/Integrated Math III
World History
Civics and Economics
U.S. History (2003 standards)
American History I
American History II
OCS English Language Arts I
OCS English Language Arts III
OCS English Language Arts IV
OCS Financial Management
OCS Applied Science
OCS Introductory Math
11/14/2018 • page 70

71. Administration – Spring Semester
Grade Four Science
Grade Six Science
Grade Seven Science
Grade Four Social Studies
Grade Five Social Studies
Grade Six Social Studies
Grade Seven Social Studies
Grade Eight Social Studies
11/14/2018 • page 71

72. Coming in
Literacy Assessment (running record) for grades K-3 Pre- and Post-Assessments for Healthful Living Analysis of Student Work Protocols for the Arts, World Languages, and District Electives

73. MSL Implementation Guide
Currently in DRAFT form
Balances between LEA flexibility in implementation and the need to collect secure data to send to EVAAS
Will be revised after feedback from testing coordinators and superintendents (updated version posted by September 15)
Will also be revised after Fall 2012 administration prior to launch of Spring 2013 administration
Outlines decisions that LEAs need to make about implementation of the MSLs

74. Part I - Context Provides a summary of the educator evaluation system
Outlines the purpose of the MSLs
Inclusion in the guide does not imply that testing coordinators are responsible for training on the evaluation system or Standard 6
Acknowledges that teachers are also an audience for the guide and need to understand how the MSLs connect with educator effectiveness
Separate guide on how assessment data are used to measure growth is forthcoming

75. Part II - Timelines Timeline of administration for the MSLs
Testing window
Up to LEA discretion
Data from Fall 2012 administration due February 15, 2013
Data from Spring 2013 administration due June 28, 2013
No retesting (unless misadministration declared)
Administration should not extend testing window
MSLs designed to be administered during normal class period or during exam week

76. Part II – Length and Population
Length of the Fall 2012 MSLs
90 minutes
MSLs broken into two 45-minute sections to allow for administration in non-block schedules
Testing population
All 4 – 12 students (with or without accommodations)
LEP students who meet eligibility criteria
Not required for students being instructed on the Extended Content Standards

77. Part II – Online Administration
LEAs may administer through existing online assessment programs as long as:
The items are uploaded through a method that preserves the integrity of any images
The program can export data in the required form (required file format will be released soon)
Plans are in place for security of the MSLs
The NCDPI cannot support online administration
Note: The same form of the MSL will be used in Fall 2012 and Spring That form will then be released, and a new form will be available for use in 2013 – 2014

78. Part II – Paper Administration
Paper and pencil administration
NCDPI will provide PDF files of all MSLs (~November 8)
One PDF for regular administration
One PDF that includes the common large-print and one item per page modifications
LEAs are responsible for printing
Elimination of school-level expenses for printing and Scantron sheets for teacher-designed final exams
Answer sheets available for purchase from vendor with whom the NCDPI will establish a sole-source relationship
NCDPI will approve Race to the Top Detailed Scope of Work amendments that move funds to pay for administration of the MSLs

79. Part II – Materials Needed
Materials for administration
MSL Implementation Guide (electronic or paper version)
MSL Test Books
MSL Answer Sheets
Blank paper
Number 2 pencils
Calculators for some science and math MSLs (students may use their own as long as they are cleared by a teacher prior to testing)
Timing device

80. Part II – Handling of Materials
Ensure that access to the MSLs is limited by storing in locked location
Take steps to prohibit reproduction of any part of the MSLs
Distribute only immediately before administration
Testing Code of Ethics applies
Collect all materials and destroy any test books that students have written in
Store clean test books and unused answer sheets from Fall 2012 to use in Spring 2013

81. Part II – Secure Environment
The NCDPI strongly recommends one of the following policies regarding test administrators and proctors:
If the test administrator is the teacher of record for the grade or course, a proctor should be present during the MSL administration
Another teacher (not the teacher of record) serve as the test administrator
Other methods as determined by LEAs; principals ultimately responsible for security
LEAs are not required to use one of the above policies, but should consider security in an environment where student test results play a role in the teacher’s evaluation

82. Part II – Irregularities
LEAs should determine what constitutes an irregularity or misadministration
MSL testing irregularities should be investigated and handled at the local level; do not enter into OTISS
When a misadministration is declared, the MSL should be administered again after no fewer than five days from the original administration date

83. Part II – Teacher Scoring
Many of the MSLs include one or more performance-based tasks
The NCDPI strongly recommends one of the following policies regarding scoring of these items:
Two teachers with the appropriate content knowledge review and grade the performance-based items (one may be the teacher of record
One teacher with the appropriate content knowledge reviews and grades the performance-based items (should not be the teacher of record)
Partnerships between neighboring LEAs may be valuable

84. Part II – Scoring
The teacher(s) who scored the MSL performance items bubble(s) in the number of points awarded on the student answer sheet
Testing staff scores answer sheets in Winscan, which allows for simultaneous capture of points awarded for performance items, scoring of multiple-choice items, and generation of a raw score
Percent correct (or curved version) can be used in student grade as a final exam

87. Part III Each LEA should develop an implementation plan for the MSLs
Plans should include:
Training for teachers and school administrators on the MSLs (in conjunction with Human Resources staff)
How/if MSLs will be used for student accountability (i.e. as final exam grades)
How/if parents and guardians will be notified of the MSLs
Testing window

88. Part III
Training for teachers on how to score the performance-based items (NCDPI module released in early Fall 2012)
Administration mode and security
Uniform procedures for administration
Procedures for the distribution, collection, storage, destruction, or recycling of MSL materials
Roles and responsibilities for LEA and school-level staff members

89. Part IV
Sample scripts to use for paper and pencil administration

90. Local Plans
Each district will need to develop a plan on educator effectiveness, including the implementation of the MSLs
Key members of district planning teams:
Human Resources Director
Curriculum and Instruction Director
Accountability/Testing Director
These members should be in attendance at September 20 meeting in Greensboro

91. New Resources
“Rapid Response” Address: New Educator Effectiveness Website