1. State Consortium on Educator Effectiveness August 16, 2012 Student Learning Objectives and Educator Evaluation

2. Webinar Logistics  Everyone is muted  Use the chat function to make a comment or ask a question  You may chat privately with individuals on your team  If you have problems, you may send Naz Rajput a message via the chat function or an email at nazr@ccsso.org 2

3. Welcome  Janice Poda, CCSSO  Initiative Director Education Workforce 3

4. Presenters  Scott Marion, Associate Director, Center for Assessment 4

5. Presenters: RI Student Learning Objectives Team Laura Kacewicz: Assessment Specialist, Office of Instruction, Assessment, and Curriculum Jessica Brown: Assessment Specialist, Office of Instruction, Assessment, and Curriculum Jessica Delforge: Education Specialist, Office of Educator Quality 5

6. The Issue  States and districts are creating evaluation systems that include academic student performance information  Using Value-Added Models (VAM) and/or Student Growth Percentiles (SGP) with state standardized tests is a significant challenge  How do we incorporate student performance results from non-tested grades and subjects (NTGS) into educator evaluation systems? 6

7. What is comparability?  Educators within the units of analysis are held to similar levels of expectations, at least in some relative sense  For example, it would be a threat to the system if the teachers in grades 4-8 reading and math received noticeably lower ratings than the rest of the teachers (NTSG) in the school 7

8. Is Comparability Important?  At what levels is comparability important?  Within schools? Clearly yes.  Within districts? Probably yes.  Within states? It would be nice, but it might be too high of a bar right now. 8

9. Student Learning Objectives as a Framework  Goal: A comprehensive and thoughtful approach that includes the tested subjects/grades, the “non-tested” content area teachers, and other licensed professionals  “Tested” and “non-tested” subjects and grades can be viewed as special cases of the comprehensive framework 9

10. Creating SLOs  SLOs offer more promise than “drop from the sky” assessments for improving practice, but they have more “moving parts”  We need to focus on three key components, all of which need to be in place for SLOs to work well  The Objectives  The Targets—for both students and teachers  The Assessment(s) 10

11. Use of Student Learning Objectives in the Rhode Island Educator Evaluation Model CCSSO Special SCEE Webinar August 16, 2012

12. Made in Rhode Island Educators Working Groups Advisory Committee Educators representing 23 districts and organizations contributed to the development of the content. Six working groups of local educators designed the major components of the system. The Advisory Committee for Educator Evaluation Systems (ACEES) reviewed the content, along with the Technical Advisory Committee (TAC). RIDE field tested the model in two districts and one charter in 2010-11. 12

13. Rhode Island Model: Multiple Measures Student Learning Professional Practice Professional Responsibilities 13

14. Measures of Student Learning Student Learning Objectives Measure learning against a set of academic standards aligned with school and district goals Measurement tools include purchased assessments, district or regionally produced assessments, common or teacher-created assessments designed by schools or teacher teams Use Rhode Island student content standards or other nationally recognized standards(e.g., CCSS) Growth Model Used for contributing educators in Literacy and Math, grades 3-7 14

15. Gradual Implementation: SY2011-12 Time FrameEdition I: GradualEdition II: Full Summer 2011Training begins for evaluators Fall 2011*Gradual Implementation begins / Beginning-of-Year Conference Collecting feedback Winter 2011/2012*Observations / School visits / Mid Year Conference Early Spring 2012*Observations/ School visitsModel refinement Late Spring 2012*End-of-Year Conference Summative ratings collected V 2 Launch/ Increased communication Summer 2012Training Academies Fall 2012Full Implementation * Indicates ongoing support and training via ISPs and Modules 15

16. Gradual Implementation: SY2011-12  It allowed for more time for training so that evaluators felt adequately prepared to take on this role.  It provided everyone with the opportunity to practice and get hands- on experience with the process.  It provided time to learn and talk about the components and make informed revisions before full implementation. 16

17. Gradual vs. Full Implementation for Teachers Component Edition I Gradual Implementation 2011-2012 Edition II Full Implementation 2012-2013 Evaluation Criteria  Student Learning  Professional Practice  Professional Responsibilities  same Number of Evaluation Conferences  3 evaluation conferences between the teacher and the evaluator (Beginning, Middle, and End-of-Year)  same Classroom Observations  At least 4, including: 1 long (30+ minutes), announced and 3 short (15 + minutes), unannounced  Written feedback required after each observation  Post observation conference required after announced observation  At least 3, including: 1 announced and 2 unannounced  At least 20 minutes each  Written feedback required after each  Pre- and post-observation conferences are optional (local decision) Professional Growth Goals  At least 3 set at the beginning of the year  At least 1 set at the beginning of the year Student Learning Objectives  At least 2-4 (per teacher)  3 performance levels for individual SLOs  5 performance levels for sets of SLOs  At least 2 per teacher (no more than 4)  4 performance levels for both individual and sets of SLOs Rhode Island Growth Model  Not applicable  same Teacher Professional Practice Rubric  Holistic rubric with 21 competencies  Classroom observations and evidence collection required to assess competencies  Observation rubric with 8 competencies  All competencies are 100% observable (additional evidence collection not required) Professional Responsibilities Rubric  Holistic rubric with 10 competencies  Holistic rubric with 8 competencies

18. RI has been gathering educator feedback from the start… 18 Early Adopters Warwick and Jamestown have helped us learn from full implementation. Surveys & Focus Groups help us learn what’s most important to teachers and building administrators. Data from the Field helps us prioritize the refinements based on real experiences in the classrooms and schools. Feedback from District Evaluation Committees gives us another valuable perspective on refinements.

19. …resulting in five key priorities for model refinement 19 By listening to educators in Warwick and Jamestown and gradual implementation districts all year, RIDE has identified five priorities for model refinement that will help make the RI Model stronger – and more practical to use.

20. Priority: Clarify Expectations, Requirements and Timelines 20  What We Heard Teachers who have received information from their building administrators about the RI Model tend to understand it better than those who do not. Administrators and teachers report confusion over which aspects of the model are optional and which are required, as well as which decisions should be made locally and which are made at the state level.

21. Priority: Clarify Expectations, Requirements and Timelines 21 What We Are Doing Clarified expectations around the requirements for SLOs Requiring educators to set at least 2 and no more than 4 SLOs

22. SLO Claims, Challenges & Supports Scott Marion 22

23. Claims to evaluate SLOs 23  We can create claims for SLOs and then consider the challenges and support for these claims  These claims would be part of a theory of action and validity argument  I present just a few examples to illustrate this point, but this should be done in more detail prior to implementing such an approach on a large scale

24. Learning Goal/Objective Claim 24  Claim: Teachers have the knowledge, skills, and attitudes (& ethics) to set meaningful, ambitious, and fair learning goals for students  Challenge: Who will guide, monitor, and/or evaluate the quality of these learning goals?  This adds an extra (or at least different) significant validation requirement beyond test-based approaches  Places principals into the role of instructional leaders and many might not have the skills (but this could be an opportunity if successful)  Opportunity: Teaching quality would likely improve if teachers, working with good leaders, were supported in the way they use data to establish goals for their students.

25. The Challenge of the “Objectives”  We have not seen evidence that teachers and other educators can generate high quality SLOs without significant practice and training  Identifying meaningful learning goals appears to quite difficult  Just like we learned with performance assessment, teachers need deep subject matter knowledge to do this well  We might be able to draw from some familiar work such as Wiggins & McTighe’s Understanding by Design and the assessment specifications being developed by both large scale assessment consortia, but it is still a huge challenge 25

26. Grain Size  Finding the right grain size for the “objectives” is a major challenge. Some things to keep in mind:  There is no perfect grain size, because we are definitely in search of the Goldilocks criterion  The grain size should be inversely related, in part, to the number of SLOs on which the teacher will be evaluated  Larger grain size “objectives” might require multiple assessments, while more specific SLOs could probably be measured with a single performance tasks or other assessment 26

27. Grain Size Question?  Which will provide a more reliable estimate of a teacher’s contribution to student performance:  A single large grain size SLO measured with multiple assessments or  Several, more specific SLOs each measured with one assessment?  Might what we learned about the relative effects of adding tasks or raters on the generalizability (reliability) of performance assessments apply here…? 27

28. Rhode Island Examples: Grade 11, Writing Arguments  This objective statement is too broad: Students will improve their ability to write in response to informational text.  This objective statement is too narrow: Students will improve their ability to include textual evidence in written arguments.  This objective statement is acceptable: Students will improve their ability to analyze informational text and to write arguments informed by their analysis, grounded in germane textual evidence. 28

29. Assessment Claims, Challenges & Opportunities 29

30. Assessment and Analysis Claim 30 Claim: Teachers/schools/districts have assessment and analytic tools and capacity sufficient for judging whether students have reached the intended goals Challenge 1: Are classroom assessment tools capable of validly measuring ambitious goals? Challenge 2: If external assessments are used, would that lead to narrow goals to match the more limited tools (tail wagging the dog)? Challenge 3: Is the analytic capacity available for supporting inferences about teacher contributions to student performance? Opportunity: Could this be a lever for improving the quality of classroom assessment and evaluation tools and processes?

31. Assessments  We need high quality assessments to evaluate the extent to which students have achieved the goals  We do not have time today to talk about the challenges associated with finding assessments to evaluate SLOs, but for now…  Think broadly about “assessment”  Do not let the assessment drive the goal; the assessment should be used to support learning goals  The learning goal and assessment should be things that teachers would use in the classroom as part of good instructional practice 31

32. RI Priority: Focus on Accurate Measures of Student Learning 32  What We Heard in Rhode Island Teachers report that there is value in setting measureable goals for student learning, but more common assessments are needed to do this consistently across schools and districts.

33. Priority: Focus on Accurate Measures of Student Learning 33 What We Are Doing Connecting with other RI RttT initiatives, including the Comprehensive Assessment System project that promotes assessment literacy and the Interim Assessments project, which will provide tests in math, ELA, Science and Social Studies Transitioning from scoring sets of Student Learning Objectives on a 5-point rubric to a 4-point rubric. The resulting 4-point scale translates to a Final Summative Matrix that becomes 4x4 instead of 4x5. The new matrix still highlights the critical importance of student learning as a primary indicator of educator effectiveness.

34. 34 Growth and Starting Levels

35.  Many rules associated with incorporating student performance, require the use of “growth” measures, which are generally defined as the difference between two measures at different time points  There are many reasons why this policy is misguided, most of which are discussed in detail in:  Marion, et al. (2012). Considerations for Analyzing Educators’ Contributions to Student Learning in Non-tested Subjects and Grades with a Focus on Student Learning Objectives. www.nciea.org www.nciea.org  Bottom line is that we need to figure out a way to contextualize students’ varying starting points… 35

36. A “Rough Conditioning” Approach for SLOs  Using prior performance information (e.g., last year) or some early assessments in the current year, we can group students into 3-4 “performance” groups  SLO targets would then be differentiated according to the students’ starting group.  At least two ways to differentiate targets:  Different levels of achievement (e.g., basic, proficient)  Different proportions of students reaching the same target (e.g., 80% of Level 3 students will achieve target, 65% of Level 2 students will achieve goal) 36

37. 37 Lessons Learned … And Still To Be Learned

38. Rhode Island: Lessons Learned from Development of SLOs Understanding of Process for all involved Consistency in implementation Participant investment Balancing many contexts (e.g., special education) 38

39. Rhode Island: Lessons Learned from Development of SLOs Support and Training From State to Evaluators, from Evaluators to Educators Ongoing support and training is needed Materials as guidance for implementation 39

40. Rhode Island: Lessons Learned from Development of SLOs Collaboration is key Within schools and districts Teacher team SLOs Encourages best practices (i.e., scoring each other’s work, using the same curriculum) Eases burden on evaluators 40

41. Scott Marion: We Still Have Much to Learn  If anyone tells you they have all the issues related to SLOs or NTSG solved, hold onto your wallet  This is all relatively new and there does not yet exist a body of research to shed light on the validity of using SLOs for educator evaluation  We think it offers tremendous potential for improving practice, but not without significant support 41

42. Scott Marion: We Still Have Much to Learn  We need to create an evaluation framework to help us design appropriate studies of SLOs and other aspects of educator evaluation, fortunately… 42

43. http://www.ride.ri.gov/educatorquality/EducatorEvaluation For more information on RI and to download detailed documents 43

44. Center for Assessment: Evaluating Educator Evaluation Systems  The Center for Assessment’s annual conference (Reidy Interactive Lecture Series—RILS)  September 13-14, 2012 in Boston  Focused specifically on developing approaches for evaluating educator evaluation systems, with a terrific set of speakers, including:  Courtney Bell (ETS), Henry Braun (BC), Heather Hill & Corinne Herlihy (Harvard), and representatives from NYC, Denver, and Montgomery County public schools  For information: www.nciea.org or smarion@nciea.orgwww.nciea.orgsmarion@nciea.org 44

45. Continue the Conversation Please continue the conversation in the private Evaluation Discussion Group Thread 45

46. To Join the Private Evaluation Discussion Group  SCEE members may log into the collaboration site (www.ccsso.org/scee)www.ccsso.org/scee  Mouse-over the “Discussions” navigation tab  Click on “Discussions Directory”Discussions Directory  Click on “Evaluation Private Discussions”Evaluation Private Discussions  Under the discussion group title and icon, click on “Request an Invitation” (or just click on this link)Request an Invitation 46

47. Upcoming Webinars  September 11:  What’s next with Common Core Implementation?  October 3: SPECIAL SCEE WEBINAR  The Principal’s Role in Evaluating Teachers  October 9:  The Complexity of Teaching and Leading: Observing Teacher Practice 47

48. Thank you 48