Proposed End-of-Course (EOC) Cut Scores for the Spring 2015 Test Administration Presentation to the Nevada State Board of Education March 17, 2016

Agenda Background Standard Setting Process Proposed 2015 Cut Scores Toward a Compensatory Model 2

End of Course Examinations Nevada students, starting with the class of 2017, must take four End of Course (EOC) examinations. Each EOC exam measures how well a student understands the subject areas tested. This requirement comes from the 2013 legislative session, SB 288, NRS The EOC exams will take the place of the Nevada High School Proficiency Examinations (HSPE). EOC exams were given for the first time in the spring of 2015 and were administered in the following subjects: English Language Arts I: focus on reading comprehension English Language Arts II: focus on writing Math I: emphasis on Algebra I Math II: emphasis on Geometry 3

Transition Period During the current transition phase, the classes of 2017 and 2018 will only need to participate in the EOC Exams and will not be required to earn a passing grade in order to graduate. The Class of 2019 will be the first class of students required to earn a passing score on the EOC Examinations to graduate. 4

2015 Cut Scores In late 2015 and early 2016, standard settings were conducted to establish cut scores on the EOC tests given in Spring –These cut scores are designed to be used only on the 2015 tests. –A new standard setting will be conducted after the Spring 2016 administration. 5

Standard Setting as Part of a Process Content Standards Design AdoptionImplementation Assessment Development Eligible Content Item Development Form Construction Setting Achievement Standards Achievement Level Descriptors (ALDs) Cut Score Recommendations Policy Review 6

Content-Based Process Standards Nevada Academic Content Standards define what students should learn and what is assessed. ALDs Describe in words the content-based expectations for students in each achievement level. Cut Scores As part of a standard setting process, educators examine the ALDs and tests to transform content- based expectations into numeric cut scores. 7

Spring 2015 Administration Students who have taken (or are enrolled in) a course that includes the recommended standards for an EOC Exam are eligible to take that test. The Spring 2015 administration included 7 th, 8 th, 9 th, and 10 th graders who were eligible. 8

Insights from Spring 2015 An operational field test was conducted in There were no stakes for students in grades 9 and 10: these students may not have been motivated. Students may have found the assessment items difficult, based on p-values, especially for new types of test questions. Educators at the standard setting commented on the importance of aligning instruction to the complexity of the curriculum and assessment, all as aligned to the content standards. 9

Establishing 2015 Cut Scores  Review Standards and ALDs Standard setting participants reviewed the Nevada Academic Content Standards. Then they studied the achievement level descriptors (ALDs).  Content-Based Standard Settings Educators reviewed the test items. Then they made content-based cut score recom- mendations using the Yes/No Angoff or Body of Work procedures.  Reviewing the Cut Scores The standard setting committee studied the impact of the cut scores on students. Then the policy review committee considered the cut scores and made recom- mendations.  Finalizing Cut Scores The cut scores have been approved by the Council to Establish Academic Standards. They are now submitted for Board approval. 10

Achievement Level Descriptors The ALDs were developed by Nevada educators. They reflect a sample of the knowledge and skills expected of students at each of the four achievement levels –Level 1 (lowest) –Level 2 –Level 3 –Level 4 (highest) 11

A process that allows experts to consider the content-based expectations for students in each achievement level, and to transform those expectations into numeric cut scores on the assessments. Standard Setting 12

Four Committees Standard SettingPolicy Review ELA Feb Nevada educators: 6 classroom teachers 4 non-teacher educators 3 other 6 Nevada administrators: 4 administrators 2 other Math Nov Nevada educators: 5 classroom teachers 5 administrators 4 other 8 Nevada administrators: 5 administrators 3 other 13

Yes/No Angoff Implemented for ELA I, Math I, and Math II. Focuses on students’ expected knowledge and skills for each achievement level. Grounded in content expertise and guided by the achievement level descriptors (ALDs) Item-centered method where participants determine expected performance of a student in each level for each item –“Should a student who is just in Level 3 answer this item correctly? Yes or no?” 14

Body of Work Implemented for ELA II. Also grounded in content expertise and guided by the ALDs. Student-centered method where participants study the writing prompts, examine the scoring rubric, and read many examples of student writing. –“Based on only what the student has written, which achievement level best describes the student’s writing?” 15

Policy Review The committee considered: –the ALDs, –the cut scores from standard setting, and –contextual information on how Nevada students performed on statewide tests. The committee’s goal was: –to recommend a single set of cut scores, and –to make sure the system of achievement standards is well articulated and sends consistent signals. 16

Multi-Step Process Standard Setting: Content View –Grounded in ALDs and content-based expectations for students in each level –Committee of Nevada educators from across the state, using their content expertise –Confidence in the process Policy Review: Systemwide View –Considered the intended effect on instruction, accountability, educator effectiveness, and policy –Recommend adjustments to promote cohesion and consistency of the system –Endorsement of the process 17

Proposed 2015 EOC Cut Scores 18

Percent of Nevada Students in Each Achievement Level Based on Recommended 2015 Cut Scores 19

Mathematics Results by Grade Grade 7Grade 8Grade 9Grade 10 Math I89.2%69.5%19.2%23.0% Math II***85.7%39.8%13.3% 22 Percent of Students at or Above Level 3 Based on Recommended 2015 Cut Scores for EOC Math, by Grade and Test *** These results suppressed for privacy due to low N-count.

Recommendations CourseLevel 1Level 2Level 3Level 4 ELA I % % % % ELA II % % % % Math I % % % % Math II % % % % 23

Toward a Compensatory Model for the EOC Graduation Eligibility Requirement 24

Starting with this year's 9th grade class, students must pass four EOC tests to meet Nevada graduation requirements. –These include ELA I, ELA II, Math I and Math II (or Integrated Math I and Integrated Math II). –NDE intends to introduce a new test of EOC Science and to consolidate the two ELA tests into a single assessment. Graduation Requirements 25

NDE is concerned that the current conjunctive requirement of passing four EOC tests may penalize students who narrowly miss passing a test by a few points. Accordingly, NDE is considering moving toward a compensatory model where students' performance on one test can compensate for lower performance on another. Context 26

NDE is considering various models with compensatory elements. –NDE would prefer to establish uniform decision rules which can be used over time, even if the tests or cut scores themselves change. –The Spring 2016 assessment could be used as a baseline year. Considering a Model 27

Three Example Models Model based on composite score Model based on status Model based on new cut score 28

Model Based on Composite Score A student’s combined test scores for the four EOC assessments meets or exceeds the sum of the Level 3 cut scores for the test; and each of the student’s scores on the EOC assessment meets or exceeds an alternative cut score established for that assessment that is lower than Level 3 (e.g., Level 2, one standard error of measurement lower than Level 3). 29

Model Based on Status The Level 3 achievement level is the goal for all students. To meet the graduation eligibility requirement, students must meet the Level 3 cut score on three out of the four tests, and must score at the Level 2 level or higher on all tests. 30

Model Based on New Cut Score The Level 3 achievement level is the goal for all students. A cut score below Level 3, but above Level 2, is established to indicate the minimum graduation eligibility requirement for students. To meet the graduation eligibility requirement, students must meet this new cut score on all tests. 31

Challenges The data from the Spring 2015 administration may not be similar to that expected in future years. –It may be difficult to use the Spring 2015 data to explore different scenarios for the compensatory model. A new standard setting is scheduled for EOC tests, to be held after the Spring 2016 administration. –A standard setting will also be needed for the new test of EOC Science, and may be needed after ELA I and II are consolidated. 32

Invitation for Possible Board Action to Approve 2015 Cut Scores for Four EOC Exams 33

Final 2015 Recommendations CourseLevel 1Level 2Level 3Level 4 ELA I ELA II Math I Math II