Next-Generation MCAS: Update and review of standard setting Deputy Commissioner Jeff Wulfson Associate Commissioner Michol Stapel MCAS Chief Analyst Robert Lee May 23, 2017

Next-Generation MCAS Updates

Standard Setting for Next-Generation MCAS

Next-Generation MCAS Milestones November 2015 Board establishes next-generation MCAS March 2017 Board approves new achievement levels April-May 2017 First administration of next- generation MCAS in grades 3-8 August 14-18, 2017 Standard setting panels convened Massachusetts Department of Elementary and Secondary Education

What is standard setting? Cut score recommendations (how many points needed to reach each achievement level) Standard Setting Massachusetts Department of Elementary and Secondary Education

Student Expectations Content Standards Massachusetts Curriculum Frameworks in English Language Arts (ELA) and Mathematics Grade- and Subject-Specific Achievement Level Descriptors Descriptions of the knowledge, skills, and abilities associated with the top three achievement levels Massachusetts Department of Elementary and Secondary Education

Example of Achievement Level Descriptors (Draft) Grade 4 ELA - Writing Massachusetts Department of Elementary and Secondary Education

Logistics Dates: August 14–18, 2017 Location: Danvers, MA Standard setting meetings: 4 days Vertical articulation meeting: ½ day Location: Danvers, MA Meetings: 6 committees ELA: Grades 3–4, Grades 5–6, Grades 7–8 Mathematics: Grades 3–4, Grades 5–6, Grades 7–8 Massachusetts Department of Elementary and Secondary Education

Committee Composition Committee Participants 20–24 participants per committee (4-5 table leaders) Qualifications Be subject-matter experts Understand the examinee population and the instructional environment Be knowledgeable about the curriculum frameworks Be representative of key stakeholder groups Approximately equal representation of panelists at each grade level, including several panelists with experience at one grade higher than upper grade Massachusetts Department of Elementary and Secondary Education

Overview of Standard Setting Activities Plenary session that describes the purpose of standard setting and provides background to panelists and observers Panelists divide into grade- and subject-specific groups to complete standard setting tasks and make judgments, lead by facilitators Experience the test; review rubrics, scoring guides, and samples of student work Develop draft borderline achievement level descriptors Item judgment and feedback rounds Table leaders from different grade panels in the same subject come together to look at results across grades and make adjustments as needed to provide a coherent system (“vertical articulation”) Massachusetts Department of Elementary and Secondary Education

Achievement Levels and Cut Scores Massachusetts Department of Elementary and Secondary Education

Panelist Judgment Progress (Extended Modified Yes/No Angoff Method) Imagine a student with just enough ability to match the achievement level descriptor (“borderline student”) For each test item, answer the following question: For each item, the question is answered three times: Partially Meets Expectations Meets Expectations Exceeds Expectations The number of points the “borderline student” earns on the whole test will represent the cuts at each achievement level “How many points would a borderline student at the achievement level likely earn if he or she answered the question?” Massachusetts Department of Elementary and Secondary Education

Example of how ratings produce proposed cuts Massachusetts Department of Elementary and Secondary Education

Judgment Rounds Round 1: Panelists make individual judgments Rounds 2 and 3: Panels refine individual judgments, based on feedback and discussion Feedback data Recommended cuts that result from their item judgments (individual and group recommendations) Impact data (e.g., What percent of students would be Meeting Expectations?) Rounds of judgments conducted for the upper grade, then the lower grade Massachusetts Department of Elementary and Secondary Education

Vertical Articulation Examples of considerations during vertical articulation Looking across grades at the proportion of students at each achievement level Evaluating areas of disagreement among panelists or between panels Process Day 1-Day 4 Panelists and policymakers review impact data after each round Informs judgments and facilitation during the rounds Day 5 Table leaders convene to discuss the proposed cuts among all grades in a subject Table leaders recommend any needed adjustments to cuts, or provide an explanation for remaining differences Massachusetts Department of Elementary and Secondary Education

After Standard Setting Results are summarized by the contractor and presented to ESE staff and the Commissioner for approval Each student receives a scaled score and achievement level Reports printed and sent to parents/guardians (target date: end of October) Massachusetts Department of Elementary and Secondary Education