Plans for Reporting Results and Performance Level Setting The Second Annual New Jersey Leadership Summit January 21, 2015 Jeffrey Nellhaus Chief of Assessment, PARCC, Inc.

1.Priorities Provide a brief review of the priorities that drive PARCC 2.Reports of Results You will become knowledgeable about the kinds of metrics and reports that will be used to report the results of the PARCC assessments next fall 3.Performance Level Setting (aka standard setting) You will be knowledgeable about the process PARCC intends to conduct this summer to identify the threshold scores for each of its five performance levels Objectives for this morning … 2

PARCC Consortium in AR CO IL LA MA MS NM NY OH TN MD NJ RI Governing Board States DC Administering the test in 12 members 5 million students Nearly 15% of students in grades 3-11 nationally About 3 in 4 will take the tests on computer

Develop tests worth taking Texts worth reading Essays worth writing Problems worth solving Accessible to all students Goals … Report results worth using Valid Reliable Timely Actionable 4

Determine whether students are college- and career-ready or “on track” Provide tools to assess student learning and support instruction during the school year Report comparable results across schools, districts and member states Generate data, including growth metrics, for accountability uses Priority Purposes Use technology to increase student access and engagement, and create efficiencies 5

PARCC Assessment System 6

Draft Score Report Designs “Results Worth Using” 7

PARCC Score Report Design Study PARCC conducted a 5-month, multi-phase Score Report Design Study to establish priorities for the kinds of data the reports should include and how they should be presented. Stakeholders providing input included: Teachers and Administrators (Central Office and Building) Parents Higher Education Representatives PARCC state education leadership PARCC Technical Advisory Committee (TAC) 8

Data Management & Reporting System PARCC score reports will be available to local educators online through its interactive web-based Data Management and Reporting System, and to parents via a printed student reports PARCC’ web-based system will provide authorized users to customize reports using the following “filters” 9 Student School District State PARCC Year Subject Type of Test Grade Student Group Overall Scores Status Scores Growth Scores Sub-Scores Item Level Scores

Performance Levels Math ELA/L Scale Scores Mathematics ELA/L Reading Writing Sub-Scores Whether the student’s performance in the sub- category was At/Above, Near, or Below the performance of students who scored “just Level 4” overall in the content area Metrics for Reporting Results At the level, student the number of points earned At the school level, the average number of points earned by all the students in the school who took the item. Item Scores

Draft of Individual Student Report 11

Draft of Student Roster - Sub-Scores 12

Draft of Student Roster – Item Scores 13

Draft School by Grade Summary Report 14

Draft District by School Summary Report 15

Draft State by District Summary Report 16

Draft PARCC by State Summary Report 17

Performance Level Setting 18

PARCC will report results using five performance levels (names under discussion) Students performing at Levels 4 or 5 on Algebra II, Integrated Math 3, or ELA/L grade 11 assessments will earn a College- and Career-Ready Determination Public colleges and universities in PARCC states have agreed to allow students who have been accepted and have earned CCR Determinations to enter directly into credit-bearing courses without remediation PARCC Performance Levels 19

Performance Level Setting: What is it? 20 Use Performance Level Descriptors & Data from Research Identify Threshold Scores Five PARCC Performance Levels

Who will participate? High School Panels (6) Grade 3-4 Panel (2) Grade 5-6 Panel (2) Grade 7-8 Panel (2) Performance Level Setting 21 K-12 educatorsPostsecondary faculty Business peopleCommunity members Grade-Span Panels Each Panel will Include 20 Participants

Schedule of Performance Level Setting Events 22 PARCC Performance Level Setting Schedule Panel Start DateEnd Date Subject/Grade Mathematics Panels ELA/Literacy Panels Dry Run5/21/20155/28/2015Algebra II and Integrated 3Grade 11 High School7/27/20157/31/2015 Algebra I and Integrated 1 Geometry and Integrated 2 Algebra II and Integrated 3 Grade 9 Grade 10 Grade 11 Grades 7–88/17/20158/21/2015Grades 7 and 8 Grades 3–68/24/20158/28/2015 Grades 5 and 6 Grades 3 and 4 Grades 5 and 6 Grades 3 and 4

Performance Level Setting: What Process Will be Used? PARCC will Use an Evidence-Based Process The process combines expert judgment with data from research The process will generate threshold scores that – Reflect PARCC’s performance level descriptors (PLDs) – Are consistent with the results reported by other well-regarded state, national and international assessment programs Because comparability is a priority, every state will adopt the same threshold scores 23

Performance Level Setting: Training Panelists will: Practice with online tutorial Take the computer-based assessment Review the performance level descriptors (PLDs) Write “borderline” performance level descriptors (PLDs) Practice with the process 24

Proposed Performance Level Setting Method 25 Yes/No Method (proposed) Review test items Answer the question: Based on the PLDs, would a student performing at Level X be likely to answer the item correctly? Impact data Impact data includes: Empirical data from research Judgments of other panelists % of students performing at each level % who got item correct Multiple Rounds After an initial round of review, panelists discuss their judgments, are given certain impact data and conduct a second round of review, After the second round, panelists discuss their judgments again, are given additional data and conduct a third and final round.

Grade 3 Sample Item and PLDs Grade 3 Math : Sub-Claim A The student solves problems involving the Major Content for the grade/course with connections to the Standards for Mathematical Practice. Level 5: Distinguished Command Level 4: Strong Command Level 3: Moderate Command Level 2: Partial Command Time 3.MD MD.1-2 Tells, writes and measures time to the nearest minute. Creates two-step real world problems involving addition and subtraction of time intervals in minutes. Tells, writes and measures time to the nearest minute. Solves two–step word problems involving addition and subtraction of time intervals in minutes. Tells, writes and measures time to the nearest minute. Solves one-step word problems involving addition or subtraction of time intervals in minutes. Tells, writes and measures time to the nearest minute. Solves one-step word problems involving addition or subtraction of time intervals in minutes, with scaffolding, such as a number line diagram. 26 Patricia needs to read for 120 minutes each week. She read for 26 minutes on Monday. She read for 39 minutes on Tuesday. She read for 38 minutes on Thursday. How many more minutes does Patricia need to read this week? minutes Click on the answer box and type in a number. Backspace to erase.

Research to Inform the Performance Level Setting Process Research Questions – What percentage of students are likely to be college- and career-ready, according to PARCC’s definition? – What percentage of students are likely to be on track to college and career readiness at earlier grades? Two studies to inform performance level setting 27 Postsecondary Educators Judgment Study Benchmark Study SAT ACT NAEP TIMSS PISA

Performance Level Setting Timeline 28

Sign up for the PARCC Updates newsletter at Sign up for the PARCC Updates newsletter at me Follow me on Follow us on Visit our website Sign up for the PARCC Updates newsletter on our website Updates and more information 29