Learning about the Item Review Process: An Overview

Item Development Timeline The first phase of item development contracts were awarded to ETS and Pearson in 2012. Phase I of item development began in the fall of 2012 and was completed in late summer 2013. Phase II began in October 2013 and will run through fall 2014.  2

Who are PARCC’s item reviewers? PARCC item reviewers come from PARCC governing and participating states K-16 educators, state department of education staff, and external experts Deep content expertise Experience with students from various backgrounds Many have participated in item reviews for their own states Experience in various geographic and educational settings

PARCC Item Review Committees Various review teams meet, both in person and virtually, to evaluate the items over an extended period of time. Only items that are approved by these teams of reviewers will appear on the PARCC summative assessments. Group Membership Charge Core Leadership Groups PARCC state DOE staff and HE faculty; approx. 60 members in each content area Review all test items developed for the PARCC summative assessments for suitability of content, age-appropriateness, and alignment to CCSS. Approve recommended revisions to items. State Educator Reviewers K-12 LEA staff and HE faculty; approx. 80 members in each content area, plus 45 passage reviewers Ensure that items are age-appropriate and are measuring the content of the Common Core State Standards for a given grade level. Bias & Sensitivity Reviewers Citizens and educators from various backgrounds; approx. 50 members in each content area Ensure that items and passages are fair, unbiased, and age-appropriate for a given grade level. Core Leadership groups are content area-based—there is one for ELA and one for Math State Educator Committees include: ELA Item—about 80 members ELA Passage—about 45 members Math Item--about 80 members Bias & Sensitivity Committees are also content area-based Operational Working Groups also play a large role in item review (next slide) 4

Operational Working Groups PARCC Operational Working Groups (OWGs) involved in item review: ELA/Literacy Mathematics Accessibility, Accommodations, and Fairness (AAF) OWG members are content experts from state departments of education After each round of item review, OWGs participate in “item reconciliation” to review suggested revisions from the item review committees Reconciliation following bias & sensitivity review includes members from content OWGs and AAF OWG

PARCC Item Review Process Items Developed Core Leadership Review State Educator Review Bias & Sensitivity Review Field Test Item Bank OWG Reconciliation Life cycle of an item Before items go into the item bank, they also undergo review and testing of their embedded technology

PARCC Item Review Meetings PARCC item review meetings are conducted both in-person and virtually, and last 3 – 5 days Meetings consist of independent review of items followed by group discussion ELA teams are organized by grade level (grades 9/10 are combined) Math and Bias & Sensitivity teams are organized by grade band Core Leadership groups have 4-5 reviewers per team; State Educator and Bias & Sensitivity groups have 6-8 reviewers per team Item review teams are facilitated by contractor staff Each team includes an OWG member

ELA Review Considerations/Criteria Does the item allow for the student to demonstrate the intended evidence statement(s) and to demonstrate the CCSS to be measured? Is the wording of the item clear, concise, and appropriate for the intended grade level? Does the item provide sufficient information and direction for the student to respond completely? Is the item free from internal clueing and miscues? Do the graphics and stimuli included as part of the item accurately and appropriately represent the applicable content knowledge? Are any graphics included as part of the item clear and appropriate for the intended grade level? If the item has a technology-based stimulus or requires a technology-based response, is the technology design effective and grade appropriate? Is the scoring guide/rubric clear, correct, and aligned with the expectations for performance that are expressed in the item or task? If the item is part of a PBA task, does it contribute to the focus and coherence of the task model? This slide lists general item review guidelines for ELA reviewers

Mathematics Review Considerations/Criteria Does the task measure the intended evidence statement(s)? Does the task measure the intended mathematical practice(s)? Is the task mathematically correct and free from errors? Is the wording of the task clear, concise, and grade-level appropriate? Are the graphics/stimuli in the task clear, accurate, appropriate for the task, and appropriate for the grade? Do each prompt and all associated graphics/stimuli contribute to the quality of the task? Is the scoring guide/rubric clear, correct and aligned with the expectations for performance that are expressed in the task? General review criteria for math reviewers

Bias & Sensitivity Review Considerations/Criteria Does the item disadvantage any population (gender, race, ethnicity, language, religion, socioeconomic status, disability or geographic region) for non-educationally relevant reasons? Does the item contain controversial or emotionally charged subject matter that is not supported by the Common Core State Standards? Is the item potentially offensive, demeaning, insensitive, or negative toward any population? Does the item depict any population in a stereotypical manner? General review criteria for bias reviewers

Questions?