Technology Enhanced Items — Signal or Noise?  Wednesday, June 28, 2017 | 8:30 am–10:00 am JW Grand Ballroom | Salon 2 (JW Marriott Austin)

Presenters Catherine Welch, Univ of Iowa/ITP Jon S. Twing, Pearson Wayne Camara, ACT Stephen T. Murphy, Measured Progress  Discussant: Joyce Zurkowski, CDOE  Moderator: Douglas F. Becker, HMH In these studies, they concluded that TE are more difficult than MC; TE are more discriminating than MC. 2. TE provided more information at the high ability range. Their statistics of interest are all IRT based, such as item characteristic function, item information functions, information efficiency.

TEIs in RFPs The vendor’s response must reflect familiarity with computer-based testing and the use of a variety of item types, including technology enhanced items (TEI) to assess students’ higher order cognitive skills as well as their knowledge of core ideas and concepts.  (RFP 2017-073 DOE New Hampshire Statewide Assessments)  STATE desires use of multiple item types and technology enhanced items to capitalize on efficiency while ensuring that NDSA assessments are aligned to the full breadth, depth, and cognitive complexity of STATE’s content standards.  (North Dakota State Assessment RFP 201-2017-175) Technology: Assessments will be delivered primarily online, include effective technology-enhanced items, and a facile, intuitive test management system.  (Nebraska Statewide Assessments RFI11162016) In these studies, they concluded that TE are more difficult than MC; TE are more discriminating than MC. 2. TE provided more information at the high ability range. Their statistics of interest are all IRT based, such as item characteristic function, item information functions, information efficiency.

Previous Studies Wan & Henly (AME 2012) Crabtree & Welch (NCME 2016) TEIs and MCs in statewide science achievement test Figural Response, Constructed Response Crabtree & Welch (NCME 2016) TEIs and MCs in an end-of-course Algebra exam Bukhari, Boughton, & Kim (NCME 2016) TEIs and MCs in grade 8 math and grade 7 ELA in PARCC and SBAC Matching, Equation and Expression Entry, Select and Order, Multiple Correct Response; Multiple Correct Response, Evidence-Based Selected Response

Previous Studies IRT based ICC, average item information, information efficiency Conclusions TEIs are more difficult than MCs. TEIs are more discriminating than MCs. TEIs provide more information at high ability In these studies, they concluded that TE are more difficult than MC; TE are more discriminating than MC. 2. TE provided more information at the high ability range. Their statistics of interest are all IRT based, such as item characteristic function, item information functions, information efficiency.

TEI’s at other conferences Tech-Enhanced Items: How Can They Provide Better Measurement? This presentation will critically evaluate a number of TEI formats and how they might be scored, from both a classical and item response theory (IRT) perspective. This discussion will be couched in the consideration of actual data delivered in K-12 educational assessments. One notable aspect is that substantial information can be reaped from incorrect responses, which are generally not considered except for advanced IRT models like partial credit or nominal response models. Even these do not always consider all available data. For example, a straightforward multiple- response item with six possible options and two correct answers might be scored as 0-1-2 points—better than simply 0-1—but not considering which of the four incorrect options were selected ignores any possible information that might be present there. Of course, this typically happens with multiple-choice items too, as the nominal response model is virtually never used in practice. In other cases— such as PARCC two-part items—IRT modeling frankly falls apart.

TEI’s at other conferences Investigation of Psychometric Properties of Technology-Enhanced Items Considering Content Characteristics Rong Jin, Stephen Murphy, Sid Sharairi -- Houghton Mifflin Harcourt Publishing  Recent research explored technology-enhanced (TE) items only by subject . This study dives into DOK, master domain, and grade level to explore over six hundreds TEs in twelve formats from a K-8 large scale mathematics assessment and compares the measurement properties between multiple-choice and TE, and between TE formats .  In a session mostly dealing with CAT and stopping rules

Domain and Item Type Mean TEI > MC in each domain. Generally speaking, The average Median Response Time of TE are higher than MC in every domain. Especially in Domain EE. The Domain related to grade 6 above usually have high Response Time in TE. Such as EE, NS, SP, RP. The domain covers grade 0-5 usually have low response time in TE. Such as CC (0), OA, NBT, MD (0-5) The standard deviation of Response Time in TE are usually higher than MC in most domain. Epically in Domain EE, G, NS, SP, RP (grade 6 above) Mean TEI > MC in each domain. Range for TEI and MC is grade related low: CC, OA, NBT, MD (grade K-5) high: EE, NS, SP, RP (grade 6+) Standard Deviation MC < TEI in each domain. Large (40+) for TEI in domain G, NS, SP, especially in EE.

Conclusions MC Vs. TEI TEI Types On average, TEI are more difficult, discriminating, and time consuming than MC in each DOK or domain. On average, when DOK increases, both MC and TEI become more challenging, less discriminating, and more time consuming but to different extents. On average, both MC and TEI in domains available in grade 6+ are more challenging and time consuming than those in domains in grade K-5. TEI Types Easy: Number Line, Ordering, and Matching. Hard: Multiple Select, Matrix, Graph, Enter Math. Time consuming: Enter Math and Graph.

Presenters Catherine Welch, Univ of Iowa/ITP Stephen T. Murphy, Measured Progress  Wayne Camara, ACT Jon S. Twing, Pearson Discussant: Joyce Zurkowski, CDOE  Moderator: Douglas F. Becker, HMH