1. Technology Enhanced Items — Signal or Noise?
 Wednesday, June 28, 2017 | 8:30 am–10:00 am JW Grand Ballroom | Salon 2 (JW Marriott Austin)

3. Presenters Catherine Welch, Univ of Iowa/ITP Jon S. Twing, Pearson
Wayne Camara, ACT
Stephen T. Murphy, Measured Progress 
Discussant: Joyce Zurkowski, CDOE 
Moderator: Douglas F. Becker, HMH
In these studies, they concluded that
TE are more difficult than MC;
TE are more discriminating than MC.
2. TE provided more information at the high ability range.
Their statistics of interest are all IRT based, such as item characteristic function, item information functions, information efficiency.

4. TEIs in RFPs
The vendor’s response must reflect familiarity with computer-based testing and the use of a variety of item types, including technology enhanced items (TEI) to assess students’ higher order cognitive skills as well as their knowledge of core ideas and concepts.  (RFP DOE New Hampshire Statewide Assessments) 
STATE desires use of multiple item types and technology enhanced items to capitalize on efficiency while ensuring that NDSA assessments are aligned to the full breadth, depth, and cognitive complexity of STATE’s content standards.  (North Dakota State Assessment RFP )
Technology: Assessments will be delivered primarily online, include effective technology-enhanced items, and a facile, intuitive test management system.  (Nebraska Statewide Assessments RFI )
In these studies, they concluded that
TE are more difficult than MC;
TE are more discriminating than MC.
2. TE provided more information at the high ability range.
Their statistics of interest are all IRT based, such as item characteristic function, item information functions, information efficiency.

5. Previous Studies Wan & Henly (AME 2012) Crabtree & Welch (NCME 2016)
TEIs and MCs in statewide science achievement test
Figural Response, Constructed Response
Crabtree & Welch (NCME 2016)
TEIs and MCs in an end-of-course Algebra exam
Bukhari, Boughton, & Kim (NCME 2016)
TEIs and MCs in grade 8 math and grade 7 ELA in PARCC and SBAC
Matching, Equation and Expression Entry,
Select and Order, Multiple Correct Response;
Multiple Correct Response, Evidence-Based Selected Response

6. Previous Studies IRT based
ICC, average item information, information efficiency
Conclusions
TEIs are more difficult than MCs.
TEIs are more discriminating than MCs.
TEIs provide more information at high ability
In these studies, they concluded that
TE are more difficult than MC;
TE are more discriminating than MC.
2. TE provided more information at the high ability range.
Their statistics of interest are all IRT based, such as item characteristic function, item information functions, information efficiency.

7. TEI’s at other conferences
Tech-Enhanced Items: How Can They Provide Better Measurement?
This presentation will critically evaluate a number of TEI formats and how they might be scored, from both a classical and item response theory (IRT) perspective. This discussion will be couched in the consideration of actual data delivered in K-12 educational assessments. One notable aspect is that substantial information can be reaped from incorrect responses, which are generally not considered except for advanced IRT models like partial credit or nominal response models. Even these do not always consider all available data. For example, a straightforward multiple- response item with six possible options and two correct answers might be scored as points—better than simply 0-1—but not considering which of the four incorrect options were selected ignores any possible information that might be present there. Of course, this typically happens with multiple-choice items too, as the nominal response model is virtually never used in practice. In other cases— such as PARCC two-part items—IRT modeling frankly falls apart.

8. TEI’s at other conferences
Investigation of Psychometric Properties of Technology-Enhanced Items Considering Content Characteristics
Rong Jin, Stephen Murphy, Sid Sharairi -- Houghton Mifflin Harcourt Publishing 
Recent research explored technology-enhanced (TE) items only by subject . This study dives into DOK, master domain, and grade level to explore over six hundreds TEs in twelve formats from a K-8 large scale mathematics assessment and compares the measurement properties between multiple-choice and TE, and between TE formats . 
In a session mostly dealing with CAT and stopping rules

9. Domain and Item Type Mean TEI > MC in each domain.
Generally speaking,
The average Median Response Time of TE are higher than MC in every domain.
Especially in Domain EE.
The Domain related to grade 6 above usually have high Response Time in TE.
Such as EE, NS, SP, RP.
The domain covers grade 0-5 usually have low response time in TE.
Such as CC (0), OA, NBT, MD (0-5)
The standard deviation of Response Time in TE are usually higher than MC in most domain.
Epically in Domain EE, G, NS, SP, RP (grade 6 above)
Mean
TEI > MC in each domain.
Range for TEI and MC is grade related
low: CC, OA, NBT, MD (grade K-5)
high: EE, NS, SP, RP (grade 6+)
Standard Deviation
MC < TEI in each domain.
Large (40+) for TEI in domain G, NS, SP, especially in EE.

10. Conclusions MC Vs. TEI TEI Types
On average, TEI are more difficult, discriminating, and time consuming than MC in each DOK or domain.
On average, when DOK increases, both MC and TEI become more challenging, less discriminating, and more time consuming but to different extents.
On average, both MC and TEI in domains available in grade 6+ are more challenging and time consuming than those in domains in grade K-5.
TEI Types
Easy: Number Line, Ordering, and Matching.
Hard: Multiple Select, Matrix, Graph, Enter Math.
Time consuming: Enter Math and Graph.

11. Presenters Catherine Welch, Univ of Iowa/ITP
Stephen T. Murphy, Measured Progress 
Wayne Camara, ACT
Jon S. Twing, Pearson
Discussant: Joyce Zurkowski, CDOE 
Moderator: Douglas F. Becker, HMH