1. TEI Session – Discussant Comments
Sue Lottridge, Pacific Metrics

2. Technology Enhanced Items
What are TEIs?
Administered on computer
Novel to a population
Involving interaction beyond a single click
Drag and drop, drop down, typing
Introduction of ‘new’ technologies
Greater complexity
Need for research
Stakeholder beliefs
Monitoring

3. TEI Scoring and Performance Factors
Item Design
Structure
Clarity
Constraints & Enablers
Examinee
Experience with Item Type
Facility with Tools
“Gets” the Item
Scoring
Rubric Alignment
Scoring Implementation
Impact on Test Score
Experience with Item Type

4. Aligning projected elicited response against construct
Item Design
Aligning projected elicited response against construct
Partial credit determination
Expert judgement
89/1128 eligible for partial credit
May not be generalizable by task type
“Ordered” items
Can be difficult to predict examinee behavior
Likely need pattern data to determine if partial credit scoring may be relevant

5. Keyboarding vs. Writing for Year 3 Examinees Writing quality
Examinee Behavior
Keyboarding vs. Writing for Year 3 Examinees
Writing quality
Engagement
Editing
Protect against automatic credit

6. Scoring Comparison to Human Scoring/Judgement Automated Essay Scoring
Rules review
Response Patterns
Relationship with total score (item analysis)
Categorizing patterns
Unusual essay identification
Unordered IRT (NRM)
Enable viewing of option curves
Enable use of information from ‘options’ to inform ability estimation
Operational scoring
Ongoing monitoring required

7. Thank you!