1. NAPLAN Online: Contentious Issues and Risk Mitigation Strategies
Research Based Approaches to the Development and Scoring of Technology-Enhanced Items
Stanley Rabinowitz, Ph.D.
NCSA 2016

2. Contentious Issues
Year 3 Keyboarding
Automated Essay Scoring

3. Year 3 Keyboarding: Issue
All students at all year levels are expected to complete NAPLAN online for all content areas, including year 3 writing
Two important (but different questions):
is it reasonable to expect Year 3 students to use a keyboard to do the Year Writing?
are all year students ready to use a keyboard for the Year 3 writing in 2017? 2018?

4. NAPLAN Research Findings to Date
Evidence in Australia and internationally demonstrates that year 3 students have the requisite capacity:
NAPLAN trialling data for year 3 students:
No significant difference in quantity or quality of writing online vs paper
Students more likely to edit writing
Students more engaged online vs paper (based on cognitive interviews)
Similar expectations for international year 3 assessments
Australian Curriculum expects keyboarding and construction of text using word processing programs from F-3 (Similar expectations for international counterparts)
Research in Australia and internationally indicates year 3 students average about words per minute with 85% accuracy (average NAPLAN year 3 essay 100 words)

5. Year 3 Keyboarding Mitigation Strategies
2017 opt-in decisions should include keyboarding readiness
Schools/sectors have until 2019 for all to be ready
Identify best practice and model programs
Ongoing consultation (Primary Principals, AEU, Teacher-prep programs)
More targeted research planned as part of AES validation studies and trialling study

6. Automated Essay Scoring: Issue
AES is an essential tool for early return of NAPLAN writing results
Evidence in Australia and internationally demonstrates that AES works for NAPLAN-type writing
Extensive consultation indicates that opposition to AES is part philosophical and part “emotional” (Head vs Heart)

7. NAPLAN Research Findings to Date
AES solutions achieved the level of agreement with human markers which are observed between sets of human markers on all criteria.
Comparable agreement rates for both total score and 10 criterion scores, including audience and ideas!
Human markers
Automated essay scoring system
Marker 2
AES 1
AES 2
AES 3
AES 4
Total S
core
Marker 1
0.78
0.74
0.77
0.71
0.76
0.75
0.81

8. AES Mitigation Strategies
ACARA will monitor and evaluate the implementation of the automated solution for NAPLAN 2017 online writing tests:
extensive training of the automated essay solution using up to teacher-scored essays.
checks during the marking process (eg. correlation with reading test results, pre-screening based on factors known to be problematic for AES, random), with identified essays reviewed by human markers.
double marking of a further sample of student essays by experienced human markers; can range up to 100% depending on the profession and public’s comfort level with the AES process.

9. AES Mitigation Strategies (cont.)
More targeted research planned in Spring 2016 using actual scoring engine on broad sample of NAPLAN students with both narrative and persuasive prompts:
Evaluation and validity study: to investigate the robustness of the system and its capacity to maintain the validity of writing scores
Finalisation and readiness study: to fine-tune the system for NAPLAN needs and prepare it for operational deployment in 2017