NAPLAN Online: Contentious Issues and Risk Mitigation Strategies Research Based Approaches to the Development and Scoring of Technology-Enhanced Items Stanley Rabinowitz, Ph.D. NCSA 2016

Contentious Issues Year 3 Keyboarding Automated Essay Scoring

Year 3 Keyboarding: Issue All students at all year levels are expected to complete NAPLAN online for all content areas, including year 3 writing Two important (but different questions): is it reasonable to expect Year 3 students to use a keyboard to do the Year Writing? are all year students ready to use a keyboard for the Year 3 writing in 2017? 2018?

NAPLAN Research Findings to Date Evidence in Australia and internationally demonstrates that year 3 students have the requisite capacity: NAPLAN trialling data for year 3 students: No significant difference in quantity or quality of writing online vs paper Students more likely to edit writing Students more engaged online vs paper (based on cognitive interviews) Similar expectations for international year 3 assessments Australian Curriculum expects keyboarding and construction of text using word processing programs from F-3 (Similar expectations for international counterparts) Research in Australia and internationally indicates year 3 students average about 10-15 words per minute with 85% accuracy (average NAPLAN year 3 essay 100 words)

Year 3 Keyboarding Mitigation Strategies 2017 opt-in decisions should include keyboarding readiness Schools/sectors have until 2019 for all to be ready Identify best practice and model programs Ongoing consultation (Primary Principals, AEU, Teacher-prep programs) More targeted research planned as part of AES validation studies and trialling study

Automated Essay Scoring: Issue AES is an essential tool for early return of NAPLAN writing results Evidence in Australia and internationally demonstrates that AES works for NAPLAN-type writing Extensive consultation indicates that opposition to AES is part philosophical and part “emotional” (Head vs Heart)

NAPLAN Research Findings to Date AES solutions achieved the level of agreement with human markers which are observed between sets of human markers on all criteria. Comparable agreement rates for both total score and 10 criterion scores, including audience and ideas! Human markers Automated essay scoring system Marker 2 AES 1 AES 2 AES 3 AES 4 Total S core Marker 1 0.78 0.74 0.77 0.71 0.76 0.75 0.81

AES Mitigation Strategies ACARA will monitor and evaluate the implementation of the automated solution for NAPLAN 2017 online writing tests: extensive training of the automated essay solution using up to 1000 teacher-scored essays. checks during the marking process (eg. correlation with reading test results, pre-screening based on factors known to be problematic for AES, random), with identified essays reviewed by human markers. double marking of a further sample of student essays by experienced human markers; can range up to 100% depending on the profession and public’s comfort level with the AES process.

AES Mitigation Strategies (cont.) More targeted research planned in Spring 2016 using actual scoring engine on broad sample of NAPLAN students with both narrative and persuasive prompts: Evaluation and validity study: to investigate the robustness of the system and its capacity to maintain the validity of writing scores Finalisation and readiness study: to fine-tune the system for NAPLAN needs and prepare it for operational deployment in 2017