Delivery platforms for national and international computer-based surveys History, issues and current status Sam Haldane Australian Council for Educational Research (ACER)

PISA CBAS Objectives – Comparability – translation – rich content Requirements – Security – Hardware, software – Test Administrator knowledge

CBAS - Implementation Installed client / server software Java – Java Media Framework (movies and sounds) – Internationalization / localization support – Networking Hardware platform – Client / server carry-in laptops + network

CBAS - Economics Delivery – carry in laptops = high cost and low failure rate Data capture – data management procedures Marking – not required due to nature of items Ease of translation – Low word count – Must install custom software – Translation of interactive elements

PISA ERA Objectives – Lower cost – Logistically easier – Use existing school infrastructure – Comparability not so important – Complex hypertext environment

PISA ERA Requirements – Cannot assume inter / intranet – Cannot assume host OS version – Cannot always install software – Must run on lowest common denominator hardware

PISA ERA Implementation – Bootable CD with USB storage in field trial – Bootable USB option in main survey – Linux OS using TAO as delivery platform – CD / USB contains all software needed (server, browser, Flash etc.) – Minimum possible hardware requirements – Need for TA to change boot order in BIOS

ERA - Economics Delivery – using existing school infrastructure = low cost but high failure Data capture – de-centralised on USB storage – data management logistics Marking – expert marked items, need for online marking software Translation – need for separate translation system, and localization of marking system

NAP – ICT Literacy Assessment Commissioned by MCEETYA 1 Australia wide assessment of ICT literacy in year 6 and year 10 (12 and 16 years old) Measures ability to – use ICT appropriately to access, manage, integrate and evaluate information – develop new understandings – communicate with others in order to participate effectively in society 1 Ministerial Council on Education, Employment, Training and Youth Affairs

NAP – ICT Literacy Assessment Three deployment models – Internet delivery – school computers used as clients with remote server – Carry-in server – school computers used as clients, with server laptop plugged into school LAN – Carry-in minilabs (10 machines) – same model as CBAS, for less equipped schools High success rate - 99% school-level response rate after replacements

NAP – ICT Literacy Assessment Technical issues – Remote desktop client requires non-standard port – Internet Explorer plugin requires installation, and Windows OS – TA training for all three deployment models – Schools requiring minilabs usually remote

CBAS vs ERA Carry-in laptops vs bootable CD / USB Comparability issues – ERA hardware differs greatly whereas CBAS was standard High failure rate in ERA, perhaps more so in some schools

The future Toolbox of delivery methods that are used depending on school infrastructure Internet delivery will have highest ROI – ease of deployment, administration costs, but lack of infrastructure

The future No silver bullet Carry-in server model best trade off at the moment Remote desktop based solution (for some projects) Interactive content with Flash Ajax – Google Apps possible future direction