Usability Research: Lessons Learned For Digital Assessment Delivery Cathy Wendler Educational Testing Service June 21, 2013

Overview This session reflects on lessons learned as part of the digital delivery of high stakes assessments Digital delivery refers to administering tests using non-paper formats: tablets, PCs, laptops, iBT This session reports on five efforts related to digital delivery This session reflects on lessons learned as part of the digital delivery of high stakes assessments We use digital delivery to mean assessments that are not administered on paper but may be delivered on tablets, PCs, laptops, or through the internet This session presents the results of five different efforts related to digital test administration with a focus on understanding the benefits and challenges such delivery presents for test takers, teachers, and schools

Computer Based Achievement Assessment of Young Students Report on three small studies General computer usability study Modality comparability study Audio delivery usability study Results used to inform design and presentation decisions Provide evidence that young students could manage the CBT assessment The first presentation discusses three small studies that were conducted on English language arts and mathematics assessments given to students in Grades 1 through 3: two usability studies and a modality comparison study Results helped inform design and presentation decisions on the transition of the assessments to CBT delivery Results also provided evidence that young students could navigate the CBT assessment Christine Mills, a psychometrician from Educational Testing Service, is the presenter

Designing Items to Adapt to User-Supplied Hardware Report on usability study that compared performance for a test delivered on an iPad versus a laptop computer Specific question types included those that involve typing or touching the screen to perform various activities The second presentation reports on an usability study conducted to examine question types that could potentially cause a performance difference for a test delivered on an iPad versus a laptop computer Question types included those that involve typing or touching the screen using a finger on an iPad instead of a mouse to perform various actions such as highlighting text or drag and drop Study included students ages 11-13 with varying degrees of familiarity with touch screen tablets Debbie Pisacreta, a research systems specialist from Educational Testing Service, is the presenter

Accessible Tablets in Mainstream Classrooms Professional development for teachers Awareness of built-in accessibility features of iPads Training to teach teachers how to use features Curriculum that helps teachers Teach students who are visually impaired or blind in using features How to implement the iPad as an educational tool for reading, writing, and research in mainstream classrooms The next presenter is Diane Brauner, who is a certified orientation and mobility specialist and teacher of the visually impaired from North Carolina She observed that tablets with built-in accessibility features were being used in mainstream classrooms but not by students who are visually impaired or blind As a result, she engages in creating professional development activities for teachers that help them be aware of the accessibility features of the iPad, provides training in how to use the features, and develops curriculum that allows teachers to instruct visually impaired and blind students in using the features as well as implementing the iPad as an educational tool She will be giving a hands-on demonstration of some of the iPad accessibility features

K12 EL Identification Tool John Jay High School involvement with the pilot of a prototype English learner identification tool Innovative, digital tool Feedback collected from teachers who were involved in the pilot What they liked What challenges they faced Jay Sumpter, the principal of the Jay Science and Engineering Academy in San Antonio, is our fourth presenter He discusses the involvement of John Jay High School with the pilot of an innovative, K-12 English learner identification tool given on iPads In particular, he discusses feedback from teachers who were involved in the pilot of the prototype tool He also provides some of the challenges that the school faced when administering the tool

California Computer Based Testing Tryout in Science Report of study that collected feedback and data useful for the state in transitioning to CBT and CAT and allowed Students and schools to try CBT Development of technology-enhanced questions Evaluation of the feasibility of artificial intelligence scoring Science tests Grades 5 and 8 science; high school biology The final presentation reports on a study done by the California Department of Education using grades 5 and 8 science and high school biology tests This study collected feedback and data that will help the state in transitioning to computer-based and computer-adaptive testing The study allowed students and schools to try the CBT, allowed for the development and administration of new technology-enhanced questions, and evaluated the feasibility of using artificial intelligence for scoring Feedback from schools and students is presented as well as some of the challenges faced during testing Eric Zilbert, from the Assessment Development and Administration Division of the California Department of Education, is the presenter

Usability Research: Lessons Learned For Digital Assessment Delivery Detailed presentations and handouts are posted on the CCSSO conference site I believe you will find all of the presentations in the session to be informative as well as enjoyable All of the presentations and handouts are posted on the CCSSO conference site