Best Practices in Quality Test Administration Lessons Learned in the Administration of the National Assessment of Educational Progress (NAEP) Gina Broxterman, National Center for Education Statistics (NCES) June 20, 2016

NAEP Quality Assurance (QA): Purpose Monitoring NAEP processes across the entire assessment program Providing independent checks and feedback for various processes Ensuring continuous improvement of the NAEP program Ensuring standardization of assessment across the nation

NAEP QA: Pre-Administration Item development Items go through a multi-step quality assurance (QA) process before they are used in the operational assessment Training of field staff Field Staff are required to complete a combination of distance and in-person training Transition to digitally based assessments Configuration of equipment Safeguards of delivery mode Safeguards for system challenges

NAEP Assessment Administration

NAEP QA: Administration Best Practices NAEP Activities Series of quality assurance debriefs Item Development Quality assurance reviews of items Design, Analysis and Reporting Review of performance data Sample and Data Collection NAEP field staff fill out debriefing questionnaires Contractors host an in-person debrief meeting with all stakeholders to discuss all facets of NAEP 2016 administration From the observation data, Sampling and Design contractor develops action items that will improve implementation for upcoming administrations

NAEP QA: Administration Best Practices NAEP State Coordinators (NSC) Conduct observations Provide feedback on the uniformity of the administration across all states Embed QA practices in NSC data collection processes Pre- and post-demographic data reviews Convene working groups to revise processes for NAEP administration Identified Issue: 2015 MyNAEP system was cumbersome for users Action Plan: reduce jargon; reduce number of pages; add worksheets to streamline school's data collection; and provide multiple options for updating student lists

NAEP QA: Administration Best Practices HumRRO Facilitate the Quality Assurance Technical Panel (QATP) to provide technical advice to NCES about potentially problematic NAEP issues, and design and conduct quality assurance research studies Review NAEP contractor quality control plans and provide recommendations for improvements to NAEP operations and processes

Lessons Learned

NAEP: Lessons Learned - Translation Issue: Development schedule didn’t allow for thorough translation process Action Plan: Conduct translations earlier in the item development process Issue: Inefficient process for training NAEP scorers Action Plan: Develop an annotated guide that specifies purpose, style, format, and expected level of details, including the extent to which student responses should be quoted

NAEP 2016: Lessons Learned - DBA Issue: Standardization of software and updates across large volume of equipment created potential risk. Action Plan: Associate the tablet's login screen background color with a particular software version so that field staff can easily identify the version type Issue: Tablets and the Pelican case were hard to transport/lift Action Plan: Create a best practices guide to be added to the field staff manual of best ways to transport the pelican case in various school settings Issue: Session script for DBA assessments was not developmentally appropriate for fourth grade students Action Plan: Develop grade-specific session scripts that provides additional guidance for fourth-grade students

NAEP 2016: Lessons Learned - Training Issue: Creation of materials at the field staff level that opens potential for quality assurance issues. Action Plan: Develop additional quick check fact sheets for NAEP 2017 and organize in a central location for field staff Issue: Need more hands-on practice during in-person trainings Action Plan: Incorporate more hands-on practice with the field equipment

Future NAEP Administration: Quality Assurance

Future NAEP Administrations: QA Goals Embed QA with Technology into processes Quality Assurance Dashboard: Develops set of metrics used to track and evaluate success of assessment adminstration Capitalizes on technology to track assessments in progress and understand when and why issues arise.

Future NAEP Administrations: QA Goals Embed QA with Technology into processes Quality Assurance Dashboard: Develops set of metrics used to track and evaluate success of assessment adminstration Capitalizes on technology to track assessments in progress and understand when and why issues arise.