1. Best Practices in Quality Test Administration
Lessons Learned in the Administration of the National Assessment of Educational Progress (NAEP)
Gina Broxterman, National Center for Education Statistics (NCES)
June 20, 2016

2. NAEP Quality Assurance (QA): Purpose
Monitoring NAEP processes across the entire assessment program
Providing independent checks and feedback for various processes
Ensuring continuous improvement of the NAEP program
Ensuring standardization of assessment across the nation

3. NAEP QA: Pre-Administration
Item development
Items go through a multi-step quality assurance (QA) process before they are used in the operational assessment
Training of field staff
Field Staff are required to complete a combination of distance and in-person training
Transition to digitally based assessments
Configuration of equipment
Safeguards of delivery mode
Safeguards for system challenges

4. NAEP Assessment Administration

5. NAEP QA: Administration Best Practices
NAEP Activities
Series of quality assurance debriefs
Item Development
Quality assurance reviews of items
Design, Analysis and Reporting
Review of performance data
Sample and Data Collection
NAEP field staff fill out debriefing questionnaires
Contractors host an in-person debrief meeting with all stakeholders to discuss all facets of NAEP 2016 administration
From the observation data, Sampling and Design contractor develops action items that will improve implementation for upcoming administrations

6. NAEP QA: Administration Best Practices
NAEP State Coordinators (NSC)
Conduct observations
Provide feedback on the uniformity of the administration across all states
Embed QA practices in NSC data collection processes
Pre- and post-demographic data reviews
Convene working groups to revise processes for NAEP administration
Identified Issue: 2015 MyNAEP system was cumbersome for users
Action Plan: reduce jargon; reduce number of pages; add worksheets to streamline school's data collection; and provide multiple options for updating student lists

7. NAEP QA: Administration Best Practices
HumRRO
Facilitate the Quality Assurance Technical Panel (QATP) to provide technical advice to NCES about potentially problematic NAEP issues, and design and conduct quality assurance research studies
Review NAEP contractor quality control plans and provide recommendations for improvements to NAEP operations and processes

8. Lessons Learned

9. NAEP: Lessons Learned - Translation
Issue:
Development schedule didn’t allow for thorough translation process
Action Plan:
Conduct translations earlier in the item development process
Issue:
Inefficient process for training NAEP scorers
Action Plan:
Develop an annotated guide that specifies purpose, style, format, and expected level of details, including the extent to which student responses should be quoted

10. NAEP 2016: Lessons Learned - DBA
Issue:
Standardization of software and updates across large volume of equipment created potential risk.
Action Plan:
Associate the tablet's login screen background color with a particular software version so that field staff can easily identify the version type
Issue:
Tablets and the Pelican case were hard to transport/lift
Action Plan:
Create a best practices guide to be added to the field staff manual of best ways to transport the pelican case in various school settings
Issue:
Session script for DBA assessments was not developmentally appropriate for fourth grade students
Action Plan:
Develop grade-specific session scripts that provides additional guidance for fourth-grade students

11. NAEP 2016: Lessons Learned - Training
Issue:
Creation of materials at the field staff level that opens potential for quality assurance issues.
Action Plan:
Develop additional quick check fact sheets for NAEP 2017 and organize in a central location for field staff
Issue:
Need more hands-on practice during in-person trainings
Action Plan:
Incorporate more hands-on practice with the field equipment

12. Future NAEP Administration: Quality Assurance

13. Future NAEP Administrations: QA Goals
Embed QA with Technology into processes
Quality Assurance Dashboard:
Develops set of metrics used to track and evaluate success of assessment adminstration
Capitalizes on technology to track assessments in progress and understand when and why issues arise.

14. Future NAEP Administrations: QA Goals
Embed QA with Technology into processes
Quality Assurance Dashboard:
Develops set of metrics used to track and evaluate success of assessment adminstration
Capitalizes on technology to track assessments in progress and understand when and why issues arise.