Getting Instructionally Actionable Information from Summative Assessment: NAEP Writing Process Data Peggy G. Carr, Ph.D. Acting Commissioner National Center for Education Statistics Council of Chief State School Officers Austin, Texas June 28, 2017

National Assessment of Educational Progress (NAEP) Congressionally mandated project administered by the National Center for Education Statistics Common measure of student achievement across the country in a variety of subject areas since 1969 Gold standard for large-scale assessments

NAEP’s Goals NAEP’s Goals Inform educators, researchers, policymakers, and stakeholders about academic progress in the U.S. and function as a common yardstick for states Track trends in student performance over time Report on group-level student performance and contextual information on student learning Serve as a leader in large-scale assessments, taking a lead role in innovation NAEP’s Goals Help inform educators, researchers, policymakers, and other stakeholders about the progress of academic education in the U.S Track trends in student performance over time Report on group-level student performance and contextual information on student learning

Who Participates in NAEP? Trial Urban Districts (TUDAs) The Nation States

Transition to Digitally Based Assessments (DBA) The Expanding Role of Technology in Classrooms and Students’ Lives at Home Increased Measurement Precision Economies and Efficiencies Enhanced Reporting Through Process Data

NAEP Writing Assessment Framework calls for measuring students’ ability to write on computer for specific purposes and audiences Recent DBA administrations 2011 (grades 8 and 12) 2017 (grades 4 and 8) Results reported as Average scale scores (0-300 scale) Achievement levels (Basic, Proficient, Advanced) Each student responds to two prompts that are drawn from the larger pool of 22

Sample Item – Grade 8 CAR-FREE COMMUNITY In some countries, special communities are being built where no cars are allowed. Instead, the community has public transportation, like buses and trains, so people have a way to travel where they need to go. Stores and schools are built within these communities so people do not have to travel far. What do you think of the idea of a car-free community? Write a letter to the editor of your local newspaper to express your point of view. Support your point of view with reasons and examples.

Sample Item – Grade 12 USE OF TECHNOLOGY Click the play button to watch a presentation about young people’s use of technology. Then do the task below. Write an essay for a college admissions committee about one kind of information or communication technology you use. Describe what it is and explain why the technology is important to you. Develop your essay with details so the admissions committee can understand the value of this technology. You may use information from the presentation in your essay.

What is Process Data? The “behind the scenes” actions and behaviors that can be captured during the assessment Process data can: Inform task and item development Improve the efficiency and accuracy of NAEP Help monitor, debug, and improve eNAEP during administration Support expanded and enriched reporting Provide added support for the validity of NAEP scale scores Contribute to innovation in large-scale assessments

Examples of 2011 Writing Process Data 29% of eighth-graders used the thesaurus 1 or more times. 80% or more twelfth-graders did not use the cut, copy, and paste features. 74% of twelfth-graders right-clicked to access the spell-check option 1 or more times. 71% of eighth-graders used the text-to-speech function 1 or more times.

Examples of 2017 Writing Process Data Key strokes Back space Pauses between letters/words/phrases Length of sustained typing Delete Insert text Use of upper case Activating drop-down menu Activate spell check Activate thesaurus Select and use suggested words from spell check and thesaurus Cut and paste

Visualizing the Writing Process

Reporting on Achievement Gaps White students outperformed Black students by 26 points in 2011

Reporting on Achievement Gaps Racial gap in the 2016 pilot study Based on sum scores from two prompts

Shedding New Light on Gaps Writing Process Data Writing Fluency Active Writing Time Frequency of Text-to-speech Uses Speed of Typing Familiar Words

Potential Enhancements for Reporting Descriptive statistics on student writing process data Comparison of writing process data at national and state levels Sub-group differences in writing process data Correlations between writing process data and constructs measured in student questionnaires

Final Thoughts Process data offers new opportunities to enrich and expand how we report results Collaboration among educators and researchers will help us to prioritize and focus our work Our ability to more thoroughly understand and unpack gaps in student performance will lead to more useful reporting

Thank you