1. Getting Instructionally Actionable Information from Summative Assessment: NAEP Writing Process Data
Peggy G. Carr, Ph.D.
Acting Commissioner
National Center for Education Statistics
Council of Chief State School Officers
Austin, Texas
June 28, 2017

2. National Assessment of Educational Progress (NAEP)
Congressionally mandated project administered by the National Center for Education Statistics
Common measure of student achievement across the country in a variety of subject areas since 1969
Gold standard for large-scale assessments

3. NAEP’s Goals NAEP’s Goals
Inform educators, researchers, policymakers, and stakeholders about academic progress in the U.S. and function as a common yardstick for states
Track trends in student performance over time
Report on group-level student performance and contextual information on student learning
Serve as a leader in large-scale assessments, taking a lead role in innovation
NAEP’s Goals
Help inform educators, researchers, policymakers, and other stakeholders about the progress of academic education in the U.S
Track trends in student performance over time
Report on group-level student performance and contextual information on student learning

4. Who Participates in NAEP?
Trial Urban Districts (TUDAs)
The Nation
States

5. Transition to Digitally Based Assessments (DBA)
The Expanding Role of Technology in Classrooms and Students’ Lives at Home
Increased Measurement Precision
Economies and Efficiencies
Enhanced Reporting Through Process Data

6. NAEP Writing Assessment
Framework calls for measuring students’ ability to write on computer for specific purposes and audiences
Recent DBA administrations
2011 (grades 8 and 12)
2017 (grades 4 and 8)
Results reported as
Average scale scores (0-300 scale)
Achievement levels (Basic, Proficient, Advanced)
Each student responds to two prompts that are drawn from the larger pool of 22

7. Sample Item – Grade 8 CAR-FREE COMMUNITY
In some countries, special communities are being built where no cars are allowed. Instead, the community has public transportation, like buses and trains, so people have a way to travel where they need to go. Stores and schools are built within these communities so people do not have to travel far.
What do you think of the idea of a car-free community? Write a letter to the editor of your local newspaper to express your point of view. Support your point of view with reasons and examples.

8. Sample Item – Grade 12 USE OF TECHNOLOGY
Click the play button to watch a presentation about young people’s use of technology. Then do the task below.
Write an essay for a college admissions committee about one kind of information or communication technology you use. Describe what it is and explain why the technology is important to you. Develop your essay with details so the admissions committee can understand the value of this technology. You may use information from the presentation in your essay.

9. What is Process Data?
The “behind the scenes” actions and behaviors that can be captured during the assessment
Process data can:
Inform task and item development
Improve the efficiency and accuracy of NAEP
Help monitor, debug, and improve eNAEP during administration
Support expanded and enriched reporting
Provide added support for the validity of NAEP scale scores
Contribute to innovation in large-scale assessments

10. Examples of 2011 Writing Process Data
29% of eighth-graders used the thesaurus 1 or more times.
80% or more twelfth-graders did not use the cut, copy, and paste features.
74% of twelfth-graders right-clicked to access the spell-check option 1 or more times.
71% of eighth-graders used the text-to-speech function 1 or more times.

11. Examples of 2017 Writing Process Data
Key strokes
Back space
Pauses between letters/words/phrases
Length of sustained typing
Delete
Insert text
Use of upper case
Activating drop-down menu
Activate spell check
Activate thesaurus
Select and use suggested words from spell check and thesaurus
Cut and paste

12. Visualizing the Writing Process

13. Reporting on Achievement Gaps
White students outperformed Black students by 26 points in 2011

14. Reporting on Achievement Gaps
Racial gap in the 2016 pilot study
Based on sum scores from two prompts

15. Shedding New Light on Gaps
Writing Process Data
Writing Fluency
Active Writing Time
Frequency of Text-to-speech Uses
Speed of Typing Familiar Words

16. Potential Enhancements for Reporting
Descriptive statistics on student writing process data
Comparison of writing process data at national and state levels
Sub-group differences in writing process data
Correlations between writing process data and constructs measured in student questionnaires

17. Final Thoughts
Process data offers new opportunities to enrich and expand how we report results
Collaboration among educators and researchers will help us to prioritize and focus our work
Our ability to more thoroughly understand and unpack gaps in student performance will lead to more useful reporting

18. Thank you