1. Paul Deane National Conference on Student Assessment, June 27-29, 2016
Studying the Writing Process: Exploring the Use of Writing Process Logs to Support Instruction
Paul Deane
National Conference on Student Assessment, June 27-29, 2016

2. What is a Writing Process Log?
A record of what happens while students write, from which we can
Reconstruct what the writer’s text looked like at every step of the process
Capture the timing of each event
Analysis of writing process logs can provide important evidence about what is happening while students write

3. Why Writing Process Logs Matter
The sequence and timing of actions reflects the thought processes that led to the final response
Can support richer inferences (and more useful feedback) in the classroom
Also provides important validity evidence for writing assessment

4. Writing Process Logs Capture Information That is Invisible in the Final Written Product
What did the writer edit or revise?
Where did the writer have to stop and think?
How much time and effort did the writer put into different parts of the writing task?

5. Writing Process Log Analysis: A New Field of Study
Builds on extensive research by European scholars
Leijten, M., & Van Waes, L. (2013). Keystroke logging in writing research: Using Inputlog to analyze and visualize writing processes. Written Communication, 30,
Applications to educational assessment developed at ETS
(Almond et al., 2012; Deane & Quinlan, 2010; Deane, Quinlan, & Kostin, 2011; Deane et al., 2014; Deane & Zhang, 2015; Zhang & Deane, 2015)
NAEP 2017 reporting for writing will include writing process data

6. Writing Process Log Replay Demo

7. Plan for the Rest of the Talk
Background: the cognitive processes involved in writing
Process features that distinguish stronger from weaker writers
How writing process logs can support instruction
Discussion

8. The Cognitive Processes Involved in Writing

9. Core Cognitive Writing Processes
Evaluator
Proposer
Translator
Transcriber
This process is relevant to all CRs, not just long writing tasks
Hayes, J. R. (2012). Modeling and remodeling writing. Written communication, 29(3),

10. Core Cognitive Writing Processes
Evaluator
Proposer
Translator
Transcriber
Idea generation: draws upon
Rhetorical goals
Conceptual knowledge
Writing Schemas
Current plan
Feedback from collaborators/critics

11. Proposer => Generating Appropriate Ideas …
Fluent writers can generate ideas as they go along
Initial pauses for planning
Internal pauses to generate new ideas
Pauses for idea-generation mostly happen at natural breaks in the flow of thought

12. Core Cognitive Writing Processes
Evaluator
Proposer
Translator
Transcriber
Linguistic processing: driven by
Text structure
Sentence structure
Vocabulary knowledge

13. Translator => Finding the Right Words & Phrases
Fluent writers express their ideas in “bursts”
(Pause to decide on how to express ideas; then write out the words.)
Sentence planning difficulties lead to
shorter bursts
“false starts”

14. Core Cognitive Writing Processes
Evaluator
Proposer
Translator
Transcriber
Orthographic processing: driven by
Spelling knowledge
Handwriting or keyboarding skills

15. Transcriber => Spelling, keyboarding …
To be fluent, writers need to be able to type efficiently and automatically
Difficulties with spelling may lead to pauses before or within a word
Difficulties in keyboarding may lead to pauses between nearly any pair of characters, driven by the need to find the next key

16. Core Cognitive Writing Processes
Evaluator
Proposer
Translator
Transcriber
Evaluation of progress towards goals (may trigger editing/revision): depends on
Rereading text to date
Reviewing external resources such as task materials, written plans, feedback from collaborators and critics
Metacognitive and metalinguistic awareness

17. Evaluator => Interruptions to sequential text production
Quick edits to correct typos and spelling errors
Deletion of recently-produced text to change wording or phrasing
Moving around in the text
Local edits
New content inserted at multiple points
Rearrangement of content

18. Implications
Patterns in the keystroke log for a writing task should tell us about:
Motivation and engagement
Likely sources of difficulty
Strategies for task completion
However:
Different writing tasks impose different constraints on the writer

19. Predicting Writing Quality

20. Context: Direct Writing Assessment
Essays produced in single session
All work, including planning, drafting, revising/editing, must happen in a limited time
Puts a premium on writing fluency (students who are not fluent, will have trouble completing task in a reasonable time)

21. Data Source: ETS Research Studies
Several studies 2009 – 2014, totaling more than 10,000 students in U.S. schools, mostly grades 7-9
Part of an ETS research initiative, CBAL (Cognitively-Based Assessments of, for and as Learning)
Examined which features of keystroke logs are predictive of writing quality in this population

22. Results: In Single-Session Drafting, Stronger Writers …
Spend more time writing
Produce text in longer “bursts”
Produce fewer “false starts” where they have to delete text they already produced
Pause less between or within words
Type relatively quickly
Are more likely to notice and correct typos
Spend more time moving around in the text making edits or revisions
These are not the only features we could measure. I’ve experimented with many different feature sets. These have the advantage of simplicity and stability across prompts.

23. Factor Structure 1. Keyboarding Fluency 2. Phrasal/Chunk Level Editing
3. Local (Word-Internal) Editing
4. Planning and Deliberation
Stronger writers spend more time on planning and deliberation and phrasal/chunk level editing, and less time on local editing and keyboarding.

24. Using Writing Process Data to Support Instruction

25. General themes
Keystroke logs can help pinpoint what is behind high or low writing performance
In the classroom, it can help teachers determine how to approach student needs
At the school level, it can suggest priorities

29. At the school level, analysis of keystroke logs can highlight such issues as …
Low motivation
Difficulty generating ideas
Difficulty finding the right way to express ideas
Spelling difficulties
Keyboarding speed and accuracy
Willingness to revise, edit, or proofread

30. Discussion and Conclusions