Paul Deane National Conference on Student Assessment, June 27-29, 2016 Studying the Writing Process: Exploring the Use of Writing Process Logs to Support Instruction Paul Deane National Conference on Student Assessment, June 27-29, 2016

What is a Writing Process Log? A record of what happens while students write, from which we can Reconstruct what the writer’s text looked like at every step of the process Capture the timing of each event Analysis of writing process logs can provide important evidence about what is happening while students write

Why Writing Process Logs Matter The sequence and timing of actions reflects the thought processes that led to the final response Can support richer inferences (and more useful feedback) in the classroom Also provides important validity evidence for writing assessment

Writing Process Logs Capture Information That is Invisible in the Final Written Product What did the writer edit or revise? Where did the writer have to stop and think? How much time and effort did the writer put into different parts of the writing task?

Writing Process Log Analysis: A New Field of Study Builds on extensive research by European scholars Leijten, M., & Van Waes, L. (2013). Keystroke logging in writing research: Using Inputlog to analyze and visualize writing processes. Written Communication, 30, 358-392 Applications to educational assessment developed at ETS (Almond et al., 2012; Deane & Quinlan, 2010; Deane, Quinlan, & Kostin, 2011; Deane et al., 2014; Deane & Zhang, 2015; Zhang & Deane, 2015) NAEP 2017 reporting for writing will include writing process data

Writing Process Log Replay Demo

Plan for the Rest of the Talk Background: the cognitive processes involved in writing Process features that distinguish stronger from weaker writers How writing process logs can support instruction Discussion

The Cognitive Processes Involved in Writing

Core Cognitive Writing Processes Evaluator Proposer Translator Transcriber This process is relevant to all CRs, not just long writing tasks Hayes, J. R. (2012). Modeling and remodeling writing. Written communication, 29(3), 369-388.

Core Cognitive Writing Processes Evaluator Proposer Translator Transcriber Idea generation: draws upon Rhetorical goals Conceptual knowledge Writing Schemas Current plan Feedback from collaborators/critics

Proposer => Generating Appropriate Ideas … Fluent writers can generate ideas as they go along Initial pauses for planning Internal pauses to generate new ideas Pauses for idea-generation mostly happen at natural breaks in the flow of thought

Core Cognitive Writing Processes Evaluator Proposer Translator Transcriber Linguistic processing: driven by Text structure Sentence structure Vocabulary knowledge

Translator => Finding the Right Words & Phrases Fluent writers express their ideas in “bursts” (Pause to decide on how to express ideas; then write out the words.) Sentence planning difficulties lead to shorter bursts “false starts”

Core Cognitive Writing Processes Evaluator Proposer Translator Transcriber Orthographic processing: driven by Spelling knowledge Handwriting or keyboarding skills

Transcriber => Spelling, keyboarding … To be fluent, writers need to be able to type efficiently and automatically Difficulties with spelling may lead to pauses before or within a word Difficulties in keyboarding may lead to pauses between nearly any pair of characters, driven by the need to find the next key

Core Cognitive Writing Processes Evaluator Proposer Translator Transcriber Evaluation of progress towards goals (may trigger editing/revision): depends on Rereading text to date Reviewing external resources such as task materials, written plans, feedback from collaborators and critics Metacognitive and metalinguistic awareness

Evaluator => Interruptions to sequential text production Quick edits to correct typos and spelling errors Deletion of recently-produced text to change wording or phrasing Moving around in the text Local edits New content inserted at multiple points Rearrangement of content

Implications Patterns in the keystroke log for a writing task should tell us about: Motivation and engagement Likely sources of difficulty Strategies for task completion However: Different writing tasks impose different constraints on the writer

Predicting Writing Quality

Context: Direct Writing Assessment Essays produced in single session All work, including planning, drafting, revising/editing, must happen in a limited time Puts a premium on writing fluency (students who are not fluent, will have trouble completing task in a reasonable time)

Data Source: ETS Research Studies Several studies 2009 – 2014, totaling more than 10,000 students in U.S. schools, mostly grades 7-9 Part of an ETS research initiative, CBAL (Cognitively-Based Assessments of, for and as Learning) Examined which features of keystroke logs are predictive of writing quality in this population

Results: In Single-Session Drafting, Stronger Writers … Spend more time writing Produce text in longer “bursts” Produce fewer “false starts” where they have to delete text they already produced Pause less between or within words Type relatively quickly Are more likely to notice and correct typos Spend more time moving around in the text making edits or revisions These are not the only features we could measure. I’ve experimented with many different feature sets. These have the advantage of simplicity and stability across prompts.

Factor Structure 1. Keyboarding Fluency 2. Phrasal/Chunk Level Editing 3. Local (Word-Internal) Editing 4. Planning and Deliberation Stronger writers spend more time on planning and deliberation and phrasal/chunk level editing, and less time on local editing and keyboarding.

Using Writing Process Data to Support Instruction

General themes Keystroke logs can help pinpoint what is behind high or low writing performance In the classroom, it can help teachers determine how to approach student needs At the school level, it can suggest priorities

At the school level, analysis of keystroke logs can highlight such issues as … Low motivation Difficulty generating ideas Difficulty finding the right way to express ideas Spelling difficulties Keyboarding speed and accuracy Willingness to revise, edit, or proofread

Discussion and Conclusions