Course Transformation: Using an in-class activity to improve student writing Klaus Libertus January 2018

Goal of the Course Transformation Improve student writing in a w-course Decrease instruction on content Increase hands-on practice opportunities Introduce online lectures Allow students to review materials Study at own pace Offer more writing opportunities Writing has to be practiced More instructor feedback

Targeted Course Research Methods Lab Lab component to Research Methods RM is required for Psychology Major RM Lab is 40% of overall course grade Lab meets for 3h and 20 min/week Enrollment 150 – 240 students 9-10 sections with 24 students each

Goals of RM Lab Provide hands-on activities related to: Learn: Experimental design Data collection Data analysis Learn: Scientific writing Use APA formatting Research literature

Significant amount of time should focus on teaching writing Writing in RM Lab RM Lab is a designated w course “W-Courses are designed to teach writing within a discipline through writing assignments that are distributed across the entire term. Students should produce at least 20-24 pages of written work; a significant portion of this work should be substantially revised in response to instructor feedback and class discussion.” http://www.as.pitt.edu/fac/teaching/general-requirements Significant amount of time should focus on teaching writing

But there is one problem… THERE’S NO WRITING IN RM LAB At least not: During class With instructor feedback

What happens in RM Lab? Prior to transformation: Experimentation content Demos Lecture on basics Design of own Experiment Writing lectures APA lecture Content lectures Review lectures Two statistic reviews Too much “lab” time is lecture. Lectures, especially review of stats, are not effective: Majority of writing mistakes are related to APA format and Statistics How much time do these activities take up?

Use of class time prior transformation According to student perceptions (Spring 2016) 30% on writing? But what kind of writing activities do students mean?

Writing inside/outside RM Lab Writing lectures Basic “what goes where” Some in-class activities that invite brief drafts of paper sections Not graded, not collected Usually not completed by all Homework assignments Two papers 6 “busy work” assignments Simplified Idealized Unconnected Writing in class and at home: Did not form a clear sequence Was not related to actual scientific writing Little feedback on homework, no feedback on in-class writing

Course Transformation Approach Reduce lecture time Eliminate in-class review Create online review materials Provide immediate feedback with online mini-quizzes Allows students to review once or multiple times, as needed Replace assignments with in-class writing Writing sequence based on published research Early writing as low stakes assignments Put meaningful writing INTO class time

Transformation Initiated during Fall 2016

Course Transformation Fall 2016 class was divided into two syllabi Old 4 sections (about 24 students each) Revised 5 sections (about 24 students each) Most instructors were randomly assigned Two instructors came from Anthropology and randomly split (one in old and one in revised) Klaus Libertus was assigned Revised syllabus To facilitate content creation

Old vs. Revised Syllabus Week Topic 1 Introduction, data collection 2 APA Style Review 3 Stats Review 4 Library tutorial 5 Paper 1 analysis and outline 6 Paper 1 peer review 7 Stats Review 2 8 Paper 1 due 9 Study 2 Design 10 Data Collection 11 Data analysis 12 Paper 2 draft 13 Writing clinic 14 Paper 2 final Week Topic 1 Introduction, data collection 2 APA homework, In-class writing 1 3 Stats homework, In-class writing 2 4 Library tutorial 5 Paper 1 data analysis and outline 6 Paper 1 peer review 7 Stats homework 2, ICW 3 8 Paper 1 due, 9 Study 2 Design 10 Data collection 11 Data analysis 12 Paper 2 draft 13 Writing clinic 14 Paper 2 final 50% of sections used traditional syllabus 3 review lectures 50% of sections used revised syllabus 3 in-class writing activities IDENTICAL SECOND HALF OF CLASS

Review lectures  Online videos Old syllabus retained lectures from previous semesters Revised syllabus replaced APA formatting and Statistics lectures Online lectures with same content Provided to students via MyPitt Video (Panopto) Lecture content was identical in both cases For initial transformation only Spring 2018 class now uses additional videos from external sources (YouTube, Kahn Academy,…)

In-class writing sequence Revised syllabus included 3 in-class writing activities Time used for review lectures in old syllabus Sequence is based on published research article Open data article Students will “reverse-engineer” parts of the article Compare their own writing to published article in the end Goal: Engaging with real, concrete research increases student interest in the material

In-class writing sequence The In-class writing activity was: Open materials Open instructor ASK ME ANYTHING With outline/notes provided that students could COPY FROM No need to worry about Plagiarism Rationale: Fully scaffolded writing Experience writing process without need to first do background research Low stakes 12.5% of grade in initial transformation for all 3 assignments Two 5% assignments, one 2.5% assignment Stakes have been lowered for Spring 2018

Note on Assignment article Selected article: Rogers, T., & Feller, A. (2016). Discouraged by Peer Excellence: Exposure to Exemplary Peer Performance Causes Quitting. Psychological Science, 27(3), 365-374. doi:10.1177/0956797615623770 Article was NOT selected for its content Study 2 uses an online survey and conducts t-test and Chi-square tests Matches what is taught in Statistics review Data is open and available for download Class does include a peer review segment Implications of article for peer review were discussed Main message: “Don’t get discouraged” Not a main objective of this assignment

Evaluation of Transformation

Two evaluations of transformation Within semester Old vs. revised syllabus Fall 2016 class Different Instructors Different TFs Several grades By instructor themselves Included surveys assessing student perceptions Total of 9 sections 5 revised, 4 old Between semesters Old vs. revised syllabus Spring 2016 vs Spring 2017 Same instructor Klaus Libertus Rating of Paper 2 only By “blind” grader NOT instructor One section only

A) Within semester Measurable course goals Performance Attitudes Overall Lab grades Lab grades on Paper 1 Lab grades on Paper 2 Attitudes 3 online surveys (Pre, Mid, Post) Towards Psychology in general Experience of Lab Feeling prepared for writing Class time distribution

1) Performance Overall lab grades Grades on Paper assignments Paper 1 after revision Same topic for all sections Paper 2 Different topic in each section Note: Paper 2 is written in pairs

1) Performance No differences in overall grades No differences in Paper grades

1) Performance No differences between original and revised syllabus Not too surprising Different instructors Better approach Grade papers of same instructor from different years Have naïve grader Will be used in evaluation approach B

2) Attitudes Towards Psychology 10 questions Rated on 7 point Likert scale Strongly disagree – Strongly agree 5 positive, 5 negative items Derived one composite score

Results: Attitude towards Psychology Ratings were slightly higher in revised class Not statistically significant Most likely ceiling effect – everybody loves Psychology

2) Attitudes Lab Experience 5 questions Same 7 point Likert scale 3 positive, 2 negative items Derived one composite score

Results: Lab Experience * * Ratings significantly higher for Revised syllabus at Mid and Post Overall lab experience has improved with revised syllabus

2) Attitudes Preparation for writing Given before submission of Paper 1 and Paper 2 4 questions, all positive, composite score

Results: Preparation for writing † * Students felt more prepared for writing using revised syllabus Effect reduced for POST (p = .078)

2) Attitudes Distribution of class time Student perception on how much time was devoted on different aspects of class

Results: Distribution of class time Students felt that more time was spent on writing Strong increase, 10% more time Same pattern when comparted to Spring 2016

B) Between semesters Measurable course goals Performance Lab grades on Paper 2 Rated by blind coder Grading dimensions Scientific writing style (appropriate for Psychology) Logic of argumentation (esp. Intro and Discussion) Clarity of writing Overall quality of the paper

Continuous ratings

Results: Paper 2 performance Statistically, no significant differences between groups (note: small N, only 32 papers). Patterns suggest improvement.

Summary Comparison of old and revised syllabus reveals: Increase in instruction on writing Improvement to lab experience for students Stronger feeling of preparedness Improvements in writing clarity and quality

Continue to use revised syllabus Reduce stakes for in-class writing Improve quality of online lectures/videos Increased use of outside sources

Thank you for your attention and feedback Special thanks to Beth Matway who helped develop the writing sequence Special thanks to Carol Washburn for helping to design the revised syllabus Special thanks to Barb Kucinski and Ben Rottman for their comments and suggestions on this transformation Thank you for your attention and feedback