1. Teaching Peer Review of Writing in a Large First-Year Electrical and Computer Engineering Class: Comparison of Two Methods Michael Ekoniak Molly Scanlon Marie Paretti M. Jean Mohammadi-Aragh

2. Background Engineers must be adept at written communication Long history – over 100 years “In every case knowledge of a book will be regarded as less important than the ability to write good English” [ASEE President, 1901] A contemporary problem Writing across the Curriculum ABET Engineering Criteria 2000 Engineer of 2020

3. What’s Happening Now? Growing body of work on integration of writing within engineering courses. Mostly descriptive, not experimental Issues facing integration of writing and engineering courses: Some faculty reluctant to incorporate writing Large time commitment Uncomfortable with their abilities Often incorporated in an ineffective or less than ideal way Lack of appropriate feedback and revision Feedback and revision is extremely important

4. What Can We Do? Problems: Large classes (especially in the first and second years) High student-faculty ratios Potential Solution: Peer Review Can it work? Can peer feedback be as useful to students as instructor feedback? Recent research says yes… if it’s from multiple peers But how can we teach students to do it?

5. Purpose of Study The purpose of this study is to examine the results of two types of peer review instruction in a first-year electrical and computer engineering course. In two sections of the course, a writing instructor provided in-class training on peer review techniques through a short lesson, workshop, and instructional handout; in three additional sections students were provided the handout but no in-class instruction. Because not all engineering instructors who introduce writing assignments with peer review into their courses will have the time or institutional support to have a writing instructor provide in- class training, this research aims to compare the peer review results achieved in the two groups by answering the following research question:

6. Research Question Are peer review comments qualitatively or quantitatively different between the in-class instruction and handout groups?

7. Context Large public research university. First-year Engineering (FYE) course for students majoring in Electrical Engineering, Computer Engineering, and Computer Science A Contemporary Issue Report (CIR) assignment students write a research paper in which they explore a contemporary issue in ECE

8. Contemporary Issue Report Assignment sheet Description of assignment Example report (on Cochlear Implants) MS Word Template Rubric One and Done Feedback and revision

9. Intervention Students received one of two types of instruction: 1.In-class workshop plus handout, or 2.Handout only Collaboration between course instructors and an experienced writing teacher. The in-class workshop was conducted by the writing teacher. Common Writing Across the Curriculum (WAC) approach Handout only is less resource intensive… can we use it? (RQ1)

10. Handout 1.Guidelines for giving quality peer review comments 2.Example annotated report 3.Evaluation Rubric

11. Categorizing Comments on Writing Cho, K., Schunn, C. D., & Charney, D. (2006) [17] Characterization of peer feedback comments in an undergraduate psychology research methods course Too Limited Straub & Lunsford (1996) [18] Characterization of feedback given by expert writing teachers Smith Taylor (2003; 2011); Smith Taylor & Patton (2006) [19]-[21] Characterization of feedback given by engineering instructors

12. Coding Strategy Each comment was coded along three axes: The focus of a comment identifies what the comment refers to in the original text Mode and tone refer to the qualities of the comment itself. Coding each comment along these axes—rather than a coding scheme requiring each comment to fall into only one category— allowed us to capture finer variation in student responses during analysis.

13. Analysis 40 drafts from each treatment group (80 documents total) Coded using MaxQDA 1048 individual comments Mean number of comments in each analytical category calculated Overlap of axes also considered T-test for statistical significance

14. Results Drafts from the in-class instruction group included slightly fewer mean comments overall. HandoutIn-Class # Comments Overall13.4512.75 FocusHandoutIn-Class Edit4.232.98 Form3.282.50 Content6.307.18 Mean number of content-focused comments was higher, while there were fewer form-focused comments.

15. Results There were more positive and “sandwich” comments and fewer negative/neutral. ToneHandoutIn-Class Positive2.203.23 Sandwich0.500.88 Negative/Neutral6.555.65

16. Results The mode of those comments included fewer edits and authoritative comments and more coaching and readerly comments. ModeHandoutIn-Class Edit4.232.98 Authoritative6.686.23 Coaching1.751.93 Readerly0.831.63

17. Discussion The direction of these results is consistent with the in-class instruction group producing comments that are of a higher quality based on both the criteria from the peer review instruction and the three axes identified from the literature for the analytical framework. But… The difference between groups was not statistically significant for most of the criteria. Only mean number of readerly comments had a statistically significant difference between in-class instruction and handout groups

18. Conclusions The results of comment analysis suggest that using a handout for peer feedback instruction may be as effective as conducting an in- class workshop, as the differences between groups was not statistically significant for most evaluation criteria. However, the criterion that was significantly different between the groups, mean number of readerly comments, is noted in the literature as being particularly important for helping students make revisions (Smith Taylor, 2011). Moreover, the direction of differences in means in other categories, especially when taken as whole, suggests that in-class peer feedback instruction may have an effect even though it was not a statistically significant effect in this study.

19. Conclusions The major limitation of this study that will be addressed in future work is that the actual revisions students made based on the peer feedback received was not considered. As the literature indicates that the mode and focus of comments is important in the revision process, analysis of actual revisions in the context of the comment analysis could strengthen the conclusion that a well developed handout can be as effective as in-class instruction or might show that the lack of sufficient readerly comments does make an important difference in final drafts.

20. Recommendations for Instructors Find ways to integrate written communication into your classes Always provide opportunities for feedback and revision Use multiple-peer feedback in addition to or in place of instructor- only feedback Train students on giving quality feedback But you probably don’t have to use class time to do it! Work with a writing expert to develop materials if possible

21. Questions?

23. Coding Strategy: Focus

24. Coding Strategy: Mode

25. Mode: It’s all about agency and control 1.“Put this in the introduction” 2.“I think you should put this in the introduction” 3.“You might consider putting this in the introduction” 4.“Putting this in the introduction will allow the reader to more easily understand the rest of the paper.”

26. Mode Examples Edit Authoritative

27. Coaching Readerly

28. Coding Strategy: Tone Positive Negative/Neutral “Sandwich”