1. Eliciting Effective Peer Feedback Edward F. Gehringer Department of Computer Science North Carolina State University The Peerlogic project has been funded by the National Science Foundation Our work on resources for teaching peer review are funded by a Google Research Award

2. Outline  Why use peer review?  Rubrics  Formative vs. summative  What to say in a review  How to say it  How to read a review  Learning from reviewing Gehringer, Eliciting effective peer feedback efg@ncsu.edu

3. Advantages of peer review? Gehringer, Eliciting effective peer feedback efg@ncsu.edu

4. Some advantages of peer review  Feedback is more extensive quicker scalable  Can’t blame the reader!  Forces students to think metacognitively Gehringer, Eliciting effective peer feedback efg@ncsu.edu

5. Outline  Why use peer review?  Rubrics  Formative vs. summative  What to say in a review  How to say it  How to read a review  Learning from reviewing Gehringer, Eliciting effective peer feedback efg@ncsu.edu

6. Rubrics  Why use a rubric? Tell students what to look for “Fairness” in assessment  Students can help create the rubric  How detailed? Gehringer, Eliciting effective peer feedback efg@ncsu.edu

7. Example rubric—Wikipedia article  Ratings Rate the overall readability of the article. Explain why you give this score. List any related terms or concepts for which the writer failed to give adequate citations and links. Rate the helpfulness of the citations. Rate how logical and clear the organization is. Point out any places where you think that the organization of this article needs to be improved. Randomly pick some sentences or paragraphs and search for it with a search engine. List any sources that may infringe copyrights. Gehringer, Eliciting effective peer feedback efg@ncsu.edu

8. Example rubric—Wikipedia article  Checklist The discussion is appropriate and reasonable, i.e., not too easy or too difficult for your peers to follow. Most of the sources are current (less than 5 years old). This article is up to date. Taken together the sources represent a good balance of potential references for this topic Gehringer, Eliciting effective peer feedback efg@ncsu.edu

9. Example rubric—Wikipedia article  Short response Please make a comment about the sources. Explain how the author can improve the use of sources in the article. What other sources or perspectives might the author want to consider? Give compliments for the article. Separate them with line breaks. Give suggestions for the article. Separate them with line breaks. Gehringer, Eliciting effective peer feedback efg@ncsu.edu

10. Rubric advice

11. Outline  Why use peer review?  Rubrics  Formative vs. summative  What to say in a review  How to say it  How to read a review  Learning from reviewing Gehringer, Eliciting effective peer feedback efg@ncsu.edu

12. Formative vs. summative peer review  Formative—text feedback  Summative—Likert scale  Should peer review be used summatively? Gehringer, Eliciting effective peer feedback efg@ncsu.edu

13. Formative feedback  More useful for student Consists chiefly of text Authentic audience reaction Get help in revision Students need plenty of it Gehringer, Eliciting effective peer feedback efg@ncsu.edu

14. Summative assessment  More useful for instructor Consists primarily of numbers Directs instructor’s attention to weaknesses Can use related text comments to decide whether to factor into grade Gehringer, Eliciting effective peer feedback efg@ncsu.edu

15. Summative rubric  Originality: If you found any plagiarism in round 1, has it been removed? Then, randomly pick some sentences or paragraphs and search for them with a search engine. Describe any text that may infringe copyrights.  Coverage: does the artifact cover all the important aspects that readers need to know about this topic? Are all the aspects discussed at about the same level of detail?  References: do the major concepts have citations to more detailed treatments? Are any links unavailable?  Did the authors revise their work in accordance with your suggestions? Gehringer, Eliciting effective peer feedback efg@ncsu.edu

16. Peer grading—how reliable?  Two studies on Coursera MOOC [2013] Piech et al.: ≥ 26% of grades ± 5% from “ground truth.” Kulkarni et al.: 40% of grades off by 1 letter grade! But …  very little quality contol  this was, after all, a MOOC Gehringer, Eliciting effective peer feedback efg@ncsu.edu

17. Outline  Why use peer review?  Rubrics  Formative vs. summative  What to say in a review  How to say it  How to read a review  Learning from reviewing Gehringer, Eliciting effective peer feedback efg@ncsu.edu

18. Five types of feedback  Describe what you see in the work, e.g., by summarizing.  Clarify what is being said, e.g., ask qq.  Alter or point out corrections to be made  Suggest improvements  Evaluate how well it meets its objectives Gehringer, Eliciting effective peer feedback efg@ncsu.edu

19. What errors do students make in reviews? Gehringer, Eliciting effective peer feedback efg@ncsu.edu

20. Helpful vs. unhelpful feedback Helpful feedback is Constructive Specific Balanced Succinct Respectful vs. Unhelpful feedback is:  Too positive or too negative  General & unspecific  Rambling  Aggressive – makes author feel “attacked” “Helpful vs. unhelpful” slides taken from Pearce & Mulder, “Student peer review: an introductory tutorial.”Student peer review: an introductory tutorial.

21. 1. What are the main strengths of this report?  Unhelpful comment: “Your report was really good! I enjoyed reading it.” Author’s response: “I’m flattered you liked my report, but I don’t have a sense of what you thought was good about it.” Helpful comment: “This report was succinct and well written. The aims of the report were clear and I found it easy to identify your take- home messages...” Helpful vs. unhelpful feedback Gehringer, Eliciting effective peer feedback efg@ncsu.edu

22. Helpful vs. unhelpful feedback 2. Where are the main areas for improvement?  Unhelpful comment: “Your report was poorly written and hard to read!” Author’s response: “This comment doesn’t really help me to improve anything!” Helpful comment: “There are a few areas that might make this report stronger. Expanding the Introduction to include more background information would help set the scene a little more (para 2). The arguments could also be strengthened by adding additional references, for examples lines 3, 16 and 55...”

23. Helpful vs. unhelpful feedback 3a. Is the balance between the sections about right?  Unhelpful comment: “No – there wasn’t enough space left for covering the background of the study.” Helpful comment: “The balance feels very good; however you may consider expanding the background section with greater information on theoretical concepts being tested” Author’s response: “Although stating good and bad points, none of it was portrayed negatively. The comments were given helpfully, with clear points for me to follow.”

24. Helpful vs. unhelpful feedback 3b. Is the balance between the sections about right?  Unhelpful comment: “The overall balance was good, with no section outweighing any other at all.” Author’s response: “Very positive review, but not much given that I can improve on - I highly doubt it was almost perfect.” Helpful comment: “Not the best balance: The introduction and rationale sections were too lengthy. While very clear, they could be trimmed down quite a bit and made to be much more concise. For example, I think lines 108 to 113 are unnecessary...”

25. “I like the writing style, and I think the article is relatively easy to follow and the paragraphs are well linked. The article might be stronger if some of the sentences were more simple and succinct such as line 1 and 7 in paragraph 1, and line 3 in paragraph 4.” Example review 1 Balanced? Constructive? Specific? Clear? Gehringer, Eliciting effective peer feedback efg@ncsu.edu “Helpful vs. unhelpful” slides taken from Pearce & Mulder, “Student peer review: an introductory tutorial.”Student peer review: an introductory tutorial.

26. Example review 2 “This paper has poor structure and flow. There are several grammatical and spelling errors and some of the paragraphs should be shortened. I got confused about what you were trying to say at some points.” Balanced? Constructive? Specific? Clear? Gehringer, Eliciting effective peer feedback efg@ncsu.edu “Helpful vs. unhelpful” slides taken from Pearce & Mulder, “Student peer review: an introductory tutorial.”Student peer review: an introductory tutorial.

27. “Some sentences lacked commas where there should have been one, or were too long at times (e.g. line 34 and line 41). Otherwise, the article as a whole had a smooth flow and the intent behind each paragraph clear and understandable.” Example review 3 Balanced? Constructive? Specific? Clear? Gehringer, Eliciting effective peer feedback efg@ncsu.edu “Helpful vs. unhelpful” slides taken from Pearce & Mulder, “Student peer review: an introductory tutorial.”Student peer review: an introductory tutorial.

28. Read a work twice before reviewing  Once for the big picture  Once for details  Take notes! Gehringer, Eliciting effective peer feedback efg@ncsu.edu

29. Outline  Why use peer review?  Rubrics  Formative vs. summative  What to say in a review  How to say it  How to read a review  Learning from reviewing Gehringer, Eliciting effective peer feedback efg@ncsu.edu

30. How to say it  Psychology is as important as assessment  Proceed from general to specific  Suggestions should be realistic in scope  Review the content, not the author  Avoid “always,” “never,” etc.  Start and end with positive Gehringer, Eliciting effective peer feedback efg@ncsu.edu

31. Outline  Why use peer review?  Rubrics  Formative vs. summative  What to say in a review  How to say it  How to read a review  Learning from reviewing Gehringer, Eliciting effective peer feedback efg@ncsu.edu

32. How to “receive” a review  Realize that reviews will vary in quality.  Make an effort to digest comments. Even if they seem “off the wall,” they could still be how your readers see the work.  Be grateful for others’ help.  Ask for clarifications. Gehringer, Eliciting effective peer feedback efg@ncsu.edu

33. Receiving feedback: tips Don’t panic! Read all the comments & make notes Take time to reflect Address major issues Tackle smaller points Proofread final document This slide taken from Pearce & Mulder, “Student peer review: an introductory tutorial.”Student peer review: an introductory tutorial.

34. Outline  Why use peer review?  Rubrics  Formative vs. summative  What to say in a review  How to say it  How to read a review  Learning from reviewing Gehringer, Eliciting effective peer feedback efg@ncsu.edu

35. Is it better to give than to receive?  Lundstrom & Baker 2009 Class divided into 2 groups Group A provided feedback to Group B Students in Group A improved more than students in Group B.  Shah-Nelson 2014 Survey answered by 1040 students in MOOCs Educational value of  “grading others’ papers” was 3.17 on a scale of 1 to 4  “feedback from other students” was only 2.92 on scale. Gehringer, Eliciting effective peer feedback efg@ncsu.edu

36. Summary  Why use peer review?  Rubrics  Formative vs. summative  What to say in a review  How to say it  How to read a review  Learning from reviewing Gehringer, Eliciting effective peer feedback efg@ncsu.edu