1. Improving Learning from Peer Review with NLP and ITS Techniques (July 2009 – June 2011) Kevin Ashley Diane Litman Chris Schunn

2. Thank You for the Support!  New interdisciplinary research group  Research outcomes –Refereed publications –Pending IES and NSF proposals  Technology development –New version of SWoRD –“Intelligent” scaffolding components

3. Outline  SWoRD  Intelligent Scaffolding for Reviewers and Authors  AI-supported Argument Diagramming  Summary

4. SWoRD [Cho & Schunn, 2007]  Authors submit papers  Reviewers submit (anonymous) feedback  Authors revise and resubmit papers  Authors provide back-ratings to reviewers regarding feedback helpfulness

5. SWoRD Rebuild  SWoRD 3.5@LRDC dying, v4.0@Missouri struggling  New SWoRD v5.0@LRDC –Rebuilt from scratch (more stable, expandable) –More instructional flexibility »# and type of rating dimensions, reviewing dimensions, # of drafts, grading options, … –Better instructor oversight of students »Missing papers & reviews, high conflict reviews, inter- rater accuracy, … –Better research support »Can directly download ‘research’ data

6. SWoRD 5.0 Users  Active classes in Spring 2011: 39  Users in Spring 2011: ~2000  TOTAL User accounts: ~3900  Countries: –USA –Canada –United Kingdom –Netherlands –Estonia –Hungary –Turkey –China  Disciplines: –Psychology –Astronomy & Physics –Computer Science –Biology –Economics –Engineering –Speech-Language Pathology –English & Rhetoric –Philosophy –Women's Health  Levels: –University –High School –Middle School

7. Some Remaining Weaknesses 1. Feedback is often not stated in effective ways 2. Feedback and papers often do not focus on core aspects

8. Feedback Features and Positive Writing Performance [Nelson & Schunn, 2008] Solutions Summarization Localization Understanding of the Problem Implementation

9. Our Approach: Detect and Scaffold 1. Detect and direct reviewer attention to key feedback features such as solutions 2. Detect and direct reviewer and author attention to thesis statements in papers and feedback

11. Detecting Key Features of Text Using Educational Data Mining  Natural Language Processing (NLP) to extract attributes from text, e.g. –Regular expressions (e.g. “the section about”) –Domain lexicons (e.g. “federal”, “American”) –Syntax (e.g. demonstrative determiners) –Overlapping lexical windows (quotation identification)  Machine Learning (ML) to predict whether feedback contains localization and solutions, and whether papers contain a thesis statement

12. Learned Localization Model [Xiong, Litman & Schunn, 2010]

13. Quantitative Model Evaluation (10 fold cross-validation) Feedback Feature Classroom Corpus NBaseline Accuracy Model Accuracy Model Kappa Human Kappa Localization History87553%78%.55.69 Psychology311175%85%.58.63 Solution History140561%79%.55.79 CogSci583167%85%.65.86

14. Predicting Feedback Helpfulness [Xiong & Litman, under review]  Recall that SWoRD supports numerical back ratings of feedback helpfulness –My concerns come from some of the claims that are put forth. Page 2 says that the 13 th amendment ended the war. Is this true? Was there no more fighting or problems once this amendment was added? … (rating 5) –Your paper and its main points are easy to find and to follow. (rating 1)

15. Predicting Expert Ratings (Average of Writing and Domain Experts)  Structural attributes (e.g. review length, number of questions), lexical statistics, and meta-data (e.g. paper ratings) developed for product reviews (e.g. Amazon) are also useful for peer feedback  Features specialized for peer-review (e.g. localization) can further improve performance  Current work: student helpfulness ratings

16. The Problem

17. Students unable to synthesize what the sources say…

18. The Problem Students unable to synthesize what the sources say… … or to apply them in solving the problem.

19. Our Solution Source texts Author creates Argumen t Diagram Peers review Argumen t Diagram s Author revises Argumen t Diagram Author writes paper Peers review papers Author revises paper AI: Guides preparing diagram and using it in writing AI: Guides reviewing

20. Argument diagram student created with LASAD 1 · HypothesisLink: 1 If: Participants are assigned to the active condition Then: they will be better at correctly identifying stimuli than participants in the passive condition. 2 · HypothesisLink: 2 If: The participant has small hands Then: they will be better at recognizing objects than regardless of what condition they’re in.. 9 · (+) supportsLink: 1 Active touch participants were able to more accurately identify objects because they had the use of sensitive fingertips in exploring the objects 7 · (+) supportsLink: 1 Active touch is more effective than passive touch 11 · (+) supportsLink: 2 Active touch improved through the development levels but passive touch stayed the same (hand size may play role) 20 · (+) supportsLink: 2 Sensory perceptors in smaller hands are closer together, allowing for more accurate object acuity 8 · CitationLink: 1 (Craig 2001) 6 · CitationLink: 1 (Gibson 1962) 10 · CitationLink: 2 (Cronin 1977) 17 · CitationLink: 2 (Peters 2009)

21. LASAD analyzes diagrams  With even small set of types of argument nodes and relations and of constraint-defining rules…  Even simple argument diagrams provide pedagogical information that can be automatically analyzed. E.g., has student: –Addressed all sources and hypotheses? (No) –Indicated that citations support claims/hypotheses? (Not vice versa as here) –Related all sources and hypotheses under single claim? (No) –Related some citations to more than one hypothesis? (No interactions here) –Included oppositional relations as well as supports? (No) –Avoided isolated citations? (Yes) –Avoided disjoint sub-arguments? (No)

22. Prototype SWoRD Interface for feedback to reviewer pre-review submission Claims or reasons are unconnected to the research question or hypothesis. Lippman, 2010 is not organized around a hypothesis. Siler 2009 is more focused on the response to the task not focused on the actual type of task which is what the hypothesis for the effect of IV2. Doesn’t support the research question. H2 needs reasoning to connect prior research with the hypothesis, e.g. “because multi-step algebra problems are perceived as more difficult, people are more likely to fail in solving them.” Support 2 is weak because it’s basically citing a study as the reason itself. Instead, it should be a general claim, that uses Jones, 2007 to back it up. Lippman, 2010 is free floating and needs to be linked to either the research question or a hypothesis. Say where these issues happen! (like the green text in other comments) Say where these issues happen! (like the green text in other comments) Suggest how to fix these problems! (like the blue text in other comments) Suggest how to fix these problems! (like the blue text in other comments) = Localization hints X X = Solution hints X X

23. Prototype tool to translate student argument diagrams into text A Translation of Your Argument Diagram (click to edit) Next Steps A Translation of Your Argument Diagram (click to edit) Next Steps The first hypothesis is, “If participants are assigned to the active condition, then they will be better at correctly identifying stimuli than participants in the passive condition.” This hypothesis is supported by (Craig 2001) where it was found that “Active touch participants were able to more accurately identify objects because they had the use of sensitive fingertips in exploring the objects.” The hypothesis is also supported by (Gibson 1962) where … The second hypothesis is, … 1 2 Export text Quit Save progress Possible things to improve your argument: Add a missing citation Add third hypothesis Indicate which hypothesis is an interaction hypothesis and specifying an interaction variable(s) Relate one or more hypotheses along with their supporting sources under a single sub claim Include any oppositional relations between citations and a hypothesis Relate the disjointed subarguments concerning the hypotheses under one overall argument Possible things to improve your argument: Add a missing citation Add third hypothesis Indicate which hypothesis is an interaction hypothesis and specifying an interaction variable(s) Relate one or more hypotheses along with their supporting sources under a single sub claim Include any oppositional relations between citations and a hypothesis Relate the disjointed subarguments concerning the hypotheses under one overall argument