1. Using Artificial Intelligence to Support Peer Review of Writing Diane Litman Department of Computer Science, Intelligent Systems Program, & Learning Research and Development Center

2. Context Speech and Language Processing for Education Learning Language (reading, writing, speaking) Using Language (to teach everything else) Tutors Scoring Readability Processing Language Tutorial Dialogue Systems / Peers CSCL Discourse Coding Lecture Retrieval Questioning & Answering Peer Review

3. Outline  SWoRD  Improving Review Quality  Identifying Helpful Reviews  Summary and Current Directions

4. SWoRD [Cho & Schunn, 2007]  Authors submit papers  Reviewers submit (anonymous) feedback  Authors revise and resubmit papers  Authors provide back-ratings to reviewers regarding feedback helpfulness

5. Some Weaknesses 1. Feedback is often not stated in effective ways 2. Feedback and papers often do not focus on core aspects

6. Our Approach: Detect and Scaffold 1. Detect and direct reviewer attention to key feedback features such as solutions 2. Detect and direct reviewer and author attention to thesis statements in papers and feedback Improving Learning from Peer Review with NLP and ITS Techniques (with Ashley, Schunn), LRDC internal grant

7. Feedback Features and Positive Writing Performance [Nelson & Schunn, 2008] Solutions Summarization Localization Understanding of the Problem Implementation

9. I. Detecting Key Feedback Features  Natural Language Processing (NLP) to extract attributes from text, e.g. –Regular expressions (e.g. “the section about”) –Domain lexicons (e.g. “federal”, “American”) –Syntax (e.g. demonstrative determiners) –Overlapping lexical windows (quotation identification)  Machine Learning (ML) to predict whether feedback contains localization and solutions, and whether papers contain a thesis statement

10. Learned Localization Model [Xiong, Litman & Schunn, 2010]

11. Quantitative Model Evaluation Feedback Feature Classroom Corpus NBaseline Accuracy Model Accuracy Model Kappa Human Kappa Localization History87553%78%.55.69 Psychology311175%85%.58.63 Solution History140561%79%.55.79 CogSci583167%85%.65.86

12. II. Predicting Feedback Helpfulness  Can expert helpfulness ratings be predicted from text? [Xiong & Litman, 2011a]  Impact of predicting student versus expert helpfulness ratings [Xiong & Litman, 2011b]

13. Results: Predicting Expert Ratings (average of writing and domain experts)  Techniques used in ranking product review helpfulness can be effectively adapted to peer-reviews (R =.6)  Structural attributes (e.g. review length, number of questions)  Lexical statistics  Meta-data (e.g. paper ratings)  However, the relative utility of such features varies  Peer-review features improve performance (R =.7)  Theory-motivated (e.g. localization)  Abstraction (e.g. lexical categories) better for small corpora

14. Changing the meaning of “helpfulness”  Helpfulness may be perceived differently by different types of people  Average of two experts (prior experiment)  Writing expert  Content expert  Student peers 14

15. Content versus Writing Experts –Writing-expert rating = 2 –Content-expert rating = 5 15 Your over all arguements were organized in some order but was unclear due to the lack of thesis in the paper. Inside each arguement, there was no order to the ideas presented, they went back and forth between ideas. There was good support to the arguements but yet some of it didnt not fit your arguement. First off, it seems that you have difficulty writing transitions between paragraphs. It seems that you end your paragraphs with the main idea of each paragraph. That being said, … (omit 173 words ) As a final comment, try to continually move your paper, that is, have in your mind a logical flow with every paragraph having a purpose. Writing-expert rating = 5 Content-expert rating = 2 Argumentation issue Transition issue

16. Results: Other Helpfulness Ratings  Generic features are more predictive for student ratings  Lexical features: transition cues, negation, suggestion words  Meta features: paper rating  Theory-supported features are more useful for experts  Both experts: solution  Writing expert: praise  Content expert: critiques, localization 16

17. Summary  Artificial Intelligence (NLP and ML) can be used to automatically detect desirable feedback features –localization, solution –feedback and reviewer levels  Techniques used to predict product review helpfulness can be effectively adapted to peer-review –Knowledge of peer-reviews increases performance –Helpfulness type influences feature utility 17

18. Current and Future Work  Extrinisic evaluation in SWoRD –Intelligent Scaffolding for Peer Reviews of Writing (with Ashley, Godley, Schunn), IES (recommended for funding)  Extend to reviews of argument diagrams –Teaching Writing and Argumentation with AI-Supported Diagramming and Peer Review (with Ashley, Schunn), NSF  Teacher dashboard –Keeping Instructors Well-informed in Computer-Supported Peer Review (with Ashley, Schunn, Wang), LRDC internal grant 18

19. Thank you! Questions? 19

20. Peer versus Product Reviews  Helpfulness is directly rated on a scale (rather than a function of binary votes)  Peer reviews frequently refer to the related papers  Helpfulness has a writing-specific semantics  Classroom corpora are typically small 20

21. Generic Linguistic Features typeLabelFeatures (#) StructuralSTR revLength, sentNum, question%, exclamationNum LexicalUGR, BGR tf-idf statistics of review unigrams (#= 2992) and bigrams (#= 23209) SyntacticSYN Noun%, Verb%, Adj/Adv%, 1stPVerb%, openClass% Semantic (adapted) TOPcounts of topic words (# = 288) ; posW, negW counts of positive (#= 1319) and negative sentiment words (#= 1752) Meta-data (adapted) METApaperRating, paperRatingDiff 21

22. TypeLabelFeatures (#) Cognitive Science cogS praise%, summary%, criticism%, plocalization%, solution% Lexical Categories LEX2Counts of 10 categories of words LocalizationLOC Features developed for identifying problem localization Specialized Features 22

23. Lexical Categories Extracted from: 1.Coding Manuals 2.Decision trees trained with Bag-of-Words 23 TagMeaningWord list SUGsuggestionshould, must, might, could, need, needs, maybe, try, revision, want LOClocationpage, paragraph, sentence ERRproblemerror, mistakes, typo, problem, difficulties, conclusion IDEidea verbconsider, mention LNKtransitionhowever, but NEGnegativefail, hard, difficult, bad, short, little, bit, poor, few, unclear, only, more POSpositivegreat, good, well, clearly, easily, effective, effectively, helpful, very SUMsummarizationmain, overall, also, how, job NOTnegationnot, doesn't, don't SOLsolutionrevision, specify, correction

24. Discussion 24 Effectiveness of generic features across domains Same best generic feature combination (STR+UGR+MET) But…

25. Results: Specialized Features 25 Introducing high level features does enhance the model’s performance.  Best model: Spearman correlation of 0.671 and Pearson correlation of 0.665. Feature Typerrsrs cogS 0.425+/-0.0940.461+/-0.072 LEX2 0.512+/-0.0130.495+/-0.102 LOC 0.446+/-0.1330.472+/-0.113 STR+MET+UGR (Baseline) 0.615+/-0.1010.609+/-0.098 STR+MET+LEX2 0.621+/-0.0960.611+/-0.088 STR+MET+LEX2+TOP 0.648+/-0.0970.655+/-0.081 STR+MET+LEX2+TOP+cogS 0.660+/-0.0930.655+/-0.081 STR+MET+LEX2+TOP+cogS+LOC 0.665+/-0.0890.671+/-0.076

26. Student rating = 3 Expert-average rating = 5 Students versus Experts 26 The author also has great logic in this paper. How can we consider the United States a great democracy when everyone is not treated equal. All of the main points were indeed supported in this piece. I thought there were some good opportunities to provide further data to strengthen your argument. For example the statement “These methods of intimidation, and the lack of military force offered by the government to stop the KKK, led to the rescinding of African American democracy.” Maybe here include data about how … (omit 126 words) praise Critique –Student rating = 7 –Expert-average rating = 2

27. Sample Result: All Features 27 Feature selection of all features Students are more influenced by meta features, demonstrative determiners, number of sentences, and negation words Experts are more influenced by review length and critiques Content expert values solutions, domain words, problem localization Writing expert values praise and summary