Download presentation
Presentation is loading. Please wait.
Published byOswald Poole Modified over 8 years ago
1
1 Evaluation of Opinion Questions ä Session leaders: Ed Hovy, Kathy McKeown ä Topics ä Is evaluating opinion questions feasible at all? How can we construct a reliable evaluation? ä What are a subset of opinion questions that could be handled? ä What are possible methods for evaluating results? ä What should the form of the response be?
2
2 Is evaluating opinions feasible? ä Preliminary results indicate yes ä Necessary: ä Multiple human judgments on a response ä Evaluation must be computed against multiple models ä Measure agreement between models ä Models of opinions and arguments need to inform evaluation design
3
3 What subset of opinion questions should be addressed? ä Some characteristics ä Concrete statement, specific terms ä Is European criticism of President Bush and United States foreign policy justified? ä Question can be marked as seeking opinion ä (justified, think, opinion, success) ä Decided: No need for question analysis or for determining if the question is an opinion question ä Provide a fact, topic and find all opinions related to that fact ä Closer to what analysts do ä Avoids problem of disambiguating questions
4
4 Measures for scoring responses Form of the response ä Input: proposition/topic ä Task: find the opinions and cluster them ä Expect that each cluster will represent a viewpoint (e.g., pro or con) ä Answer represents an objective view of multiple opinions from different sources, not a generated system opinion ä Form of answer: list of items/sentences with links to source documents ä Metrics used to score answer: ä number and validity of clusters, ä relevance of each item to input topic, ä lack of redundancy, ä Is the item an opinion? ä Subjective: ä Formative user studies ä Think-aloud protocols
5
5 Discussed but left for the future ä Measure level of support for a viewpoint ä Label clusters (e.g., pro, con) ä Distinguish between opinions and facts ä Mixed response from analyst on need ä Provide justification of opinions by linking to supporting facts ä Analysts did not seem to feel a high priority ä Distinguish public and privately held opinions ä Identify deceptions ä What overt behavior results from opinions?
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.