AQUAINT Proposal for a Pilot Study on Extended Definitions: Who is Colin Powell? What is naproxen? Ralph Weischedel & Dan Moldovan 13 June 2002.

Slides:

Advertisements

Similar presentations

Paper II Skills Question type 1– Evaluating sources.

Advertisements

By Jerry Stallworth InTech 3677 February 25, 2003.

Towards Methods for the Collective Gathering and Quality Control of Relevance Assessments SIGIR´09, July 2009.

WHAT IS AN ASSESSMENT CENTER? NOT A PLACE TO TAKE A TEST A TESTING PROCESS CANDIDATES PARTICIPATE IN A SERIES OF SYSTEMATIC, JOB RELATED, REAL-LIFE SITUATIONS.

INPO Update CMBG Meeting June 2013

NYU ANLP-00 1 Automatic Discovery of Scenario-Level Patterns for Information Extraction Roman Yangarber Ralph Grishman Pasi Tapanainen Silja Huttunen.

Beyond Right and Wrong Using Open-Ended Questioning in the Mathematics Classroom Jeff Mahood, Colegio Bolívar.

Next slide End Show Thank You A diverse group of stakeholders: the school’s principal and parents, teachers, support staff, students*, and community.

3.2. Applying the Models The Leader Model The Group Model The Rational Actor Model The Bureaucratic Politics Model.

Question Answering using Language Modeling Some workshop-level thoughts.

Action Implementation and Evaluation Planning Whist the intervention plan describes how the population nutrition problem for a particular target group.

Reference Collections: Task Characteristics. TREC Collection Text REtrieval Conference (TREC) –sponsored by NIST and DARPA (1992-?) Comparing approaches.

Requirements Analysis Concepts & Principles

About Waterloo website Project report June Outline Overview of process Project deliverables Lessons learned.

New Advanced Higher Subject Implementation Events Design and Manufacture: Advanced Higher Course Assessment.

The Main Idea The powers and roles of the U.S. president affect not only the citizens of the United States but also people throughout the world. Reading.

Probabilistic Model for Definitional Question Answering Kyoung-Soo Han, Young-In Song, and Hae-Chang Rim Korea University SIGIR 2006.

SOWO 874: DIMENSIONS OF HUMAN SERVIVE MANAGEMENT Lecture I Walter C. Farrell, Jr., Professor Management and Community Practice School of Social Work University.

Research Project Thesis Statement. Thesis: What is it? The thesis is the controlling idea of the paper. The thesis for this project will be more than.

A Proposed Risk Management Regulatory Framework Commissioner George Apostolakis Presented at the Organization of Agreement States 2012 Annual Meeting Milwaukee,

Minimal Test Collections for Retrieval Evaluation B. Carterette, J. Allan, R. Sitaraman University of Massachusetts Amherst SIGIR2006.

Understanding the DBQ Guidelines Do I understand the guidelines of the task? Based on materials made by the GH - Adapted by US History Pilot.

Proposal Development Sample Proposal Format Mahmoud K. El -Jafari College of Business and Economics Al-Quds University – Jerusalem April 11,2007.

Scott Duvall, Brett South, Stéphane Meystre A Hands-on Introduction to Natural Language Processing in Healthcare Annotation as a Central Task for Development.

Organizing Your Information

Finish paragraph with a clear thesis statement that establishes the purpose of the essay. Example: "Thus, the Civil War did, in fact, represent a political,

Answering Definition Questions Using Multiple Knowledge Sources Wesley Hildebrandt, Boris Katz, and Jimmy Lin MIT Computer Science and Artificial Intelligence.

Multilingual Relevant Sentence Detection Using Reference Corpus Ming-Hung Hsu, Ming-Feng Tsai, Hsin-Hsi Chen Department of CSIE National Taiwan University.

Close Read. As we answer questions about the text, please try to be specific in your answers. Go back to the text and find words and phrases that help.

1 15 quality goals for requirements  Justified  Correct  Complete  Consistent  Unambiguous  Feasible  Abstract  Traceable  Delimited  Interfaced.

Evaluation Proposal Defense Observations and Suggestions Yibeltal Kiflie August 2009.

Splitting Complex Temporal Questions for Question Answering systems ACL 2004.

Summarization Focusing on Polarity or Opinion Fragments in Blogs Yohei Seki Toyohashi University of Technology Visiting Scholar at Columbia University.

Text REtrieval Conference (TREC) Implementing a Question-Answering Evaluation for AQUAINT Ellen M. Voorhees Donna Harman.

Comparing Document Segmentation for Passage Retrieval in Question Answering Jorg Tiedemann University of Groningen presented by: Moy’awiah Al-Shannaq

AQUAINT Scenario Breakout -- Group 2, Team 6 12 June 2002.

Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.

Evaluative Response Directions: Complete one of the following written responses. Your response should be a minimum of two paragraphs,

AQUAINT AQUAINT Evaluation Overview Ellen M. Voorhees.

The Roles of the President

Chapter 9. Writing Coherent Documents © 2010 by Bedford/St. Martin's1 Consider these eight questions as you revise the document for coherence: Have you.

BELLWORK What are the three types of crime? (Page 430)

Answering Specific Questions: Who, What, When, Where, Why and How Grade

1 Evaluation of Multi-Media Data QA Systems AQUAINT Breakout Session – June 2002 Howard Wactlar, Carnegie Mellon Yiming Yang, Carnegie Mellon Herb Gish,

1 Evaluation of Opinion Questions ä Session leaders: Ed Hovy, Kathy McKeown ä Topics ä Is evaluating opinion questions feasible at all? How can we construct.

AQUAINT Mid-Year PI Meeting – June 2002 Integrating Robust Semantics, Event Detection, Information Fusion, and Summarization for Multimedia Question Answering.

APUSH DBQ WRITING WORKSHOP. Document Based Question  15 minute mandatory reading period  40 minutes suggested for writing  You must do the following.

General exam advice Do not write too much on ‘give’, ‘outline’, ‘identify’ or ‘state’ questions as you WILL run out of time. E.g. a three mark question.

Understanding Depth of Knowledge. Depth of Knowledge (DOK) Adapted from the model used by Norm Webb, University of Wisconsin, to align standards with.

Executive Office of the President

1 INFILE - INformation FILtering Evaluation Evaluation of adaptive filtering systems for business intelligence and technology watch Towards real use conditions.

11 Thoughts on STS regarding Machine Reading Ralph Weischedel 12 March 2012.

Evaluation Issues: June 2002 Donna Harman Ellen Voorhees.

The Introductory Paragraph and the Thesis Statement

Document Development Cycle

Argumentative Essay Follow the formula…..PEAS

Early challenges to divine right rule

How did the USSR extend its control over Eastern Europe by 1949?

DBQ Essay Outline Paragraph 1: Introduction Paragraph 2: Body including cited documents Paragraph 3: 2nd Body (next topic) including cited documents Paragraph.

Gulf War Why do nations go to war?

Warm-Up Option 1: The US has fixed world problems long enough.

Colin Powell By Omar Santos.

George BUSH senior (born 1924) Vice-President of theUSA ( )

Foreign Policy and the Gulf War

Warm-Up Option 1: The US has fixed world problems long enough.

Long Essay Prompts APUSH Practice.

بسم الله الرحمن الرحيم.

IR Theory: Evaluation Methods

Long Essay Prompts APUSH Practice.

Presentation transcript:

AQUAINT Proposal for a Pilot Study on Extended Definitions: Who is Colin Powell? What is naproxen? Ralph Weischedel & Dan Moldovan 13 June 2002

AQUAINT Who is Colin Powell? l Lt. Gen. Colin L. Powell, the first black to serve as White House national security adviser, was promoted to full general Thursday. l Gulf war leader General Colin Powell, outgoing chairman of the US Joint Chiefs of Staff, was awarded an honorary knighthood by the Queen. l Should (God forbid) the U.S. and Israel become entwined in some kind of joint defensive conflict in the Middle East, we don't have to worry about our chairman of the Joint Chiefs of Staff "communicating." Gen. Colin L. Powell speaks fluent Yiddish, learned from a South Bronx shopkeeper when he was a child. l General Colin Powell, chairman of the US joint chiefs of staff, is reported to have argued that increased US intervention could only be successful if it involved at least as large a ground force as that used in Desert Storm. Example Extracted Sentences from TREC corpora

AQUAINT Who is Colin Powell? l Lt. Gen. Colin L. Powell, the first black to serve as White House national security adviser, was promoted to full general Thursday. l Gulf war leader General Colin Powell, outgoing chairman of the US Joint Chiefs of Staff, was awarded an honorary knighthood by the Queen. l Should (God forbid) the U.S. and Israel become entwined in some kind of joint defensive conflict in the Middle East, we don't have to worry about our chairman of the Joint Chiefs of Staff "communicating." Gen. Colin L. Powell speaks fluent Yiddish, learned from a South Bronx shopkeeper when he was a child. l General Colin Powell, chairman of the US joint chiefs of staff, is reported to have argued that increased US intervention could only be successful if it involved at least as large a ground force as that used in Desert Storm. Redundant Less Important

AQUAINT Who is Colin Powell? l The first black to serve as White House national security adviser l Gulf war leader l General l Awarded an honorary knighthood by the Queen l Chairman of the US joint chiefs of staff Possible Answers

AQUAINT Technical/Evaluation Challenges l For 2002 pilot l For systems: Need to model and detect —Redundancy —Importance l For successful evaluation —Will pooling of answers provide adequate basis for recall estimation? —Can importance and redundancy be judged to measure precision? l For future expansion of task beyond 2002 l User specification of interest l More sophisticated association of time information, a topic being addressed in 2002 NRRC TERQAS workshop l Generation of paragraphs, tables (of positions held), timelines (of major events), …

AQUAINT Ingredients for Eval IssueSuggestion Type of question Who is ; What is ; & What is (and their variants) where the answer is supported by the corpus What system should return Ranked list of noun phrases/verb phrases that give properties/facts about the subject and the document justifying each property/fact What resources are fair Not just a catalog containing the answer; system must return document justifying each list element Guidelines on human assessments Guidelines about ‘important’ facts/properties to be given before eval Mark off-topic material wrong Mark redundant items in list as wrong Mark unimportant material wrong ScoringAverage precision on ranked lists