Towards a Personal Briefing Assistant

Slides:



Advertisements
Similar presentations
Temporal Query Log Profiling to Improve Web Search Ranking Alexander Kotov (UIUC) Pranam Kolari, Yi Chang (Yahoo!) Lei Duan (Microsoft)
Advertisements

Chapter 5: Introduction to Information Retrieval
1 Evaluation Rong Jin. 2 Evaluation  Evaluation is key to building effective and efficient search engines usually carried out in controlled experiments.
The Research Consumer Evaluates Measurement Reliability and Validity
IS2210: Systems Analysis and Systems Design and Change
Query Dependent Pseudo-Relevance Feedback based on Wikipedia SIGIR ‘09 Advisor: Dr. Koh Jia-Ling Speaker: Lin, Yi-Jhen Date: 2010/01/24 1.
GENERATING AUTOMATIC SEMANTIC ANNOTATIONS FOR RESEARCH DATASETS AYUSH SINGHAL AND JAIDEEP SRIVASTAVA CS DEPT., UNIVERSITY OF MINNESOTA, MN, USA.
ACTIVELY ENGAGING THE STAKEHOLDER IN DEFINING REQUIREMENTS FOR THE BUSINESS, THE STAKEHOLDER, SOLUTION OR TRANSITION Requirements Elicitation.
Predicting the Semantic Orientation of Adjective Vasileios Hatzivassiloglou and Kathleen R. McKeown Presented By Yash Satsangi.
Approaches to automatic summarization Lecture 5. Types of summaries Extracts – Sentences from the original document are displayed together to form a summary.
Research Proposal Development of research question
Recommender systems Ram Akella November 26 th 2008.
ML ALGORITHMS. Algorithm Types Classification (supervised) Given -> A set of classified examples “instances” Produce -> A way of classifying new examples.
Minnesota Manual of Accommodations for Students with Disabilities Training Guide
Neural Technology and Fuzzy Systems in Network Security Project Progress Group 2: Omar Ehtisham Anwar Aneela Laeeq
CONTENT-BASED BOOK RECOMMENDING USING LEARNING FOR TEXT CATEGORIZATION TRIVIKRAM BHAT UNIVERSITY OF TEXAS AT ARLINGTON DATA MINING CSE6362 BASED ON PAPER.
Chapter 5: Information Retrieval and Web Search
1 Development of Valid and Reliable Case Studies for Teaching, Diagnostic Reasoning, and Other Purposes Margaret Lunney, RN, PhD Professor College of.
S/W Project Management
N By: Md Rezaul Huda Reza n
Evaluation Experiments and Experience from the Perspective of Interactive Information Retrieval Ross Wilkinson Mingfang Wu ICT Centre CSIRO, Australia.
A Simple Unsupervised Query Categorizer for Web Search Engines Prashant Ullegaddi and Vasudeva Varma Search and Information Extraction Lab Language Technologies.
1 Automating Slot Filling Validation to Assist Human Assessment Suzanne Tamang and Heng Ji Computer Science Department and Linguistics Department, Queens.
Testing & modeling users. The aims Describe how to do user testing. Discuss the differences between user testing, usability testing and research experiments.
Chapter 6: Information Retrieval and Web Search
Binxing Jiao et. al (SIGIR ’10) Presenter : Lin, Yi-Jhen Advisor: Dr. Koh. Jia-ling Date: 2011/4/25 VISUAL SUMMARIZATION OF WEB PAGES.
Ensemble Learning Spring 2009 Ben-Gurion University of the Negev.
Contextual Ranking of Keywords Using Click Data Utku Irmak, Vadim von Brzeski, Reiner Kraft Yahoo! Inc ICDE 09’ Datamining session Summarized.
Topics Covered Phase 1: Preliminary investigation Phase 1: Preliminary investigation Phase 2: Feasibility Study Phase 2: Feasibility Study Phase 3: System.
Evaluation of Agent Building Tools and Implementation of a Prototype for Information Gathering Leif M. Koch University of Waterloo August 2001.
‘Sujala’ Karnataka Watershed Development Programme Developing the Training Strategy Introduction for the “Brainstorming Workshop 1st of October 2003.
Probabilistic Latent Query Analysis for Combining Multiple Retrieval Sources Rong Yan Alexander G. Hauptmann School of Computer Science Carnegie Mellon.
Working Technologically Early Stage 1 – Stage 3. Students evaluate by: recounting the steps taken to reach a final solution discussing their likes and.
Does GridGIS require more intelligence than GIS? Claire Jarvis Department of Geography GEOGRAPHY.
Final Year Project 1 (FYP 1) CHAPTER 1 : INTRODUCTION
Minnesota Manual of Accommodations for Students with Disabilities Training January 2010.
Identifying “Best Bet” Web Search Results by Mining Past User Behavior Author: Eugene Agichtein, Zijian Zheng (Microsoft Research) Source: KDD2006 Reporter:
On Using SIFT Descriptors for Image Parameter Evaluation Authors: Patrick M. McInerney 1, Juan M. Banda 1, and Rafal A. Angryk 2 1 Montana State University,
A Comprehensive Comparative Study on Term Weighting Schemes for Text Categorization with SVM Lan Man 3 Nov, 2004.
Learning Event Durations from Event Descriptions Feng Pan, Rutu Mulkar, Jerry R. Hobbs University of Southern California ACL ’ 06.
© NCSR, Frascati, July 18-19, 2002 CROSSMARC big picture Domain-specific Web sites Domain-specific Spidering Domain Ontology XHTML pages WEB Focused Crawling.
Therapeutic Use of Groups superKAT :). Group 3 or more people who are together for some period of time with common goals or share a common purpose.
 1- Definition  2- Helpdesk  3- Asset management  4- Analytics  5- Tools.
WP4 Models and Contents Quality Assessment
ITIL: Service Transition
Chapter 25 Process Improvement.
Learning Profiles from User Interactions
Chapter 3, Project Organization and Communication
Evaluation Requirements for MSP and Characteristics of Designs to Estimate Impacts with Confidence Ellen Bobronnikov March 23, 2011.
Queensland University of Technology
Causal Analysis & Resolution (CAR) Support Category
Towards a framework for architectural design decision support
Automatic cLasification d
The Development Process of Web Applications
Hanan Ayad Supervisor Prof. Mohamed Kamel
Chapter 6: Design of Expert Systems
Martin Rajman, Martin Vesely
Amazing The Re:Search Engine Jaime Teevan MIT, CSAIL.
PSY 614 Instructor: Emily Bullock Yowell, Ph.D.
Measuring Project Performance: Tips and Tools to Showcase Your Results
Unit4 Customer Portal Submitting & Managing Cases.
Social Knowledge Mining
European TRAINING FOUNDATION
2018 OSEP Project Directors’ Conference
Discriminative Frequent Pattern Analysis for Effective Classification
iSRD Spam Review Detection with Imbalanced Data Distributions
Grantee Guide to Project Performance Measurement
Testing & modeling users
UNIT No- III- Leverging Information System ( Investing strategy )
Dr. Phyllis Underwood REL Southeast
Presentation transcript:

Towards a Personal Briefing Assistant Nikesh Garera LTI Student Research Symposium – 24th Sep 2004 Carnegie Mellon University

Creating Briefings/Reports in academia Report Writer Funding bodies Peer groups Students

How can Briefing Assistant help ? Weekly participant interviews Learned rules Model parameters Automatic Summarizer Reordered List (items ranked according to importance) 1. _____ 2. _____ 3. _____ ....... Learning module Creates Summary Select/Deselect items Specify rules Synthesize items

Domain: Weekly Project Interviews - Concrete Achievements e.g. Presented Project Plan to visiting project managers on 20th May 2003 - Problems and Solutions e.g. May be neglecting the idea of "audience". All communication is centered on user request and neglects that the content producer has a point of view. - People and Places e.g. Met with X and Y, founders of Z, which is in the Knowledge Management business. It appears that the system and the tools they have built may be of use to us. - News and Views e.g. Read about the Shaken/KM knowledge base system - Artifacts and Processes e.g. Learned about PLONE – System for collaborative construction of websites - Project Status e.g. We are two weeks behind schedule in implementation of the prototype. - General Activity Report e.g. Considered attaching a "spy" module that tracks ALL of user’s keystrokes and mouse clicks; purpose is to see if we can mine these data. Weekly log of project-related activities Structured interview self-administered administrative assistant Refining of corpus lead to a set of 12 weekly interviews (Average of 27.67 items per week)

Identifying “Research-Acts” Domain features (Research-acts) Tasks/Actions Action-Item Aspiration Commitment Evaluation Event Hardware Human-Relations Idea Info-gain Issue Object-gain Observation Other Part-Product Product Process Publication Status Named Entities Organization Person Software Meta features Author

Design of Evaluation Experiment Seven “expert” subjects participated in the study Subjects create 5-item summaries for 12 successive weeks Summarization model is incrementally trained from week-to-week. (using logistic regression classifier)

Evaluation Metrics Need to evaluate a ranked-list given binary classification for each item in the list. Average Precision Top 30% (% summary covered in top 30 % of the ranked list) Baseline Random order (Avg prec: 0.265, Top 30%: 0.259) Conventional heuristic based summarization methods do not give much leverage. (Centroid based summarization: Avg Prec: 0.25, Top 30%: 0.3) Elapsed Time: (Time taken to create a summary from a ranked list)

Evaluation Results Elapsed Time: Metric Baseline (random) 6th biweek Improvement (factor) Avg Prec 0.265 0.586 2.21 Top 30% 0.259 0.729 2.81 Elapsed Time: Subjects also show a significant 31% decrease in task time over the 6 biweeks but this can be attributed to other factors. baseline

Different people produce different summaries Kappa statistic (Inter-rater agreement): K = (P(A) – P(E)) / (1 – P(E)) P(A) = proportion of times n raters agree P(E) = proportion of times we would expect the n raters to agree by chance Low inter-rater agreement (K = 0.26, n=7) for summary items selected Different strokes for different folks !

Differences in feature weights Subject A OBJECT-GAIN -28.96 INFO-GAIN -10.44 ASPIRATION -9.68 HUMAN-RELATIONS -9.22 HARDWARE 7.19 ACTION-ITEM 6.59 OTHER -5.1 PRODUCT 4.24 PART-PRODUCT 3.79 EVALUATION 3.77 IDEA 2.65 author=aaaaaa -2.48 Subject D PUBLICATION -7.54 OBJECT-GAIN -5.67 OTHER -5.15 HUMAN-RELATIONS 4.85 HARDWARE 4.11 COMMITMENT 3.9 EVALUATION 3.78 PROCESS -3.61 IDEA 3.5 PRODUCT 2.56 ASPIRATION -2.53 author=bbbbbb -1.96 Top 12 features for two different subjects (Features are ordered by absolute weight in the model)

Is this approach domain independent ? The assistance framework is fairly general but the features are manually annotated. We have investigated two techniques: Use information extraction approaches Such as in named-entity extraction And based on the 22 features previously described Use bag of words as features

Results: Using automatic feature extraction Smoothing of corpus Stemming, Removing stop-words Tokenization of known domain terms. A simple rote classifier was trained using cross validation to automatically annotate features. Logistic regression classifier used for learning subject models. baseline Metric Baseline (random) 6th biweek Improvement (factor) Avg Prec 0.265 0.473 1.785 Top 30% 0.259 0.614 2.36

Results: using bag of words as features Features: values were tf/idf weights of the respective words (L1-norm) Voted Perceptron classifier used for learning subject models (due to high dimensionality of feature space) baseline Metric Baseline (random) 6th biweek Improvement Avg Prec 0.265 0.413 1.56 Top 30% 0.259 0.4 1.54

Summary Looked at the application of Briefing Assistant and identified several research problems. Proposed a learning based framework for assisting the report writer in creating “tailored” reports. Evaluation study Indicates potential for increase in user productivity Also got encouraging initial results for generalizing this approach for different domains.

Interesting problems… What if dynamics of the project change ? How can a report writer instruct the system ? How can the system proactively acquire relevant information ?

Thank you