A Case Study for Adaptive News Systems with Open User Model

Slides:

Advertisements

Similar presentations

Introduction to Information Retrieval Introduction to Information Retrieval Lecture 7: Scoring and results assembly.

Advertisements

DISCOVERING EVENT EVOLUTION GRAPHS FROM NEWSWIRES Christopher C. Yang and Xiaodong Shi Event Evolution and Event Evolution Graph: We define event evolution.

Text Categorization.

Chapter 5: Introduction to Information Retrieval

C ONTEXT - AWARE SIMILARITIES WITHIN THE FACTORIZATION FRAMEWORK Balázs Hidasi Domonkos Tikk C A RR WORKSHOP, 5 TH F EBRUARY 2013, R OME.

Bringing Order to the Web: Automatically Categorizing Search Results Hao Chen SIMS, UC Berkeley Susan Dumais Adaptive Systems & Interactions Microsoft.

Collaborative Filtering Sue Yeon Syn September 21, 2005.

A Music Search Engine Built upon Audio-based and Web-based Similarity Measures P. Knees, T., Pohle, M. Schedl, G. Widmer SIGIR 2007.

The use of a computerized automated feedback system Trevor Barker Dept. Computer Science.

Search Engines and Information Retrieval

Database Management Systems, R. Ramakrishnan1 Computing Relevance, Similarity: The Vector Space Model Chapter 27, Part B Based on Larson and Hearst’s slides.

Modern Information Retrieval Chapter 2 Modeling. Can keywords be used to represent a document or a query? keywords as query and matching as query processing.

Chapter 2Modeling 資工 4B 陳建勳. Introduction.  Traditional information retrieval systems usually adopt index terms to index and retrieve documents.

Retrieval Evaluation. Brief Review Evaluation of implementations in computer science often is in terms of time and space complexity. With large document.

Usability Studies At Microsoft. My Experiences Overview The labs Intro to feature studied Usability study.

Modern Information Retrieval Chapter 2 Modeling. Can keywords be used to represent a document or a query? keywords as query and matching as query processing.

Recommender systems Ram Akella November 26 th 2008.

Xiaomeng Su & Jon Atle Gulla Dept. of Computer and Information Science Norwegian University of Science and Technology Trondheim Norway June 2004 Semantic.

12 -1 Lecture 12 User Modeling Topics –Basics –Example User Model –Construction of User Models –Updating of User Models –Applications.

Slide Image Retrieval: A Preliminary Study Guo Min Liew and Min-Yen Kan National University of Singapore Web IR / NLP Group (WING)

Planned Giving Design Center. What is the Planned Giving Design Center? National network of websites dedicated to advancing philanthropy.

Search Engines and Information Retrieval Chapter 1.

Content-Based Recommendation Systems Michael J. Pazzani and Daniel Billsus Rutgers University and FX Palo Alto Laboratory By Vishal Paliwal.

Processing of large document collections Part 2 (Text categorization) Helena Ahonen-Myka Spring 2006.

1 Information Filtering & Recommender Systems (Lecture for CS410 Text Info Systems) ChengXiang Zhai Department of Computer Science University of Illinois,

Adaptive News Access Daniel Billsus Presented by Chirayu Wongchokprasitti.

Evaluation Experiments and Experience from the Perspective of Interactive Information Retrieval Ross Wilkinson Mingfang Wu ICT Centre CSIRO, Australia.

The identification of interesting web sites Presented by Xiaoshu Cai.

Chapter 16 Structured Systems Analysis. Learning Objectives Know goals, plans, tasks, tools, & results of systems analysis Understand/appreciate costs.

Stuart Card PARC (since ’74) Area Manager of the User Interface Research Center Ph.D. in Psychology from Carnegie Mellon Co-authored “The Psychology of.

Xiaoying Gao Computer Science Victoria University of Wellington Intelligent Agents COMP 423.

UOS 1 Ontology Based Personalized Search Zhang Tao The University of Seoul.

Implicit User Feedback Hongning Wang Explicit relevance feedback 2 Updated query Feedback Judgments: d 1 + d 2 - d 3 + … d k -... Query User judgment.

1 Computing Relevance, Similarity: The Vector Space Model.

CPSC 404 Laks V.S. Lakshmanan1 Computing Relevance, Similarity: The Vector Space Model Chapter 27, Part B Based on Larson and Hearst’s slides at UC-Berkeley.

WIRED Week 3 Syllabus Update (next week) Readings Overview - Quick Review of Last Week’s IR Models (if time) - Evaluating IR Systems - Understanding Queries.

PSEUDO-RELEVANCE FEEDBACK FOR MULTIMEDIA RETRIEVAL Seo Seok Jun.

Retroactive Answering of Search Queries Beverly Yang Glen Jeh.

United Nations Economic Commission for Europe Statistical Division The UNECE Gender Database and Website UNECE Statistical Division.

Carnegie Mellon Novelty and Redundancy Detection in Adaptive Filtering Yi Zhang, Jamie Callan, Thomas Minka Carnegie Mellon University {yiz, callan,

Creating Subjective and Objective Sentence Classifier from Unannotated Texts Janyce Wiebe and Ellen Riloff Department of Computer Science University of.

WCRE - Oct Temporal Model Explorer - Farah and Lethbridge Temporal Exploration of Software Models Hanna Farah IBM Ottawa Timothy Lethbridge University.

United Nations Economic Commission for Europe Statistical Division The UNECE Gender Statistics Database and Website UNECE Statistical Division.

Event-Based Extractive Summarization E. Filatova and V. Hatzivassiloglou Department of Computer Science Columbia University (ACL 2004)

User Modeling and Recommender Systems: recommendation algorithms

1 Text Categorization  Assigning documents to a fixed set of categories  Applications:  Web pages  Recommending pages  Yahoo-like classification hierarchies.

Sparse Coding: A Deep Learning using Unlabeled Data for High - Level Representation Dr.G.M.Nasira R. Vidya R. P. Jaia Priyankka.

Information Retrieval in Practice

General-Purpose Learning Machine

Recommender Systems & Collaborative Filtering

Item-to-Item Recommender Network Optimization

WorkIT SQA Work Placement Units & Pupil Questionnaires.

Introduction to Programmng in Python

IST 516 Fall 2011 Dongwon Lee, Ph.D.

Assessment, Scoring, Recording and Result Analysis

NACE Committee Workspace (NCW)Training Workspace Features Navigation

UNITED IN SERVICE TO OUR NATION

Machine Learning Week 1.

Information Retrieval

PT2520 Unit 2: Gather Information and Define Requirements

Author: Kazunari Sugiyama, etc. (WWW2004)

Interpreting Your Results

Text Categorization Assigning documents to a fixed set of categories

EVOSS Walkthrough.

Code search & recommendation engines

Dennis Zhao,1 Dragomir Radev PhD1 LILY Lab

GCSE Computer Science System Software Workbook Year 10 Half Term 4

Jonathan Elsas LTI Student Research Symposium Sept. 14, 2007

Serik Tursunaliev, Veronica Ross

MyASQ myASQ is a centralized, online community that provides timely, relevant, and personalized engagement for members of ASQ, allowing members to make.

Presentation transcript:

A Case Study for Adaptive News Systems with Open User Model NewsMe: A Case Study for Adaptive News Systems with Open User Model Preliminary Examination Paper 2007 Chirayu Wongchokprasitti IS PhD Student School of Information Sciences

NewsMe

NewsMe Overview Personalized News Access System Feed the news that response to the user’s interest 82 RSS news feeds, 21 sources 8 News Topics Ranking the news Open User Model based system

NewsMe Interface 4 News Sections Recent News Recommended News My Profile News History

User Feedback Method Add a news item to Tracked News Add a news item to Blacklist

User Model Manipulation Update rating of news in user model

User Model Manipulation (Con’t) List all history of viewed news Update rating of news in user model

Learning User Models for News Access The system uses a machine learning approach to build a simple model of each user’s interests. A similarity-based method achieves the balance of learning and adapts quickly to change interests while avoiding brittleness.

Learning User Models for News Access (cont.) The purpose of the user model First, it should contain information about recently read events, so that stories which belong to the same thread can be identified. To allow for identification of news that user already knows. The k-nearest-neighbor algorithm (kNN) is used to achieve the desired functionality. Convert news contents to tf-idf vectors (term-frequency/inverse-document-frequency). Use the cosine similarity measure to quantify the similarity of two vectors.

Learning User Models for News Access (cont.) Decay Function Freshness of news content is our issue. Freshness should decay exponentially day by day. Freshness of news remains a half after fed 7 days. is the initial freshness of news content. is a decay instance, which its value is around 0.099.

Study Design 20 Participants Assign to be Information Analysts 2 News Topic: US and Business 2 Sessions, 3 stages per session Retrieved News: Nov 28th – Dec 12th, 2006 Google Notebook extension (http://www.google.com/notebook)

Implicit VS Explicit Feedback Implicit feedback Assuming every news user read is a tracked news Explicit feedback Users add news items to their user model Tracked news as Positive sample Blacklist News as Negative sample

Performance hypotheses are: H1: The open model system with user profile manipulation by users performs better than the open model system without them, H1.1: The open model system with explicit feedback generates results with better performance, and, H1.2: Users with explicit feedback system demonstrate higher task performance.

Hypotheses (Con’t) User Perspective hypotheses are: H2: Users prefer the user profile manipulation features in the open model system, H2.1: Users appreciate better in the system with explicit feedback, and, H2.2: Users appreciate the ability to control their profiles.

Preliminary Results The Ground Truth System Performance Analysis User Performance Analysis User Feedback Analysis

The Ground Truth F-measure defines as follows: Summary of news items in the study

System Performance Analysis

System Precision @ First Screen

System Precision @ 60

System Precision @ 100

News Items Manipulation vs. System Performance (Stage 2)

Tracked News Blacklist (Stage 2)

Tracked News History (Stage 2)

Blacklist Tracked News (Stage 2)

News Items Manipulation vs. System Performance (Stage 3)

Tracked News Blacklist (Stage 3)

Tracked News History (Stage 3)

Blacklist Tracked News (Stage 3)

User Performance Analysis

User Precision

User Avg. Rank of Selected Items

User Feedback Analysis A two-way ANOVA was performed on the questionnaire data to examine significant differences in user answers by system and by stage. On the question 3, subjects indicated they trusted in system’s ability to find useful information for the US topic versus the Business topic in overall (p-value = 0.017). On the question 7, subjects indicated My Profile helps them to understand how the system finds useful news items for the US topic versus the Business topic in overall (p-value = 0.013).

Future Work Open Model with explicit feedback did not outperform the baseline. The experiment indicates that without caution, user model manipulation not only benefit the performance but lower the output. Binary rating might not be a suitable way. Fuzzy rating is a good way to study further.

Q & A