1. Lecture 16: Filtering & TDT
Principles of Information Retrieval
Prof. Ray Larson
University of California, Berkeley
School of Information Management & Systems
Tuesday and Thursday 10:30 am - 12:00 pm
Spring 2006
IS 240 – Spring 2006

2. Overview Review Filtering & Routing TDT – Topic Detection and Tracking
LSI
Filtering & Routing
TDT – Topic Detection and Tracking
IS 240 – Spring 2006

3. Overview Review Filtering & Routing TDT – Topic Detection and Tracking
LSI
Filtering & Routing
TDT – Topic Detection and Tracking
IS 240 – Spring 2006

4. How LSI Works Start with a matrix of terms by documents
Analyze the matrix using SVD to derive a particular “latent semantic structure model”
Two-Mode factor analysis, unlike conventional factor analysis, permits an arbitrary rectangular matrix with different entities on the rows and columns
Such as Terms and Documents
IS 240 – Spring 2006

5. How LSI Works
The rectangular matrix is decomposed into three other matices of a special form by SVD
The resulting matrices contain “singular vectors” and “singular values”
The matrices show a breakdown of the original relationships into linearly independent components or factors
Many of these components are very small and can be ignored – leading to an approximate model that contains many fewer dimensions
IS 240 – Spring 2006

6. How LSI Works
Titles
C1: Human machine interface for LAB ABC computer applications
C2: A survey of user opinion of computer system response time
C3: The EPS user interface management system
C4: System and human system engineering testing of EPS
C5: Relation of user-percieved response time to error measurement
M1: The generation of random, binary, unordered trees
M2: the intersection graph of paths in trees
M3: Graph minors IV: Widths of trees and well-quasi-ordering
M4: Graph minors: A survey
Italicized words occur and multiple docs and are indexed
IS 240 – Spring 2006

7. How LSI Works Terms Documents c1 c2 c3 c4 c5 m1 m2 m3 m4
Human
Interface
Computer
User
System
Response
Time
EPS
Survey
Trees
Graph
Minors
IS 240 – Spring 2006

8. How LSI Works Dimension 2 SVD to 2 dimensions Dimension 1
11graph
M2(10,11,12)
10 Tree
12 minor
9 survey
M1(10)
7 time
3 computer
4 user
6 response
5 system
2 interface
1 human
M4(9,11,12)
M2(10,11)
C2(3,4,5,6,7,9)
C5(4,6,7)
C1(1,2,3)
C3(2,4,5,8)
C4(1,5,8)
Q(1,3)
Blue dots are terms
Documents are red squares
Blue square is a query
Dotted cone is cosine .9 from
Query “Human Computer Interaction”
-- even docs with no terms in common
(c3 and c5) lie within cone.
SVD to 2 dimensions
IS 240 – Spring 2006

9. How LSI Works X = T0S0D0’ X T0 = S0 D0’ txd txm mxm mxd docs terms
T0 has orthogonal, unit-length columns (T0’ T0 = 1)
D0 has orthogonal, unit-length columns (D0’ D0 = 1)
S0 is the diagonal matrix of singular values
t is the number of rows in X
d is the number of columns in X
m is the rank of X (<= min(t,d)
IS 240 – Spring 2006

10. Overview Review Filtering & Routing TDT – Topic Detection and Tracking
LSI
Filtering & Routing
TDT – Topic Detection and Tracking
IS 240 – Spring 2006

11. Filtering Characteristics of Filtering systems:
Designed for unstructured or semi-structured data
Deal primarily with text information
Deal with large amounts of data
Involve streams of incoming data
Filtering is based on descriptions of individual or group preferences – profiles. May be negative profiles (e.g. junk mail filters)
Filtering implies removing non-relevant material as opposed to selecting relevant.
IS 240 – Spring 2006

12. Filtering Similar to IR, with some key differences
Similar to Routing – sending relevant incoming data to different individuals or groups is virtually identical to filtering – with multiple profiles
Similar to Categorization systems – attaching one or more predefined categories to incoming data objects – is also similar, but is more concerned with static categories (might be considered information extraction)
IS 240 – Spring 2006

13. Structure of an IR System
Search
Line
Interest profiles
& Queries
Documents
& data
Rules of the game =
Rules for subject indexing +
Thesaurus (which consists of
Lead-In
Vocabulary
and
Indexing
Language
Storage
Potentially
Relevant
Comparison/
Matching
Store1: Profiles/
Search requests
Store2: Document
representations
(Descriptive and
Subject)
Formulating query in
terms of
descriptors
Storage of
profiles
Information Storage and Retrieval System
Adapted from Soergel, p. 19
IS 240 – Spring 2006

14. Structure of an Filtering System
Interest profiles
Raw Documents
& data
Rules of the game =
Rules for subject indexing +
Thesaurus (which consists of
Lead-In
Vocabulary
and
Indexing
Language
Incoming
Data
Stream
Potentially
Relevant
Documents
Comparison/
filtering
Store1: Profiles/
Search requests
Doc surrogate
Indexing/
Categorization/
Extraction
Formulating query in
terms of
descriptors
Storage of
profiles
Information Filtering System
Adapted from Soergel, p. 19
Individual or Group
users
IS 240 – Spring 2006

15. Major differences between IR and Filtering
IR concerned with single uses of the system
IR recognizes inherent faults of queries
Filtering assumes profiles can be better than IR queries
IR concerned with collection and organization of texts
Filtering is concerned with distribution of texts
IR is concerned with selection from a static database.
Filtering concerned with dynamic data stream
IR is concerned with single interaction sessions
Filtering concerned with long-term changes
IS 240 – Spring 2006

16. Contextual Differences
In filtering the timeliness of the text is often of greatest significance
Filtering often has a less well-defined user community
Filtering often has privacy implications (how complete are user profiles?, what to they contain?)
Filtering profiles can (should?) adapt to user feedback
Conceptually similar to Relevance feedback
IS 240 – Spring 2006

17. Methods for Filtering Adapted from IR Collaborative filtering
E.g. use a retrieval ranking algorithm against incoming documents.
Collaborative filtering
Individual and comparative profiles
IS 240 – Spring 2006

18. TREC Filtering Track Original Filtering Track
Participants are given a starting query
They build a profile using the query and the training data
The test involves submitting the profile (which is not changed) and then running it against a new data stream
New Adaptive Filtering Track
Same, except the profile can be modified as each new relevant document is encountered.
Since streams are being processed, there is no ranking of documents
IS 240 – Spring 2006

19. TREC-8 Filtering Track
Following Slides from the TREC-8 Overview by Ellen Voorhees
IS 240 – Spring 2006

20. IS 240 – Spring 2006

21. IS 240 – Spring 2006

22. IS 240 – Spring 2006

23. IS 240 – Spring 2006

24. Overview Review Filtering & Routing TDT – Topic Detection and Tracking
LSI
Filtering & Routing
TDT – Topic Detection and Tracking
IS 240 – Spring 2006

25. TDT: Topic Detection and Tracking
Intended to automatically identify new topics – events, etc. – from a stream of text and follow the development/further discussion of those topics
IS 240 – Spring 2006

26. Topic Detection and Tracking
Introduction and Overview
The TDT3 R&D Challenge
TDT3 Evaluation Methodology
Topic Detection and Tracking
Slides from “Overview NIST Topic Detection and Tracking
Introduction and Overview” by G. Doddington
IS 240 – Spring 2006

27. TDT Task Overview* 5 R&D Challenges: TDT3 Corpus Characteristics:†
Story Segmentation
Topic Tracking
Topic Detection
First-Story Detection
Link Detection
TDT3 Corpus Characteristics:†
Two Types of Sources:
Text • Speech
Two Languages:
English 30,000 stories
Mandarin 10,000 stories
11 Different Sources:
_8 English__ Mandarin ABC CNN VOA PRI VOA XIN NBC MNB ZBN APW NYT
* see for details
† see for details
IS 240 – Spring 2006

28. Preliminaries A topic is … A story is …
a seminal event or activity, along with all directly related events and activities.
A story is …
a topically cohesive segment of news that includes two or more DECLARATIVE independent clauses about a single event.
IS 240 – Spring 2006

29. Example Topic Title: Mountain Hikers Lost
WHAT: 35 or 40 young Mountain Hikers were lost in an avalanche in France around the 20th of January.
WHERE: Orres, France
WHEN: January 1998
RULES OF INTERPRETATION: 5. Accidents
IS 240 – Spring 2006

30. The Segmentation Task:
To segment the source stream into its constituent stories, for all audio sources.
(for Radio and TV only)
Transcription:
text (words)
Story:
Non-story:
IS 240 – Spring 2006

31. Story Segmentation Conditions
1 Language Condition:
3 Audio Source Conditions:
3 Decision Deferral Conditions:
IS 240 – Spring 2006

32. The Topic Tracking Task:
To detect stories that discuss the target topic, in multiple source streams.
Find all the stories that discuss a given target topic
Training: Given Nt sample stories that discuss a given target topic,
Test: Find all subsequent stories that discuss the target topic.
on-topic
unknown
training data
test data
New This Year:
not guaranteed
to be off-topic
IS 240 – Spring 2006

33. Topic Tracking Conditions
9 Training Conditions:
1 Language Test Condition:
3 Source Conditions:
2 Story Boundary Conditions:
IS 240 – Spring 2006

34. The Topic Detection Task:
To detect topics in terms of the (clusters of) stories that discuss them.
Unsupervised topic training A meta-definition of topic is required independent of topic specifics.
New topics must be detected as the incoming stories are processed.
Input stories are then associated with one of the topics.
a topic!
IS 240 – Spring 2006

35. Topic Detection Conditions
3 Language Conditions:
3 Source Conditions:
Decision Deferral Conditions:
2 Story Boundary Conditions:
IS 240 – Spring 2006

36. The First-Story Detection Task:
To detect the first story that discusses a topic, for all topics.
There is no supervised topic training
(like Topic Detection)
Time
First Stories
Not First Stories
= Topic 1
= Topic 2
IS 240 – Spring 2006

37. First-Story Detection Conditions
1 Language Condition:
3 Source Conditions:
Decision Deferral Conditions:
2 Story Boundary Conditions:
IS 240 – Spring 2006

38. The Link Detection Task
To detect whether a pair of stories discuss the same topic.
The topic discussed is a free variable.
Topic definition and annotation is unnecessary.
The link detection task represents a basic functionality, needed to support all applications (including the TDT applications of topic detection and tracking).
The link detection task is related to the topic tracking task, with Nt = 1.
same topic?
IS 240 – Spring 2006

39. Link Detection Conditions
1 Language Condition:
3 Source Conditions:
Decision Deferral Conditions:
1 Story Boundary Condition:
IS 240 – Spring 2006

40. TDT3 Evaluation Methodology
All TDT3 tasks are cast as statistical detection (yes-no) tasks.
Story Segmentation: Is there a story boundary here?
Topic Tracking: Is this story on the given topic?
Topic Detection: Is this story in the correct topic-clustered set?
First-story Detection: Is this the first story on a topic?
Link Detection: Do these two stories discuss the same topic?
Performance is measured in terms of detection cost, which is a weighted sum of miss and false alarm probabilities: CDet = CMiss • PMiss • Ptarget CFA • PFA • (1- Ptarget)
Detection Cost is normalized to lie between 0 and 1: (CDet)Norm = CDet / min{CMiss • Ptarget, CFA • (1- Ptarget)}
IS 240 – Spring 2006

41. Example Performance Measures:
Tracking Results on Newswire Text (BBN)
0.01
0.1 1
English
Mandarin
Normalized Tracking Cost
IS 240 – Spring 2006

42. More on TDT
Some slides from James Allan from the HICSS meeting in January 2005
IS 240 – Spring 2006