Download presentation
Presentation is loading. Please wait.
1
Multimedia Databases Text II
2
Outline Spatial Databases Temporal Databases Spatio-temporal Databases Multimedia Databases Text databases Image and video databases Time Series databases Data Mining
3
Text - Detailed outline Text databases problem full text scanning inversion signature files clustering information filtering and LSI
4
Vector Space Model and Clustering keyword queries (vs Boolean) each document: -> vector (HOW?) each query: -> vector search for ‘similar’ vectors
5
Vector Space Model and Clustering main idea: document...data... aaron zoo data V (= vocabulary size) ‘indexing’
6
Vector Space Model and Clustering Then, group nearby vectors together Q1: cluster search? Q2: cluster generation? Two significant contributions ranked output relevance feedback
7
Vector Space Model and Clustering cluster search: visit the (k) closest superclusters; continue recursively CS TRs MD TRs
8
Vector Space Model and Clustering ranked output: easy! CS TRs MD TRs
9
Vector Space Model and Clustering relevance feedback (brilliant idea) [Roccio’73] CS TRs MD TRs
10
Vector Space Model and Clustering relevance feedback (brilliant idea) [Roccio’73] How? CS TRs MD TRs
11
Vector Space Model and Clustering How? A: by adding the ‘good’ vectors and subtracting the ‘bad’ ones CS TRs MD TRs
12
Outline - detailed main idea cluster search cluster generation evaluation
13
Cluster generation Problem: given N points in V dimensions, group them
14
Cluster generation Problem: given N points in V dimensions, group them
15
Cluster generation We need Q1: document-to-document similarity Q2: document-to-cluster similarity
16
Cluster generation Q1: document-to-document similarity (recall: ‘bag of words’ representation) D1: {‘data’, ‘retrieval’, ‘system’} D2: {‘lung’, ‘pulmonary’, ‘system’} distance/similarity functions?
17
Cluster generation A1: # of words in common A2:........ normalized by the vocabulary sizes A3:.... etc About the same performance - prevailing one: cosine similarity
18
Cluster generation cosine similarity: sim(D1, D2) = cos( θ ) = sum(v 1,i * v 2,i ) / len(v 1 )/ len(v 2 ) θ D1 D2
19
Cluster generation cosine similarity - observations: related to the Euclidean distance weights v i,j : according to tf/idf θ D1 D2
20
Cluster generation tf (‘term frequency’) high, if the term appears very often in this document. idf (‘inverse document frequency’) penalizes ‘common’ words, that appear in almost every document
21
Cluster generation We need Q1: document-to-document similarity Q2: document-to-cluster similarity ?
22
Cluster generation A1: min distance (‘single-link’) A2: max distance (‘all-link’) A3: avg distance A4: distance to centroid ?
23
Cluster generation A1: min distance (‘single-link’) leads to elongated clusters A2: max distance (‘all-link’) many, small, tight clusters A3: avg distance in between the above A4: distance to centroid fast to compute
24
Cluster generation We have document-to-document similarity document-to-cluster similarity Q: How to group documents into ‘natural’ clusters
25
Cluster generation A: *many-many* algorithms - in two groups [VanRijsbergen]: theoretically sound (O(N^2)) independent of the insertion order iterative (O(N), O(N log(N))
26
Outline - detailed main idea cluster search cluster generation evaluation
27
Evaluation Q: how to measure ‘goodness’ of one distance function vs another? A: ground truth (by humans) and ‘precision’ and ‘recall’
28
Evaluation precision = (retrieved and relevant) / retrieved 100% precision -> no false alarms recall = (retrieved and relevant)/ relevant 100% recall -> no false dismissals
29
Text - Detailed outline text problem full text scanning inversion signature files clustering information filtering and LSI
30
LSI - Detailed outline LSI problem definition main idea experiments
31
Information Filtering + LSI [Foltz+,’92] Goal: users specify interests (= keywords) system alerts them, on suitable news- documents Major contribution: LSI = Latent Semantic Indexing latent (‘hidden’) concepts
32
Information Filtering + LSI Main idea map each document into some ‘concepts’ map each term into some ‘concepts’ ‘Concept’:~ a set of terms, with weights, e.g. “data” (0.8), “system” (0.5), “retrieval” (0.6) -> DBMS_concept
33
Information Filtering + LSI Pictorially: term-document matrix (BEFORE)
34
Information Filtering + LSI Pictorially: concept-document matrix and...
35
Information Filtering + LSI... and concept-term matrix
36
Information Filtering + LSI Q: How to search, eg., for ‘system’?
37
Information Filtering + LSI A: find the corresponding concept(s); and the corresponding documents
38
Information Filtering + LSI A: find the corresponding concept(s); and the corresponding documents
39
Information Filtering + LSI Thus it works like an (automatically constructed) thesaurus: we may retrieve documents that DON’T have the term ‘system’, but they contain almost everything else (‘data’, ‘retrieval’)
40
LSI - Discussion - Conclusions Great idea, to derive ‘concepts’ from documents to build a ‘statistical thesaurus’ automatically to reduce dimensionality Often leads to better precision/recall but: Needs ‘training’ set of documents ‘concept’ vectors are not sparse anymore
41
LSI - Discussion - Conclusions Observations Bellcore (-> Telcordia) has a patent used for multi-lingual retrieval How exactly SVD works?
42
Indexing - Detailed outline primary key indexing secondary key / multi-key indexing spatial access methods fractals text SVD: a powerful tool multimedia...
43
References Foltz, P. W. and S. T. Dumais (Dec. 1992). "Personalized Information Delivery: An Analysis of Information Filtering Methods." Comm. of ACM (CACM) 35(12): 51-60. Can, F. and E. A. Ozkarahan (Dec. 1990). "Concepts and Effectiveness of the Cover-Coefficient-Based Clustering Methodology for Text Databases." ACM TODS 15(4): 483-517. Rocchio, J. J. (1971). Relevance Feedback in Information Retrieval. The SMART Retrieval System - Experiments in Automatic Document Processing. G. Salton. Englewood Cliffs, New Jersey, Prentice-Hall Inc.
44
References - cont’d Salton, G. (1971). The SMART Retrieval System - Experiments in Automatic Document Processing. Englewood Cliffs, New Jersey, Prentice-Hall Inc. Salton, G. and M. J. McGill (1983). Introduction to Modern Information Retrieval, McGraw-Hill. Van-Rijsbergen, C. J. (1979). Information Retrieval. London, England, Butterworths. Zahn, C. T. (Jan. 1971). "Graph-Theoretical Methods for Detecting and Describing Gestalt Clusters." IEEE Trans. on Computers C-20(1): 68-86.
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.