Download presentation
Presentation is loading. Please wait.
Published byMorgan Kelley Johnson Modified over 9 years ago
1
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Evaluation of novelty metrics for sentence-level novelty mining Presenter : Lin, Shu-Han Authors : Flora S. Tsai, Wenyin Tang, Kap Luk Chan Information Sciences, InS (2010)
2
Intelligent Database Systems Lab N.Y.U.S.T. I. M. 2 Outline Introduction Motivation Objective Methodology Compare study Experiments Conclusion Comments
3
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Introduction 3 Define Novelty? Novelty is the opposite of “similarity ” or “redundancy” Novelty: Given the set of relevant sentences in all documents, identify all novel sentence. How to identify Novelty sentences? A novelty score: Measured and Scored by a novelty metric
4
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Motivation 4 Sentence 1: U.S. Stocks set for big sell-off Sentence 2 (incoming sentence) : U.S. Stocks *S2 is covered by S1 Novelty(S1, S2) = 1 – similarity(S1, S2) There is low similarity between S1 and S2 SO S2 is novelty ???
5
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Objectives 5 How to choose a novelty metric? How to set a suitable threshold automatically?
6
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Methodology - Novelty Metrics 6 Symmetric (1 – similarity) S1 is novelty to S2 S2 is novelty to S1 Asymmetric S1 is not novelty to S2 S2 is novelty to S1
7
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Methodology - Symmetric metrics 7 Cosine similarity Jaccard Similarity
8
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Methodology - ASymmetric metrics 8 Overlap metric New word count metric
9
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Compare study 9 Performance Requirements (trade-off) : high (recall / precision / F-score) The distribution: (high / medium / low) novelty ratio
10
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Compare study – Performance Require 10 F-Score/precisionF-Score/recall
11
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Compare study – Prior probability 11
12
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Compare study – Prior probability 12
13
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Methodology – A new Framework Combine symmetic and asymmetric metrics Two problems: The scaling problem: comparable and consistent of metrics The combining strategy 13
14
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Experiments – Mixed metrics vs. individual metrics 14 M3 (jacc+new) tf.isf
15
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Experiments – Mixed metric M3 vs. individual metrics for novelty ratio 15
16
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Experiments – Mixed metric M3 vs. mixture of two symmetric metrics vs. mixture of two asymmetric metrics vs. mixture of all metrics for novelty ratio 16
17
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Experiments – Weight 17
18
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Conclusions Comparative study Different types of novelty metrics Symmetric: cosine / Jaccard Asymmetric: new word count / overlap Observes Its strengths Introduce Mixture of two types of novelty metrics More stable than using individual metric 18
19
Intelligent Database Systems Lab N.Y.U.S.T. I. M. 19
20
Intelligent Database Systems Lab N.Y.U.S.T. I. M. Comments Advantage A Comparative study Mixture Intuitive Drawback … Application Novelty mining 20
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.