Presentation is loading. Please wait.

Presentation is loading. Please wait.

Microsoft Research1 A Statistical Analysis of the Precision-Recall Graph Ralf Herbrich, Hugo Zaragoza, Simon Hill. Microsoft Research, Cambridge University,

Published byRoss Moore Modified over 9 years ago

Similar presentations

Presentation on theme: "Microsoft Research1 A Statistical Analysis of the Precision-Recall Graph Ralf Herbrich, Hugo Zaragoza, Simon Hill. Microsoft Research, Cambridge University,"— Presentation transcript:

1 Microsoft Research1 A Statistical Analysis of the Precision-Recall Graph Ralf Herbrich, Hugo Zaragoza, Simon Hill. Microsoft Research, Cambridge University, UK.

2 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 2 Overview 2-class ranking Average-Precision From points to curves Generalisation bound Discussion

3 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 3 “Search” cost-functions Maximise the number of relevant documents found in the top 10. Maximise the number of relevant documents at the top (e.g. weight inversely proportional to rank) Minimise the number of documents seen by the user until he is satisfied.

4 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 4 Motivation Why should 45 August, 2003 work for document categorisation? Why should any algorithm obtain good generalisation average-precision? How to devise algorithms to optimise rank dependant loss-functions?

5 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 5 2-class ranking problem {y=1} X,Y Mapping: X  R Relevancy:P(y=1|x)  P(y=1|f(x))

6 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 6 Collection samples A collection is a sample: z= ((x 1,y 1 ),...,(x m,y m ))  (X x {0,1}) m where: y = 1 if the document x is relevant to a particular topic, z is drawn from the (unknown) distribution π XY let k denote the number of positive examples

7 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 7 Ranking the collection We are given a scoring function f :X  R This function imposes an order in the collection: (x (1),…, x (m) )such that : f(x (1) ) > … > f(x (m) ) Hits (i 1,…, i k ) are the indices of the positive y (j). f(x(i))f(x(i)) y (i) = 1 1 0 1 0 0 1 0 0 0 i j = 1 2 4 7

8 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 8 Classification setting If we threshold the function f, we obtain a classification: Recall: Precision: f(x(i))f(x(i)) t

9 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 9 Precision.vs. PGC PGC PRECISION

10 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 10 The Precision-Recall Graph After reordering: f(x(i))f(x(i))

11 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 11 Graph Summarisations 00.20.40.60.81 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Recall Precision Break-Even point

12 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 12 Precision-Recall Example

13 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 13 overfitting? Average Precision (TEST SET) Average Precision (TAIN SET)

14 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 14 Overview 2-class ranking Average-Precision From points to curves Generalisation bound Discussion

15 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 15 From point to curve bounds There exist SVM margin-bounds [Joachims 2000] for precision and recall. They only apply to a single (unknown a priori) point of the curve! Precision Recall

16 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 16 Max-Min precision-recall

17 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 17 Max-Min precision-recall (2)

18 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 18 Features of Ranking Learning We cannot take differences of ranks. We cannot ignore the order of ranks.  Point-wise loss functions do not capture the ranking performance!  ROC or precision-recall curves do capture the ranking performance.  We need generalisation error bounds for ROC and precision-recall curves

19 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 19 Generalisation and Avg.Prec. How far can the observed Avg.Prec. A(f,z) be from the expected average A(f) ? How far can train and test Avg.Prec.?

20 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 20 Approach 1. McDiarmid’s inequality: For any function g:Z n  R with stability c, for all probability measures P with probability at least 1-δ over the IID draw of Z

21 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 21 Approach (cont.) 2. Set n= 2m and call the two m-halves Z 1 and Z 2. Define g i (Z):=A(f,Z i ). Then, by IID :

22 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 22 Bounding A(f,z) - A(f,z i ) 1. How much does A(f,z) change if we can alter one sample (x i,y i )? We need to fix the number of positive examples in order to answer this question! e.g. if k=1, the change can be from 0 to 1.

23 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 23 Stability Analysis Case 1: y i =0 Case 2: y i =1

24 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 24 Main Result Theorem: For all probability measures, for all f:X  R, with probability at least 1- δ over the IID draw of a training and test sample both of size m, if both training sample z and test sample z contain at least αm positive examples for all α  (0,1), then:

25 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 25

26 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 26 Positive results First bound which shows that asymptotically training and test set performance (in terms of average precision) converge! The effective sample size is only the number of positive examples. The proof can be generalised to arbitrary test sample sizes. The constants can be improved.

27 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 27 Open questions How can we let k change, so as to investigate: What algorithms could be used to directly maximise A(f,z) ?

28 Hugo Zaragoza. AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. June 2003.AMS-IMS-SIAM Joint Summer Research Conference on Machine Learning, Statistics, and Discovery. 28 Conclusions Many problems require ranking objects in some degree. Ranking learning requires to consider non- point-wise loss functions. In order to study the complexity of algorithms we need to have large deviation inequalities for ranking performance measures.

Download ppt "Microsoft Research1 A Statistical Analysis of the Precision-Recall Graph Ralf Herbrich, Hugo Zaragoza, Simon Hill. Microsoft Research, Cambridge University,"

Similar presentations

A Support Vector Method for Optimizing Average Precision

A Support Vector Method for Optimizing Average Precision
1 A Statistical Analysis of the Precision-Recall Graph Ralf Herbrich Microsoft Research UK Joint work with Hugo Zaragoza and Simon Hill.

1 A Statistical Analysis of the Precision-Recall Graph Ralf Herbrich Microsoft Research UK Joint work with Hugo Zaragoza and Simon Hill.
Pseudo-Relevance Feedback For Multimedia Retrieval By Rong Yan, Alexander G. and Rong Jin Mwangi S. Kariuki 2008-11629.

Pseudo-Relevance Feedback For Multimedia Retrieval By Rong Yan, Alexander G. and Rong Jin Mwangi S. Kariuki
Active Learning with Feedback on Both Features and Instances H. Raghavan, O. Madani and R. Jones Journal of Machine Learning Research 7 (2006) Presented.

Active Learning with Feedback on Both Features and Instances H. Raghavan, O. Madani and R. Jones Journal of Machine Learning Research 7 (2006) Presented.
Introduction Simple Random Sampling Stratified Random Sampling

Introduction Simple Random Sampling Stratified Random Sampling
On Complexity, Sampling, and -Nets and -Samples. Range Spaces A range space is a pair, where is a ground set, it’s elements called points and is a family.

On Complexity, Sampling, and -Nets and -Samples. Range Spaces A range space is a pair, where is a ground set, it’s elements called points and is a family.
Retrieval Evaluation J. H. Wang Mar. 18, 2008. Outline Chap. 3, Retrieval Evaluation –Retrieval Performance Evaluation –Reference Collections.

Retrieval Evaluation J. H. Wang Mar. 18, Outline Chap. 3, Retrieval Evaluation –Retrieval Performance Evaluation –Reference Collections.
PAC-Bayesian Theorems for Gaussian Process Classifications Matthias Seeger University of Edinburgh.

PAC-Bayesian Theorems for Gaussian Process Classifications Matthias Seeger University of Edinburgh.
Search Engines Information Retrieval in Practice All slides ©Addison Wesley, 2008.

Search Engines Information Retrieval in Practice All slides ©Addison Wesley, 2008.
Statistics and Quantitative Analysis U4320

Statistics and Quantitative Analysis U4320
The Mean Square Error (MSE):. Now, Examples: 1) 2)

The Mean Square Error (MSE):. Now, Examples: 1) 2)
0 Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John.

0 Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John.
Reduced Support Vector Machine

Reduced Support Vector Machine
1 Introduction to Kernels Max Welling October 1 2004 (chapters 1,2,3,4)

1 Introduction to Kernels Max Welling October (chapters 1,2,3,4)
Retrieval Evaluation. Brief Review Evaluation of implementations in computer science often is in terms of time and space complexity. With large document.

Retrieval Evaluation. Brief Review Evaluation of implementations in computer science often is in terms of time and space complexity. With large document.
Evaluating Hypotheses

Evaluating Hypotheses
Retrieval Evaluation: Precision and Recall. Introduction Evaluation of implementations in computer science often is in terms of time and space complexity.

Retrieval Evaluation: Precision and Recall. Introduction Evaluation of implementations in computer science often is in terms of time and space complexity.
Retrieval Evaluation. Introduction Evaluation of implementations in computer science often is in terms of time and space complexity. With large document.

Retrieval Evaluation. Introduction Evaluation of implementations in computer science often is in terms of time and space complexity. With large document.
PROBABILITY AND SAMPLES: THE DISTRIBUTION OF SAMPLE MEANS.

PROBABILITY AND SAMPLES: THE DISTRIBUTION OF SAMPLE MEANS.
Bing LiuCS Department, UIC1 Learning from Positive and Unlabeled Examples Bing Liu Department of Computer Science University of Illinois at Chicago Joint.

Bing LiuCS Department, UIC1 Learning from Positive and Unlabeled Examples Bing Liu Department of Computer Science University of Illinois at Chicago Joint.

Similar presentations

Ads by Google