1. the d-Rank project Aggregating rankings for retrieval of scientific publications in the HEP domain
Martin Rajman, Martin Vesely
Ecole Polytechnique Fédérale Lausanne (EPFL)
Jean-Yves Le Meur
CERN

2. Contents Motivations, objectives, and context
Specialized search engines
General purpose and specialized ranking criteria
Aggregating ranking methods
Experimental results
Conclusions and future work

3. Motivations
General purpose relevance criteria do not cover well specialized retrieval needs
Specialized retrieval should also take into account multiple, community and domain specific, ranking criteria
Specialized ranking criteria should rely on specific information available for the targeted document collection (meta-data) and users (query logs)

4. General objectives
Identify relevance criteria specific for document retrieval in the HEP community to design specialized ranking methods tailored for the needs of this community
Design adaptive aggregation mechanisms to extend general purpose rankings with specialized, HEP community specific, rankings relying on document meta-data and query logs

5. Context of the project CERN Document Server:
Carried out in collaboration between EPFL and CERN
Builds upon the CDS Invenio software
Works with the document collection of the
Ca. 1Mio documents in the HEP domain
Ca. 800k preprints, articles, theses, reports and scientific notes (our focus)
CERN Document Server:

6. Specialized search engines
Full document Collection
General Purpose SE
General purpose ranking
Sub-Collection
Specialized ranking
Generic resources
(e.g. download frequency)
Specialized resource
(e.g. publication date)

7. General purpose ranking methods
Download frequency (baseline in d-Rank)
Frequently downloaded documents are interesting for future retrieval
Word similarity
Traditionally used for ranking, many formulas based on the TF.IDF paradigm:
Cosine, Okapi BM25, etc.
Often used as a binary (threshold based) filter

8. Specialized ranking methods
Freshness (i.e. publication date)
often preferred in the HEP community
Citations
Citation frequency
Journal Impact Factor
Authors’ reputation
Hirsch index(es)

9. download-Hirsch index (d-Hirsch)
Similar to the standard Hirsch index
Number of downloads instead of number of citations as underlying measure
Why download frequency?
Known sooner than citations
Might predicts citations
Expected benefit: adds some sort of authors’ reputation to the simple download frequency

10. download-Hirsch index
d-Hirsch index for an author a
d-Hirsch index for a paper p

11. Aggregating ranking methods
Requirements for the constituents of an aggregated ranking
1 Good individual performance
Ranking methods that are performing well individually are preferred
2 Complementary wrt. other constituents
Ranking methods that provide true added value are preferred

12. Aggregating ranking methods
Two steps:
Normalization of ranking scores
Aggregation of normalized scores
Two variants
Local
Normalization is performed only on the documents matching the query.
Global
Normalization is performed on the whole document collection.

13. Score normalization
Relative score frequency f(s) depends on observed number of occurrences of score S and a sample size N:
Probability density kernel p(u), such that:
Probability that the observed score S has value s is then estimated by:
Normalized score then corresponds to the probability P(S≤s) that is obtained from the associated cumulative distribution function:

14. Score normalization Normalized scores: Have a fixed [0,1] range
Consitute a measurement scale that is robust w.r.t. any change of origin
When multiplied by the sample size N they can be interpreted as an estimate of the expected rank associated with the score value s

15. Rank aggregation Symmetric aggregation Sequential aggregation
weighted arithmetic mean
Sequential aggregation
weighted geometric mean
Parameters to be learnt from the CDS referential

16. CDS Referential Automatically acquired from the user access logs
Consisting of triplets
Query + document + relevance judgment
Can be represented by query-document matrix where values express the query-document relevance
Potential source for learning the ranking function

17. Experimental results carried out on the CDS document collection
considered ranking methods: Download frequency (D), Freshness (F), and download-Hirsh (H)
aggregation through weighted arithmetic mean of global or local ranks
evaluation based on the CDS referential

18. Measuring complementarity
The less overlap exists between two ranking methods the more complementary they are.
(F)
(D)
(H)
Potentially interesting documents inaccessible by default ranking
Ranking method 1
Ranking method 2
(D)
(F)
20%
(H)
19%
16%
(D+F)
D = download frequency
F = freshness
H = download-Hirsch index

19. Evaluation metrics Success@10 Mean Reciprocal Rank
Relevant document found within the top-10 documents (first result page)
Mean Reciprocal Rank
At which position, on average, the first relevant document is found

20. Performance of ranking methods
Performance of individual ranking methods
D = download frequency
F = freshness
H = download-Hirsch index
Performance of (D+F) wLR aggregation
(w(D) = w(F) = 0.5)
Ranking
method
MRR
(D)
0.532
0.229
(F)
0.593
0.173
(H)
0.365
0.138
(D+F)
wGR
0.599
0.226
wLR
0.623
0.266

21. Conclusions
The (F) and (H) specialized ranking methods indeed provide complementary information wrt. the (D) baseline (only around 20% overlap)
The aggregated (D +LR F) ranking method leads to statistically significant performance increase wrt. the (D) baseline (relative increase of 16.2% for Mean Reciprocal Rank and of 5.1% for
The Download-Hirsch index for document ranking seems to be a potentially interesting resource for document ranking but needs to be further investigated

22. Future work
Further formalize the probabilistic model for rank aggregation
Further evaluate the download-Hirsch index for document ranking in scientific publication databases
Design an automated learning procedure to learn the aggregation weights (and the ranking function in general) from the CDS referential