1. Relevance in ISR Peter Ingwersen Department of Information Studies
Royal School of Library and Information Science, Denmark

2. Agenda Brief history Types of relevance
Higher order of relevance
Experimental relevance measures
Socio-cognitive relevance
Concluding discussion
MSC-course
Peter Ingwersen, 2007

3. Up until
Cranfield:
Topicality - Utility (usefulness): Cleverdon
Saracevic 1975 – Swanson: Objective vs. Subjective relevance
Schamber 1990 / 1994 (JASIS):
Situational relevance vs. topicality
Harter 1992:
Psychological relevance (Sperber/Wilson)
Ørom & Cosijn/Ingwersen 2000:
Socio-cognitive relevance
MSC-course
Peter Ingwersen, 2007

4. Higher Order Relevances
Relevance is a multi-dimensional, multi-layered, non-binary, subjective, and dynamic phenomenon
E.g. topical, pertinence, situational, and socio-cognitive relevance.
Incorporation of higher order relevance
improves the realism of evaluation
brings human users and interaction into evaluation
Are cognitive/emotional perceptions and interpretations of document features
MSC-course
Peter Ingwersen, 2007

5. Levels of Relevance Types -1 Saracevic, 1996
System or Algorithmic relevance:
query-object (objectivity)
Topical relevance: aboutness relation of
query-object (interpretation)
Pertinence (cognitive relevance): perceived correspondence of information need-objects
MSC-course
Peter Ingwersen, 2007

6. Levels of Relevance Types -2 Saracevic, 1996; rev
Levels of Relevance Types -2 Saracevic, 1996; rev. Cosijn/Ingwersen, 2000
Situational relevance:
relation as perceived between work task or interest situation or problem and objects
Affective relevance: intentionality-objects
Saracevic´ idea revised by Cosijn/Ingwersen
Socio-Cognitive relevance: objects perceived as meaningful by users in social interaction
MSC-course
Peter Ingwersen, 2007

7. Algorithmic relevance
The ranked output of information objects - ranked by relevance scores
Commonly judged against expert assessor´s binary relevance assessments of the pooled documents (all scale variances normalized)
Assessor´s judgement seen as topicality & objective - is of course intellectual
MSC-course
Peter Ingwersen, 2007

8. Topical relevance Contains interpretations
Issues of nature of aboutness & meaning
Inconsistency among several assessors (yet: see Sigir 98 paper by Vorhees)
Used for relative performance indications
Why not apply the mean of the algorithmic output - normalized across sites (TREC) ?
MSC-course
Peter Ingwersen, 2007

9. Pertinence
Requires knowledge of intrinsic information need for an observer - difficult to obtain
May not be achieved in case of ill-defined needs
Involves other features of objects than simply topical ones (novelty - authorship - cognitive authority of carriers)
MSC-course
Peter Ingwersen, 2007

10. Situational relevance
Relates to the WORK TASK (interest) SITUATION - perceived usefulness of objects
Individual relevance assessment in IS&R
as document assessment; peer review
As document features: references or outlinks
Is observable by users (Borlund, 2000) - also when work tasks are simulated
MSC-course
Peter Ingwersen, 2007

11. Experimental issues on relevance in IS&R
MSC-course
Peter Ingwersen, 2007

12. Socio-Cognitive Relevance
Proposed by Ørom (JoD, Jan. 2000; Hjørland, 1997) as associated with the socio-cultural / epistemological / domain context.
Discussed by Cosijn/Ingwersen (IPM, May 2000) as related to organisational or social strategies, group situations & perceptions (individuals in social interaction)
MSC-course
Peter Ingwersen, 2007

13. Socio-Cognitive Relevance - 2
In the perspective of information use:
creation of scientific references and Web outlinks is commonly situational (usefulness of cited work)
CITATIONS or INLINKS over time are manifestations of socio-cognitive relevance in domain
Relation to informetrics & scientometrics and knowledge representation
MSC-course
Peter Ingwersen, 2007

14. Socio-Cognitive Relevance - 3
Strength of - e.g. - author (journal …) co-citation (co-linking) shows (socio)-cognitive authority
In the perspective of information creation, selection, processing or presentation:
Multi-authorships
Peer reviews used in editorial decisions
Programme Committee decisions (conf.)
MSC-course
Peter Ingwersen, 2007

15. Simplistic model of a Selector group in IS&R as cognitive agent
Information
objects IT
Structure
Interface
Program
Committee
Members
Org.
Cultural
Cognitive transformation (selection & structure of objects)
Social interaction over time (scientific domain)
Interactions with submissions/reviews/preogram structure
Cognitive transformations & influence over time (submissions)
Social
Context
MSC-course
Peter Ingwersen, 2007

16. Socio-cognitive assessments:
Create features of isness of information objects, e.g.
Journal name – conference… - acceptance
Publication year / date – publisher decision
Database name(s) - inclusion
Corporate source & Geo-location – employment
Such features are represented and accessible
Dependent on Media - Domains - Time
MSC-course
Peter Ingwersen, 2007

17. Concluding remarks
Topicality - Pertinence - Situational relevances are
Increasingly subjective
Typically ad hoc manifestations
Situational Rel. difficult to apply in system due to lack of proper features (usefulness?)
Yet: references and outlinks – document structure
Socio-cognitive relevance turns into
objectivity or confined subjectivity
and manifestations over time
lead to perceptions of cognitive authority
MSC-course
Peter Ingwersen, 2007