Relevance in ISR Peter Ingwersen Department of Information Studies Royal School of Library and Information Science, Denmark pi@db.dk

Agenda Brief history Types of relevance Higher order of relevance Experimental relevance measures Socio-cognitive relevance Concluding discussion MSC-course Peter Ingwersen, 2007

Up until 1990.. Cranfield: Topicality - Utility (usefulness): Cleverdon Saracevic 1975 – Swanson: Objective vs. Subjective relevance Schamber 1990 / 1994 (JASIS): Situational relevance vs. topicality Harter 1992: Psychological relevance (Sperber/Wilson) Ørom & Cosijn/Ingwersen 2000: Socio-cognitive relevance MSC-course Peter Ingwersen, 2007

Higher Order Relevances Relevance is a multi-dimensional, multi-layered, non-binary, subjective, and dynamic phenomenon E.g. topical, pertinence, situational, and socio-cognitive relevance. Incorporation of higher order relevance improves the realism of evaluation brings human users and interaction into evaluation Are cognitive/emotional perceptions and interpretations of document features MSC-course Peter Ingwersen, 2007

Levels of Relevance Types -1 Saracevic, 1996 System or Algorithmic relevance: query-object (objectivity) Topical relevance: aboutness relation of query-object (interpretation) Pertinence (cognitive relevance): perceived correspondence of information need-objects MSC-course Peter Ingwersen, 2007

Levels of Relevance Types -2 Saracevic, 1996; rev Levels of Relevance Types -2 Saracevic, 1996; rev. Cosijn/Ingwersen, 2000 Situational relevance: relation as perceived between work task or interest situation or problem and objects Affective relevance: intentionality-objects Saracevic´ idea revised by Cosijn/Ingwersen Socio-Cognitive relevance: objects perceived as meaningful by users in social interaction MSC-course Peter Ingwersen, 2007

Algorithmic relevance The ranked output of information objects - ranked by relevance scores Commonly judged against expert assessor´s binary relevance assessments of the pooled documents (all scale variances normalized) Assessor´s judgement seen as topicality & objective - is of course intellectual MSC-course Peter Ingwersen, 2007

Topical relevance Contains interpretations Issues of nature of aboutness & meaning Inconsistency among several assessors (yet: see Sigir 98 paper by Vorhees) Used for relative performance indications Why not apply the mean of the algorithmic output - normalized across sites (TREC) ? MSC-course Peter Ingwersen, 2007

Pertinence Requires knowledge of intrinsic information need for an observer - difficult to obtain May not be achieved in case of ill-defined needs Involves other features of objects than simply topical ones (novelty - authorship - cognitive authority of carriers) MSC-course Peter Ingwersen, 2007

Situational relevance Relates to the WORK TASK (interest) SITUATION - perceived usefulness of objects Individual relevance assessment in IS&R as document assessment; peer review As document features: references or outlinks Is observable by users (Borlund, 2000) - also when work tasks are simulated MSC-course Peter Ingwersen, 2007

Experimental issues on relevance in IS&R MSC-course Peter Ingwersen, 2007

Socio-Cognitive Relevance Proposed by Ørom (JoD, Jan. 2000; Hjørland, 1997) as associated with the socio-cultural / epistemological / domain context. Discussed by Cosijn/Ingwersen (IPM, May 2000) as related to organisational or social strategies, group situations & perceptions (individuals in social interaction) MSC-course Peter Ingwersen, 2007

Socio-Cognitive Relevance - 2 In the perspective of information use: creation of scientific references and Web outlinks is commonly situational (usefulness of cited work) CITATIONS or INLINKS over time are manifestations of socio-cognitive relevance in domain Relation to informetrics & scientometrics and knowledge representation MSC-course Peter Ingwersen, 2007

Socio-Cognitive Relevance - 3 Strength of - e.g. - author (journal …) co-citation (co-linking) shows (socio)-cognitive authority In the perspective of information creation, selection, processing or presentation: Multi-authorships Peer reviews used in editorial decisions Programme Committee decisions (conf.) MSC-course Peter Ingwersen, 2007

Simplistic model of a Selector group in IS&R as cognitive agent Information objects IT Structure Interface Program Committee Members Org. Cultural Cognitive transformation (selection & structure of objects) Social interaction over time (scientific domain) Interactions with submissions/reviews/preogram structure Cognitive transformations & influence over time (submissions) Social Context MSC-course Peter Ingwersen, 2007

Socio-cognitive assessments: Create features of isness of information objects, e.g. Journal name – conference… - acceptance Publication year / date – publisher decision Database name(s) - inclusion Corporate source & Geo-location – employment Such features are represented and accessible Dependent on Media - Domains - Time MSC-course Peter Ingwersen, 2007

Concluding remarks Topicality - Pertinence - Situational relevances are Increasingly subjective Typically ad hoc manifestations Situational Rel. difficult to apply in system due to lack of proper features (usefulness?) Yet: references and outlinks – document structure Socio-cognitive relevance turns into objectivity or confined subjectivity and manifestations over time lead to perceptions of cognitive authority MSC-course Peter Ingwersen, 2007