Information Retrieval Evaluation and the Retrieval Process

Why evaluate an IR system? To select between alternative systems To determine if a system meets expressed and unexpressed needs of current users and non-users To improve IR systems and determine if improvement actually occurred To develop cost models

4 levels of evaluation - Lancaster Effectiveness Benefits Cost effectiveness Cost benefits

Effectiveness What a system does well, e.g., percentage of reference questions answered accurately, the recall and precision of a literature search There are a number of measures of effectiveness

Measuring Effectiveness

Measures of Effectiveness Recall Precision Relevance Pertinence or utility Novelty ratio Fallout and noise Timeliness Coverage Generality

Recall and Precision Ratios Recall a/(a+c): proportion of relevant items retrieved out of the total number of relevant items contained in a database Precision a/(a+b): a signal-to-noise ratio-- proportion of retrieved materials that are relevant to a query Used together, the 2 ratios express the filtering capacity of the system Recall and precision tend to be inversely related

Relevance and Pertinence Relevance (or generality ratio) (a+c)/(a+b+c+d): the number or proportion of materials in a system that are relevant to a query. Can be hard to ascertain without scanning the entire database. Pertinence: the relationship between a document and an information need. Utility refers to the subset of a that is actually used.

Novelty Ratio, Fallout and Noise Novelty ratio: a subset of a that is actually new to the person evaluating relevance Fallout and Noise: the subset b of retrieved items that are not relevant

Timeliness, Coverage, and Generality Timeliness and coverage: factors that affect assessments of relevance and pertinence Generality: the number of documents related to a particular request in the entire database. The more dense the ratio, the easier a search should be Accuracy

Criteria Commonly Used to Evaluate Retrieval Performance Recall Precision User effort  Amount of time a user spends conducting a search  Amount of time a user spends negotiating his inquiry and then separating relevant from irrelevant items Response time Benefits Search costs Cost effectiveness Cost benefits

Objective vs. Subjective Knowledge Factual or artifactual knowledge vs. how knowledge is constructed or modeled within an individual’s mind Subjective knowledge (and therefore relevance judgments) varies from person to person, e.g., individual aesthetic judgments or problem solving methods

Benefits What good a system does, e.g., how an information system benefits its users Hard to measure

Search Costs Economics of using different databases Using natural language indexing can shift effort onto the searcher

Cost Effectiveness Relationship of cost criteria to quality criteria, e.g., unit cost per relevant or new item retrieved

Cost Benefits Cost savings through use of one information system over another Increased, or avoidance of loss of, productivity Improved decision-making or reduction of personnel needed to make decisions Avoidance of duplication of effort

Components of an Evaluation 1. Defining the scope of the evaluation - Formative vs. summative 2. Designing the evaluation program 3. Execution of the evaluation 4. Analysis and interpretation of the results 5. Modifying the system based on the results 6. Iteration if necessary (go back to step 3)

Real Life vs. Experimental Systems Experiments and benchmark tests -  standardized collections, queries, and relevance judgments  tested against multiple systems  evaluated on recall and precision  biases often built into system design Predictive evaluation -  expert reviews  usage simulation such as walthroughs Real life -  observing users’ interactions with system  eliciting users’ opinions

Classic IR Model - Bates Document --> Document representation  matched up with Query <-- Information need

Problems with Classic IR Model Users cannot use their own language Different users have different needs Users have different information needs at different times Users are not always able to read and write Information need may evolve during the search process Some users are not concerned about precision and recall Users may want to eliminate known items Users may want more cues to assist in assessing relevance

Other factors influencing use Accessibility - physical, intellectual, and psychological - and ease of use are the most important determinants of whether an information service is used Principle of Least Effort Perceived technical quality also affects the choice of first source Perceptions of accessibility ar einfluenced by experience

Berrypicking Search queries are not static, but evolve Searchers gather information in bits and pieces Searchers use a variety of search techniques Searchers use a variety of other sources as well as databases

Search Strategies Footnote chasing Citation searching Journal run Area scanning Subject searches in bibliographies, abstracts, and indexes Author searching

Making Retrieval More Effective The more techniques used, the more effective a search is likely to be Users should be able to search in ways that are already familiar or that they have found to be effective A visual representation of the contents of a system may aid users in orienting themselves