Tefko Saracevic1 EVALUATION in searching Requirements Criteria

Slides:



Advertisements
Similar presentations
Evaluation of electronic resources. Review of Internet quality issues Nearly anyone can publish information on the Internet so –academic journals sit.
Advertisements

1 SESSION 3 FORMAL ASSESSMENT TASKS CAT and IT ASSESSMENT TOOLS.
SECOND MIDTERM REVIEW CS 580 Human Computer Interaction.
Enhancing Data Quality of Distributive Trade Statistics Workshop for African countries on the Implementation of International Recommendations for Distributive.
IS214 Recap. IS214 Understanding Users and Their Work –User and task analysis –Ethnographic methods –Site visits: observation, interviews –Contextual.
Introduction to Research Methodology
How well did the assessment task do what we wanted it to do? Janina Drazek Manager — Assessment & Comparability, QCAR Queensland Studies Authority.
Consistency of Assessment
© Tefko Saracevic, Rutgers University1 digital libraries and human information behavior Tefko Saracevic, Ph.D. School of Communication, Information and.
© Tefko Saracevic1 Interaction in information retrieval There is MUCH more to searching than knowing computers, networks & commands, as there is more.
© Tefko Saracevic, Rutgers University1 Search strategy & tactics Governed by effectiveness & feedback.
© Tefko Saracevic, Rutgers University adapted for sectoin 21 PRINCIPLES OF SEARCHING 17:610:530 (02) Paul Kantor SCILS, Rm. 307 (732) /Ext
© Tefko Saracevic, Rutgers University1 1.Discussion 2.Information retrieval (IR) model (the traditional models). 3. The review of the readings. Announcement.
© Tefko Saracevic, Rutgers University1 Interaction in information retrieval There is MUCH more to searching than knowing computers, networks & commands,
Midterm Review Evaluation & Research Concepts Proposals & Research Design Measurement Sampling Survey methods.
© Tefko Saracevic, Rutgers University1 metadata considerations for digital libraries.
Measuring the quality of academic library electronic services and resources Jillian R Griffiths Research Associate CERLIM – Centre for Research in Library.
INFO 624 Week 3 Retrieval System Evaluation
© Tefko Saracevic, Rutgers University 1 EVALUATION in searching IR systems Digital libraries Reference sources Web sources.
© Tefko Saracevic, Rutgers University 1 evaluating information on the web Tefko Saracevic School of Communication, Information and Library Studies Rutgers.
© Tefko Saracevic, Rutgers University1 digital libraries and human information behavior Tefko Saracevic, Ph.D. School of Communication, Information and.
© Tefko Saracevic1 Search strategy & tactics Governed by effectiveness&feedback.
© Tefko Saracevic, Rutgers University1 DIGITAL LIBRARIES 17:610:553 Tefko Saracevic Michael Lesk
Getting Started Position Papers. Getting Started w Goal: Create a persuasive position paper that makes clear claims supported by good reasons and credible.
4. Interaction Design Overview 4.1. Ergonomics 4.2. Designing complex interactive systems Situated design Collaborative design: a multidisciplinary.
Business research methods: data sources
© Tefko Saracevic, Rutgers University1 PRINCIPLES OF SEARCHING 17:610:530 (01) Tefko Saracevic SCILS, Rm. 306 (732) /Ext. 8222
© Tefko Saracevic, Rutgers University1 Presentation of search results A search is not finished with the search Guidelines for deliverables.
What is a ‘collection’ in digital libraries? Changing concepts, objects & economics © Tefko Saracevic Rutgers University
© Tefko Saracevic, Rutgers University1 digital libraries and human information behavior Tefko Saracevic, Ph.D. School of Communication, Information and.
Evaluation of Evaluation in Information Retrieval - Tefko Saracevic Historical Approach to IR Evaluation.
How to write a publishable qualitative article
Measuring Learning Outcomes Evaluation
Vocabulary & languages in searching
Educational Research: Action Research in Schools
Soo Young Rieh School of Information University of Michigan Information Ethics Roundtable Misinformation and Disinformation April 3-4, 2009 University.
Foreign language and English as a Second Language: Getting to the Common Core of Communication. Are we there yet? Marisol Marcin
Evaluation of digital Libraries: Criteria and problems from users’ perspectives Article by Hong (Iris) Xie Discussion by Pam Pagels.
© Tefko Saracevic, Rutgers University1 Mediation in librarianship & information retrieval Reference interview Human-human interaction Question negotiation.
Interstate New Teacher Assessment and Support Consortium (INTASC)
1 © Tefko Saracevic, Rutgers University Evaluation of library and information services (LIS): an overview Contexts Approaches Levels Requirements Measures.
Classroom Assessments Checklists, Rating Scales, and Rubrics
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 United States License Creative Commons Attribution-NonCommercial-ShareAlike.
Ways for Improvement of Validity of Qualifications PHARE TVET RO2006/ Training and Advice for Further Development of the TVET.
ELA Common Core Shifts. Shift 1 Balancing Informational & Literary Text.
WELNS 670: Wellness Research Design Chapter 5: Planning Your Research Design.
Information Retrieval Evaluation and the Retrieval Process.
1 of 27 How to invest in Information for Development An Introduction Introduction This question is the focus of our examination of the information management.
Research: Conceptualization and Measurement Conceptualization Steps in measuring a variable Operational definitions Confounding Criteria for measurement.
Tackling the Complexities of Source Evaluation: Active Learning Exercises That Foster Students’ Critical Thinking Juliet Rumble & Toni Carter Auburn University.
Tefko Saracevic 1 Quality of information Considerations for library & information services in the networked world Tefko Saracevic, PhD
Copyright 2010, The World Bank Group. All Rights Reserved. Principles, criteria and methods Part 2 Quality management Produced in Collaboration between.
JS Mrunalini Lecturer RAKMHSU Data Collection Considerations: Validity, Reliability, Generalizability, and Ethics.
Digital Libraries1 David Rashty. Digital Libraries2 “A library is an arsenal of liberty” Anonymous.
L&I SCI 110: Information science and information theory Instructor: Xiangming(Simon) Mu Sept. 9, 2004.
The Research Problem and Objectives Lecture 6 1. Organization of this lecture Research Problem & Objectives: Research and Decision/Action Problems Importance.
Chapter. 3: Retrieval Evaluation 1/2/2016Dr. Almetwally Mostafa 1.
Paper III Qualitative research methodology. Objective 1.2 Explain strengths and limitations of a qualitative approach to research?
Colby Smart, E-Learning Specialist Humboldt County Office of Education
National Science Education Standards. Outline what students need to know, understand, and be able to do to be scientifically literate at different grade.
Copyright 2010, The World Bank Group. All Rights Reserved. Principles, criteria and methods Part 1 Quality management Produced in Collaboration between.
Design Evaluation Overview Introduction Model for Interface Design Evaluation Types of Evaluation –Conceptual Design –Usability –Learning Outcome.
Tefko Saracevic This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 United States License Creative Commons Attribution-NonCommercial-ShareAlike.
Quality Assurance processes
Chapter 8 Research: Gathering and Using Information.
© 2008 Tefko Saracevic, Rutgers University Tefko Saracevic, Ph.D.
Measuring Data Quality and Compilation of Metadata
digital libraries and human information behavior
6 Chapter Training Evaluation.
What is a ‘collection’ in digital libraries?
Presentation transcript:

Tefko Saracevic1 EVALUATION in searching Requirements Criteria

Central ideas  Evaluation is an integral part of searching  But there a number of: o contexts & approaches to evaluation o requirements for evaluation o criteria used in evaluation Tefko Saracevic 2

ToC 1.Importance, definitions 2.Contexts & approaches 3.Requirements for evaluation 4.Web evaluation and some pretty pictures at the end Tefko Saracevic 3

1. Importance, definitions Place of evaluation Tefko Saracevic 4

5 Definition of evaluation Dictionary: 1. assessment of value the act of considering or examining something in order to judge its value, quality, importance, extent, or condition In searching: assessment of search results on basis of given criteria as related to users and use criteria may be specified by users or derived from professional practice, other sources or standards Results are judged & with them the whole process, including searcher & searching

Tefko Saracevic 6 Importance of evaluation  Integral part of searching o always there - wanted or not  no matter what user will in some way or other evaluate what obtained o could be informal or formal  Growing problem for all o information explosion makes finding “good” stuff very difficult  Formal evaluation part of professional job & skills o requires knowledge of evaluation criteria, measures, methods o more & more prized

Tefko Saracevic 7 Place of evaluation User Inf. need Evaluation Search Results

Tefko Saracevic 8 General application  Evaluation (as discussed here) is applicable to results from a variety of information systems: o information retrieval (IR) systems, e.g. Dialog, Scopus … o sources included in digital libraries, e.g. Rutgers o reference services e.g. in libraries or commercial on the web o web sources e.g. as found on many domain sites  Many approaches, criteria, measures, methods are similar & can be adapted for specific source or information system

2. Contexts & approaches Broad orientation Tefko Saracevic 9

10 Broad context Evaluating the role that an information system plays as related to: èSOCIETY - community, culture, discipline... èINSTITUTION - university, organization, company... èINDIVIDUALS - users & potential users (nonusers) Roles lead to broad, but hard questions as to what CONTEXT to choose for evaluation

Tefko Saracevic 11 Questions asked in different contexts  Social: o how well does an information system support social demands & roles?  hardest to evaluate  Institutional: o how well does it support institutional/organizational mission & objectives?  tied to objectives of institution  also hard to evaluate  Individual: o how well does it support inf. needs & activities of people?  most evaluations in this context

Tefko Saracevic 12 Approaches to evaluation  Many approaches exist o quantitative, qualitative … o effectiveness, efficiency... o each has strong & weak points  Systems approach prevalent o Effectiveness: How well does a system perform that for which it was designed? o Evaluation related to objective(s) o Requires choices:  Which objective, function to evaluate?

Tefko Saracevic 13 Approaches … (cont.)  Economics approach: o Efficiency: at what costs? o Effort, time also are costs o Cost-effectiveness: cost for a given level of effectiveness  Ethnographic approach o practices, effects within an organization, community o learning & using practices & comparisons

Tefko Saracevic 14 Prevalent approach  System approach used in many different ways & purposes – in evaluation of: o inputs to system & contents o operations of a system o use of a system o outputs from a system  Also, in evaluation of search outputs for given user(s) and use o applied on the individual level  derived from assessments from users or their surrogates, e.g. searchers o this is what searchers do most often o this is what you will apply in your projects

3. Requirements for evaluation No evaluation without them Tefko Saracevic 15

Tefko Saracevic 16 Five basic requirements for system evaluation Once a context is selected need to specify ALL five: 1. Construct oA system, process, source  a given IR system, web site, digital library...  what are you going to evaluate? 2. Criteria oto reflect objective(s) of searching  e.g. relevance, utility, satisfaction, accuracy, completeness, time, costs …  on basis of what will you make judgments? 3. Measure(s) oto reflect criteria in some quantity or quality  precision, recall, various Likert scales, $$$...  how are you going to express judgment?

Tefko Saracevic 17 Requirements … (cont.) 4. Measuring instrument orecording by users or user surrogates (e.g. you) on the measure  expressing if relevant or not, marking a scale, indicating cost  people are instruments – who will it be? 5. Methodology oprocedures for collecting & analyzing data  how are you going to get all this done?  Assemble the stuff to evaluate (construct)? Choose what criteria? Determine what measures to use to reflect the criteria? Establish who will judge and how will the judgment be done? How will you analyze results? Verify validity and reliability?

Tefko Saracevic 18 Requirements … (cont.)  Ironclad rule: No evaluation can proceed if not ALL five of these are specified!  Sometimes specification on some are informal & implied, but they are always there!

Tefko Saracevic Constructs  In IR research: most done on test collections & test questions o Text Retrieval Conference - TREC  evaluation of algorithms, interactions  reported in research literature  In practice: on use & user level: mostly done on operational collections & systems, web sites o e.g. Dialog, LexisNexis, various files  evaluation, comparison of various contents, procedures, commands,  user proficiencies, characteristics  evaluation of interactions  reported in professional literature

Tefko Saracevic Criteria  In IR: Relevance basic & most used criterion o related to the problem at hand  On user & use level: many other o utility, satisfaction, success, time, value, impact,...  Web sources o those + quality, usability, penetration, accessibility...  Digital libraries, web sites o those + usability

Tefko Saracevic Criteria - relevance  Relevance as criterion (as mentioned) o strengths:  intuitively understood, people know what it means  universally applied in information systems o weaknesses:  not static - changes dynamically, thus hard to pin down  tied to cognitive structure & situation of a user – possible disagreements  Relevance as area of study  basic notion in information science  many studies done about various aspects of relevance  Number of relevance types exist o indication of different relations  had to be specified which ones

Tefko Saracevic Criteria - usability  Increasingly used for web sites & digital libraries  General definition (ISO) “extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use”  Number of criteria o enhancing user performance o ease of operations o serving the intended purpose o learnability – how easy to learn, memorize? o losstness – how often got lost in using it? o satisfaction o and quite a few more

Tefko Saracevic Measures  in IR: Precision & recall preferred (treated in unit 4) o based on relevance o could be two or more dimensions  e.g. relevant–not relevant; relevant–partially relevant–not relevant  Problem with recall o how to find what's relevant in a file?  e.g. estimate; broad & narrow searching or union of many outputs then comparison  On use & user level o Likert scales - semantic differentials  e.g. satisfaction on a scale of 1 to x (1=not satisfied, x=satisfied) o observational measures  e.g. overlap, consistency

Tefko Saracevic 24 4.Instruments  People used as instruments o they judge relevance, scale...  But people who? o users, surrogates, analysts, domain experts, librarians...  How do relevance, utility... judges effect results? o who knows?  Reliability of judgments: o about % for experts

Tefko Saracevic Methods  Includes design, procedures for observations, experiments, analysis of results  Challenges: o Validity? Reliability? Reality?  Collection - selection? size?  Request - generation?  Searching - conduct?  Results - obtaining? judging? feedback?  Analysis - conduct? tools?  Interpretation - warranted? generalizable?

4. Web evaluation Criteria Tefko Saracevic 26

Tefko Saracevic 27 Evaluation of web sources  Web is value neutral o it has everything from diamonds to trash  Thus evaluation becomes imperative o and a primary obligation & skill of professional searchers – you o continues & expands on evaluation standards & skills in library tradition  A number of criteria are used o most derived from traditional criteria, but modified for the web, others added o could be found on many library sites  librarians provide the public and colleagues with web evaluation tools and guidelines as part of their services

Tefko Saracevic 28 Criteria for evaluation of web & Dlib sources  What? Content o What subject(s), topic(s) covered? o Level? Depth? Exhaustively? Specificity? Organization? o Timeliness of content? Up-to-date? Revisions? o Accuracy?  Why? Intention o Purpose? Scope? Viewpoint?  For? Users, use o Intended audience? o What need satisfied? o Use intended or possible? o How appropriate?

Tefko Saracevic 29 criteria...  Who done it? Authority o Author(s), institution, company, publisher, creator:  What authority? Reputation? Credibility? Trustworthiness? Refereeing?  Persistence? Will it be around?  Is it transparent who done it?  How? Treatment o Content treatment:  Readability? Style? Organization? Clarity? o Physical treatment:  Format? Layout? Legibility? Visualization? o Usability  Where? Access o How available? Accessible? Restrictions? o Links persistence, stability?

Tefko Saracevic 30 criteria...  How? Functionality o Searching, navigation, browsing? o Feedback? Links? o Output: Organization? Features? Variations? Control?  How much? Effort, economics o Time, effort in learning it? o Time, effort in using it o Price? Total costs? Cost-benefits?  In comparison to? Wider world o Other similar sources?  where & how similar or better results may be obtained?  how do they compare?

Tefko Saracevic 31 Quality Content coverage accuracy timeliness … Users, use audience need appropriateness … Functionality navigation features output … Effort in using it in learning it time, cost … Access availability persistence links … Treatment content layout visualization … Authority reputation credibility “About us” … Intention purpose scope viewpoint … Main criteria for web site evaluation

Tefko Saracevic 32 Evaluation: To what end?  To asses & then improve performance – MAIN POINT o to change searches & search results for better  To understand what went on o what went right, what wrong, what works, what doesn't & then change  To communicate with user o explain & get feedback  To gather data for best practices o conversely: eliminate or reduce bad ones  To keep your job o even more: to advance  To get satisfaction from job well done

Tefko Saracevic 33 Conclusions  Evaluation is a complex task o but also an essential part of being an information professional  Traditional approaches & criteria still apply o but new ones added or adapted to satisfy new sources, & new methods of access & use  Evaluation skills are in growing demand particularly because web is value neutral  Great professional skill to sell!

Tefko Saracevic 34 Evaluation perspectives - Rockwell

Tefko Saracevic 35 Evaluation perspectives

Tefko Saracevic 36 Evaluation perspective …

Tefko Saracevic 37 Possible rewards* * but don’t bet on it!