© Ying Zhang, Tefko Saracevic 1 criteria in evaluation of use & usability in digital libraries Ying Zhang, Ph.D. University of California, Irvine, USA.

Slides:



Advertisements
Similar presentations
GMD German National Research Center for Information Technology Darmstadt University of Technology Perspectives and Priorities for Digital Libraries Research.
Advertisements

DELOS Highlights COSTANTINO THANOS ITALIAN NATIONAL RESEARCH COUNCIL.
Action Research Not traditional educational research often research tests theory not practical Teacher research in classrooms and/or schools/districts.
Enhancing Data Quality of Distributive Trade Statistics Workshop for African countries on the Implementation of International Recommendations for Distributive.
© Copyright CSAB 2013 Future Directions for the Computing Accreditation Criteria Report from CAC and CSAB Joint Criteria Committee Gayle Yaverbaum Barbara.
An Outcomes-based Assessment Model for General Education Amy Driscoll WASC EDUCATIONAL SEMINAR February 1, 2008.
© Tefko Saracevic, Rutgers University1 digital libraries and human information behavior Tefko Saracevic, Ph.D. School of Communication, Information and.
Tefko Saracevic1 EVALUATION in searching Requirements Criteria
A general course in digital libraries: A case study Tefko Saracevic, PhD School of Communication, Information & Library Studies Rutgers University, USA.
© Tefko Saracevic, Rutgers University1 Interaction in information retrieval There is MUCH more to searching than knowing computers, networks & commands,
Tefko Saracevic, Rutgers University1 The context of digital Libraries: a few illustrations Tefko Saracevic, Ph.D. School of Communication, Information.
© Tefko Saracevic, Rutgers University 1 EVALUATION in searching IR systems Digital libraries Reference sources Web sources.
© Tefko Saracevic, Rutgers University1 digital libraries and human information behavior Tefko Saracevic, Ph.D. School of Communication, Information and.
© Tefko Saracevic1 Search strategy & tactics Governed by effectiveness&feedback.
© Tefko Saracevic, Rutgers University1 What are digital libraries? Variety of perspectives and models Tefko Saracevic, Ph.D. School of Communication,
Introduction to Implementing an Institutional Repository Delivered to Technical Services Staff Dr. John Archer Library University of Regina September 21,
User interface design Designing effective interfaces for software systems Objectives To suggest some general design principles for user interface design.
Company LOGO B2C E-commerce Web Site Quality: an Empirical Examination (Cao, et al) Article overview presented by: Karen Bray Emilie Martin Trung (John)
Data Sources & Using VIVO Data Visualizing Scholarship VIVO provides network analysis and visualization tools to maximize the benefits afforded by the.
© Tefko Saracevic, Rutgers University1 digital libraries and human information behavior Tefko Saracevic, Ph.D. School of Communication, Information and.
Tefko Saracevic 1 criteria and methods in evaluation of digital libraries: use & usability Tefko Saracevic, Ph.D.
USING STUDENT OUTCOMES WHEN INTEGRATING INFORMATION LITERACY SKILLS INTO COURSES Information Literacy Department Asa H. Gordon Library Savannah State University.
FLCC knows a lot about assessment – J will send examples
Technology Support on a University Campus Contingency Theory and Collaboration.
Evaluation of digital Libraries: Criteria and problems from users’ perspectives Article by Hong (Iris) Xie Discussion by Pam Pagels.
1. 2 Considering the Establishment Survey Response Process in the Context of the Administrative Sciences Diane K. Willimack U.S. Census Bureau.
ALISE 2014 Conference Jeonghyun Kim & William E. Moen
Critical Role of ICT in Parliament Fulfill legislative, oversight, and representative responsibilities Achieve the goals of transparency, openness, accessibility,
Margaret J. Cox King’s College London
Communication Degree Program Outcomes
1 © Tefko Saracevic, Rutgers University Evaluation of library and information services (LIS): an overview Contexts Approaches Levels Requirements Measures.
Outcome Based Evaluation for Digital Library Projects and Services
LIS 506 (Fall 2006) LIS 506 Information Technology Week 11: Digital Libraries & Institutional Repositories.
Chapter 3 ORGANIZATIONAL CULTURE AND ENVIRONMENT: THE CONSTRAINTS
EDU 385 Education Assessment in the Classroom
Copyright © Allyn & Bacon 2008 Locating and Reviewing Related Literature Chapter 3 This multimedia product and its contents are protected under copyright.
Chapter 3 Copyright © Allyn & Bacon 2008 Locating and Reviewing Related Literature This multimedia product and its contents are protected under copyright.
Heuristic evaluation Functionality: Visual Design: Efficiency:
Assessing the Capacity of Statistical Systems Development Data Group.
© Tefko Saracevic, Rutgers University1 evaluation of digital libraries: an overview Tefko Saracevic, Ph.D. School of Communication, Information and Library.
GIS and Community Health. Some critiques of GIS emphasize the potentially harmful social consequences of the diffusion of GIS technology, including reinforcing.
Optimizing Resource Discovery Service Interfaces in Statewide Virtual Libraries: The Library of Texas Challenge William E. Moen, Ph.D. Texas Center for.
Science Teaching & Instructional Technology By: Asma, Melissa & Susan.
Shruthi(s) II M.Sc(CS) msccomputerscience.com. Introduction Digital Libraries have become the source of information sharing across the globe for education,
Software Architecture
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 United States License Creative Commons Attribution-NonCommercial-ShareAlike.
Tackling the Complexities of Source Evaluation: Active Learning Exercises That Foster Students’ Critical Thinking Juliet Rumble & Toni Carter Auburn University.
Copyright 2010, The World Bank Group. All Rights Reserved. Principles, criteria and methods Part 2 Quality management Produced in Collaboration between.
Symposium on Global Scientific Data Infrastructures Panel Two: Stakeholder Communities in the DWF Ann Wolpert, Massachusetts Institute of Technology Board.
JS Mrunalini Lecturer RAKMHSU Data Collection Considerations: Validity, Reliability, Generalizability, and Ethics.
Digital Libraries1 David Rashty. Digital Libraries2 “A library is an arsenal of liberty” Anonymous.
1 Integrating Human Factors into Designing User Interface for Digital Libraries Sung Been Moon
Digital Library Evaluation Flora McMartin Broad Based Knowledge
1 IFLA FRBR & MIC Metadata Evaluation Ying Zhang Yuelin Li October 14, 2003.
Evaluating Engagement Judging the outcome above the noise of squeaky wheels Heather Shaw, Department of Sustainability & Environment Jessica Dart, Clear.
© Tefko Saracevic, Rutgers University1 how were digital libraries evaluated? Tefko Saracevic, Ph.D. School of Communication, Information and Library Studies.
Implementation Science: Finding Common Ground and Perspectives Laura Reichenbach, Evidence Project, Population Council International Conference on Family.
Organizations of all types and sizes face a range of risks that can affect the achievement of their objectives. Organization's activities Strategic initiatives.
Developing a Monitoring & Evaluation Plan MEASURE Evaluation.
Tefko Saracevic This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 United States License Creative Commons Attribution-NonCommercial-ShareAlike.
© 2008 Tefko Saracevic, Rutgers University Tefko Saracevic, Ph.D.
An assessment framework for Intrusion Prevention System (IPS)
Introduction to Implementing an Institutional Repository
Meeting LIS Competences to Serve Inclusive Community through Curriculum: Case Study in LIS Study Program UIN Sunan Kalijaga Yogyakarta Indonesia Marwiyah.
Information Technology (IT)
Measuring Data Quality and Compilation of Metadata
Objectives, activities, and results of the database Lituanistika
digital libraries and human information behavior
ROLE OF «electronic virtual enhanced research-engaged student teams» WEB PORTAL IN SOLUTION OF PROBLEM OF COLLABORATION INTERNATIONAL TEAMS INSIDE ONE.
how were digital libraries evaluated?
Presentation transcript:

© Ying Zhang, Tefko Saracevic 1 criteria in evaluation of use & usability in digital libraries Ying Zhang, Ph.D. University of California, Irvine, USA Tefko Saracevic, Ph.D. Rutgers University, USA

© Ying Zhang, Tefko Saracevic 2 “Evaluating digital libraries is a bit like judging how successful is a marriage” (Marchionini, 2000)

© Ying Zhang, Tefko Saracevic 3 digital libraries since emergence in early/mid 1990’s –many institutions & fields got involved –great many practical developments –many research efforts & programs globally –large & growing expenditures in practice –applications & use growing exponentially everything about digital libraries is explosive except evaluation –relatively small, even neglected area

© Ying Zhang, Tefko Saracevic 4 literature reports on DL evaluation two distinct types: –meta or “about” literature suggest approaches, models, concepts discussed evaluation –object or “on” literature actual evaluations, contains data –data could be hard or soft so far meta literature larger but we are concentrating here on object literature only

© Ying Zhang, Tefko Saracevic 5 boundaries difficult to establish, apply –particularly as to process – e.g. crossing into IR: where does IR evaluation stop & DL evaluation start? or any technology evaluation? or evaluation of web resources and portals? is every usability study evaluation as well? brings up the perennial issues: –what is a digital library? what are all the processes that fall under DL umbrella?

© Ying Zhang, Tefko Saracevic 6 on the scene several different communities involved in digital libraries, each with quite different –perspective, concepts, meanings in dealing with DL –concentration, emphasis, approach, models many disciplines, institutions involved

© Ying Zhang, Tefko Saracevic 7 computer science perspectives concentrates on research & development (R&D) technology centered –distributed & organized knowledge resources in digital formats how to collect, store, organize, diverse types of information - texts, images, sounds, multimedia … –new kind of distributed database services to manage unstructured multimedia resources

© Ying Zhang, Tefko Saracevic 8 library & institutional perspective concentrates on institutions, service, practice – logical extension of libraries content, collection, service centered –creation of digital collections variety of materials repositories of digital materials –access to collections guided by service mission various environments, user communities various degrees of integration or separation

© Ying Zhang, Tefko Saracevic 9 professional, subject organizations concentrate on collections & their uses in specific areas –new forms of publishing in their area field, subject, application centered –highly diversified variety of organizations involved –scientific & technical societies –various fields, academic units –projects - institutions, consortia –museums, historical societies –government agencies services to communities or perceived needs

© Ying Zhang, Tefko Saracevic 10 amount of evaluation in different communities professional, subject organizations library & institutional most computer science least

© Ying Zhang, Tefko Saracevic 11 what is needed to evaluate performance? 1.construct - system, process, part to be evaluated 2.objectives - reasons, desires for evaluation 3.criteria - standards, base for reflecting objectives 4.measure - units for recording & comparing criteria 5.measuring instruments - devices, tools that record a measure 6.methodology - way of doing evaluation assembling, applying, analyzing

© Ying Zhang, Tefko Saracevic 12 examples ElementAthletic event Information retrieval (IR) Construct 10 km raceIR system, given method Objective winner?effectiveness Criteria speed - timerelevance Measure minutes, seconds precision, recall Instrument watchpeople, judges Method timing from start to finish Text Retrieval Conference (TREC)

© Ying Zhang, Tefko Saracevic 13 criteria in DL evaluation reflect performance of DL (or part) as related to selected objectives –what parameters of performance were concentrated on? in DL: no basic or standardized criteria, no overall agreement –many have been used –even for the same objectives

© Ying Zhang, Tefko Saracevic 14 concentration in this presentation on criteria used in DL evaluations –as reported in published studies that 1.directly address a DL entity or a DL process 2.contain data in whatever form no more than 150 or so evaluation reports exist totally –as extracted from Ying Zhang’s PhD dissertation “Developing a holistic framework for digital library evaluation” Rutgers 2007 to find criteria used in existing DL evaluations to validate & enlarge these criteria experimentally based on perceived importance by different DL stakeholders

© Ying Zhang, Tefko Saracevic 15 usability criteria “extent to which a user can achieve goals with effectiveness, efficiency & satisfaction in context of use” (International Standards Organization) widely used, but no uniform definition for DL general, meta criterion, covers a lot of ground umbrella for many specific criteria used in DL evaluations so what was meant by usability in DL evaluation?

© Ying Zhang, Tefko Saracevic 16 classes of criteria derived from literature content –how well are digital collections selected, developed; objects created, organized, represented, presented technology –how well do hardware & software support library functions interface –what is available for users to interact & how much is interaction supported or hindered

© Ying Zhang, Tefko Saracevic 17 classes of criteria (cont.) process/service –what processes & assistance are provided; what range of services is available; how well are they functioning; (carrying out tasks as: search, browse, navigate, find, evaluate or obtain a resource) user –what are the outcomes of DL use – changes in human information behavior, cognitive state, decision-making, problem-solving; impact on accomplishing tasks; broader impact/benefit in research, professional work context –how well does a DL fit into, respond to, follow larger context – institutional, economic, legal, social, cultural; effects on context

© Ying Zhang, Tefko Saracevic 18 sample of criteria from literature ContentTechnologyInterface completeness, size coverage, overlap quality, accuracy validity, authority adequacy, diversity informativeness freshness accessibility, availability complexity- organizational structure transparency, clarity effort to understand … response time processing time speed capacity load accessibility effectiveness efficiency compatibility quality reliability robustness… attractiveness consistency representation of concepts - labels communicativeness of messages display, attractiveness appropriateness consistency ease of use effort error detection, personalization …

© Ying Zhang, Tefko Saracevic 19 sample … (cont.) Process/ Service UserContext learnability, effort/time, support, convenience ease of use lostness (confusion) completion (achievement of task) interpretation difficulty sureness in results error rate responsiveness reliability,… satisfaction, success relevance, usefulness of results impact, value quality of experience barriers, irritability preferences learning effect productivity use/reuse,… institutional fit, usefulness productivity of & impact on community members sustainability interoperability rights management, copyright abidance organizational usability, …

© Ying Zhang, Tefko Saracevic 20 a holistic set of criteria- research design (from Zhang’s study) Methodology – three stages: 1.exploration (literature review + interview): a.Literature review: sources: LIS databases, computer science databases, and other general databases (e.g., Web of Science); results: six criteria lists at the six DL levels from 151 papers; b.Semi-structured interviews + card-sorting: participants: 3 DL administrators, 3 DL developers, 3 DL researchers; results: six lists of frequently mentioned criteria, six sets of ranked criteria based upon perceived importance; 2.confirmation (online survey): participants: 431 (25 administrators, 36 developers, 159 librarians, 53 researchers, 158 general users) from 22 countries; results: (1) lists of most and least perceived important criteria; (2) the holistic DL evaluation model; (3) a list of criteria with statistically proven inter-group divergence; (4) a list of new criteria 3.verification (experiment): participants: 33 (4 administrators, 6 developers, 7 librarians, 5 researchers, 11 general users); results: major consistent and minor inconsistent in terms of a few top and least important criteria (e.g. TN-security) and inter-group divergence (e.g. IF-interaction support)

© Ying Zhang, Tefko Saracevic 21 content list of criteria more important –accessibility –accuracy –usefulness –fidelity –integrity less important –conciseness

© Ying Zhang, Tefko Saracevic 22 technology list of criteria more important –reliability –ease of use –effectiveness –interoperability –efficiency less important –flexibility

© Ying Zhang, Tefko Saracevic 23 interface list of criteria more significant –effectiveness –ease of use –consistency –effort needed –appropriateness less significant –personalization

© Ying Zhang, Tefko Saracevic 24 user list of criteria more significant –success –satisfaction –use/reuse –productivity less significant –behavior change

© Ying Zhang, Tefko Saracevic 25 service list of criteria more significant –reliability –accessibility –usefulness –responsiveness –integrity less significant –courtesy

© Ying Zhang, Tefko Saracevic 26 context list of criteria more significant –sustainability –collaboration –rights management –managerial support less significant –extended social impact

criteria with statistically significant inter-group divergence © Ying Zhang, Tefko Saracevic 27 content o appropriateness to target users o comprehensiveness technology o appropriateness to digital information o interoperability o security interface o appropriateness to target users user o acceptance context o copyright compliance o extended social impacts o integrity to organizational practice o managerial support

© Ying Zhang, Tefko Saracevic 28 general results from all evaluation studies not synthesized here hard to synthesize anyhow generalizations are hard to come by except one!

© Ying Zhang, Tefko Saracevic 29 users and digital libraries a number of studies reported various versions of the same result: users have many difficulties with DLs –usually do not fully understand them –they hold different conception of a DL from operators or designers –they lack familiarity with the range of capabilities, content and interactions –they often engage in blind alley interactions

© Ying Zhang, Tefko Saracevic 30 analogy perceptions of users and perceptions of designers and operators of a DL are generally not very close users are from Venus and DLs are from Mars leads to the versus hypothesis

© Ying Zhang, Tefko Saracevic 31 is it: why VERSUS? –users and digital libraries see each other differently user AND digital library or user VERSUS digital library

© Ying Zhang, Tefko Saracevic 32 user AND digital library model context user cognitive task affective competence context digital library content representation organization services user model of digital library digital library model of user

© Ying Zhang, Tefko Saracevic 33 how close are they? user VERSUS digital library model what user assumes about digital library: how it works? what to expect? what digital library assumes about user: - behavior? - needs? digital library model of user user model of digital library

© Ying Zhang, Tefko Saracevic 34 the versus hypothesis in use, more often than not, digital library users and digital libraries are in an adversarial position hypothesis does not apportion blame –does not say that DL are poorly designed –or that users are poorly prepared adversarial relation may be a natural order of things

© Ying Zhang, Tefko Saracevic 35 toward conclusions: evaluation of digital libraries impossible? not really hard? very could not generalize yet no theories no general models embraced yet, although quite a few proposed in comparison to total works on DL, only a fraction devoted to evaluation

© Ying Zhang, Tefko Saracevic 36 why? – some speculations Complexity: DLs are highly complex –more than technological systems alone –evaluation of complex systems is very hard –just learning how to do this job –experimenting with doing it in many different ways Premature: it may be too early in the evolution of DL for evaluation on a more organized scale

© Ying Zhang, Tefko Saracevic 37 why? (cont.) Interest: There is no interest in evaluation –R&D interested in doing, building, implementing, breaking new paths, operating … –evaluation of little or no interest, plus there is no time to do it, no payoff Funding: inadequate or no funds for evaluation –evaluation time consuming, expensive requires commitment –grants have minimal or no funds for evaluation –granting agencies not allocating programs for evaluation –no funds = no evaluation.

© Ying Zhang, Tefko Saracevic 38 why? (cont.) Culture: evaluation not a part of research and operations of DL –below the cultural radar; a stepchild –communities with very different cultures involved language, frames of reference, priorities, understandings differ communication is hard, at times impossible –evaluation means very different things to different constituencies

© Ying Zhang, Tefko Saracevic 39 why – the end Cynical: who wants to know or demonstrate actual performance? –emperor clothes around? –evaluation may be subconsciously or consciously suppressed –dangerous?

© Ying Zhang, Tefko Saracevic 40 ultimate evaluation The ultimate evaluation of digital libraries: –assessing transformation in their context, environment –determining possible enhancing changes in institutions, learning, scholarly publishing, disciplines, small worlds … –and ultimately in society due to digital libraries.

© Ying Zhang, Tefko Saracevic 41 final conclusion finally evaluation of digital libraries still in formative years not funded much, if at all but necessary for understanding how to –build better digital libraries & services & –enhance their role

© Ying Zhang, Tefko Saracevic 42 evaluation digital library How to do it?

© Ying Zhang, Tefko Saracevic 43

© Ying Zhang, Tefko Saracevic 44