Impact measures for libraries and information services Roswitha Poll Münster Bielefeld 2006 Bielefeld 2006

The importance of libraries What libraries ensure (IFLA) Democratic values Economic progress The open information society Information equality Reduction of poverty Information literacy Intellectual freedom Cultural diversity Libraries make a difference But who will believe it? Bielefeld 2006 Bielefeld 2006

input activities impactoutcomevalue outcome a contribution of work, information, or material activities change input into output the effect or influence of one person, thing, or action, on another the consequence, visible or practical result or effect of an event or activity the importance or preciousness of something, the perception of actual or potential benefit output the amount produced, the results supplied Bielefeld 2006 Bielefeld 2006

Don Revill ACRL Bielefeld 2006 Bielefeld 2006

Don Revill ACRL Peter Brophy Differentoutcomes positive — negative direct/immediate — long-term actual — potential intended — unexpected Bielefeld 2006 Bielefeld 2006

Impact/outcome of cultural institutions knowledge information literacy academic or professional success social inclusion individual well-being changes in skills, competences, attitudes, behaviour Bielefeld 2006 Bielefeld 2006

Outcomes of universities recruitment and retention of students and excellent academic staff effective teaching - high graduation rates - high grades in examinations - high employment rates after examination effective research - high renown and use of research results and publications - high amount of special grants - awards, honours Evidence needed: The library supports these goals Bielefeld 2006 Bielefeld 2006

Problems of assessing outcome Value and outcome of a service can differ between user groups Data may not be available because of data protection rules Results of projects are often not comparable because of different data collection methods Long-term effects cannot be assessed as persons are no more available All tested methods are time-consuming Influences on individuals are manifold Bielefeld 2006 Bielefeld 2006

Methods for assessing impact Quantitative measures tests assessing user skills before and after training on a service performance monitoring / data mining observation (unobtrusive) analysis of documents and bibliographies compiled by users correlation between library use and success Bielefeld 2006 Bielefeld 2006 "Surrogate measures" have to be used

Methods for assessing impact Qualitative measures Surveys (print, telephone, online) Interviews Focus groups, discussion groups Self-assessment of users Facts and stories must be organized to show patterns The "anecdotal evidence" supports data Bielefeld 2006 Bielefeld 2006

ACRL Jennifer Cram User satisfaction as outcome measure? Bielefeld 2006 Bielefeld 2006 A change of behaviour? Or a basis for such change?

Correlation of library use and academic success Frequency (loans, visits) Range of services used (reference service, user training, ILL) Range of collections used (undergraduate collection, E-journals, special material) Attendance at training sessions Success, expressed by Duration of studies Grades in exams Student retention Employment rate after exam Quality of publications (impact factor, peer-review journals) Library use, expressed by Assessing impact Bielefeld 2006 Bielefeld 2006

The library's impact on information literacy surveys after instruction surveys after instruction pretest / posttest pretest / posttest self-assessment of users self-assessment of users behavioural observation behavioural observation transaction logs transaction logs longitudinal studies longitudinal studies assessing changes in students' bibliographiesassessing changes in students' bibliographies Outcome of a specific training or series of trainings Assessing impact Bielefeld 2006 Bielefeld 2006

Importance of the local library for research 2 Estimated percentage of the information needed for a publication / paper / report found via the local library for a publication / paper / report found via the local library questionnaire/interview after publication questionnaire/interview after publication 1Percentage of material cited in academic publications that was (could have been) retrieved via the local library. that was (could have been) retrieved via the local library. Citations in dissertationsdissertations student papersstudent papers publications of a facultypublications of a faculty Assessing impact 3 Bielefeld 2006 Bielefeld 2006

Impact on social inclusion and community life Does the existence of a specific library / of libraries further individual well-being the quality of life in a society - democracy - social inclusion - cultural life - local identity - life-long learning Method: Assessing public opinion (users and / or non- users) as to the benefits of libraries and special library services to the population (indirect benefit) to the individual user (direct benefit) to the individual user (direct benefit) Assessing impact 4 Bielefeld 2006 Bielefeld 2006

The financial value of libraries Assessing impact 5 Assessing time costs (replacement value of a client's time) Assessing time costs (replacement value of a client's time) Users invest time and effort in order to use library services. The value that they - or their institution - place on that use must be at least as high as their „sacrifice“ of time. (Costs are calculated by time and average salary) Using the contingent valuation method Using the contingent valuation method Willingness-to-pay What would users pay for the maintenance of a service? Willingness-to-accept Which sum would users accept as equivalent for the deletion of a service? Bielefeld 2006 Bielefeld 2006 For every £1 of public funding the British Library receives each year, £4.4 is generated for the economy

Using the results accountability decision making resource management improvement of services promotion of the library's role Libraries on the agenda (Claudia Lux) Bielefeld 2006 Bielefeld 2006

Outcome projects eVALUEd: in "evidence base", University of Central England: Toolkit for evaluating electronic information services IMLS (Institute of Museum and Library Services): outcome-based evaluation of projects ARL New Measures Initiative: several projects - Learning outcomes - Higher education outcomes research review - MINES (Measuring the impact of networked electronic services) Bielefeld 2006 Bielefeld 2006

Outcome projects IBEC: Information School of the University of Washington and University of Michigan School of Information: Toolkit for assessing the impact of information in communities SCONUL and LIRG (Library and Information Research Group): impact initiative and mailing list IFLA Section Statistics and Evaluation: working group on outcome/impact; bibliography of literature and projects worldwide Bielefeld 2006 Bielefeld 2006

(Goethe: Faust)