Błażej Feret The Main Libray of theTechnical University of Łódź, Poland Marzena Marcinek Cracow University of Technology Library, Poland Library performance indicators does it really make sense to measure them? CASLIN Seminar, June 2006 Data collection and analysis of library performance of library performance a case study for Polish research libraries

2 Ackowledgements Special thanks go for Mrs Lidia Derfert-Wolf, a member of the Task Group for Standardisation for Polish Research Libraries, for her kind assistance and advice.

3 Plan of the workshop 1. Introduction 15 minutes 2. A few general remarks on library quality and performance measurement Quality criteria from different perspectives10 minutes 3. A Common Project of Polish Research Libraries on Comparable Measures (incl. examples)40 minutes Czech realities20 minutes 4. Standardised terminology and descriptions of library performance indicators based on the ISO Standard Information and documentation – Library performance indicators 20 minutes Conclusions 15 minutes

4 A few general remarks on library quality and performance indicators

5 Quality: fitness for purpose fitness for purpose fitness for use fitness for use conformity to requirements conformity to requirements absence of defects absence of defects …… …… Quality of libraries (ISO 11620): totality of features and characteristics of a product or services that bear on the library's ability to satisfy stated or implied needs

6 Quality of research libraries from the perspective of: Users Funding bodies Librarians Please consider all these perspectives

7 Model of measuring quality of libraries inputs - the raw data such as finance, collection, equipment, users and staff, space, seats inputs - the raw data such as finance, collection, equipment, users and staff, space, seats outputs - the work done, i.e. circulation, cataloguing, reference services, preservation, interlibrary lending, facilities usage and e-sources searches outputs - the work done, i.e. circulation, cataloguing, reference services, preservation, interlibrary lending, facilities usage and e-sources searches outcomes – user satisfaction and the impact of library services on users at the local institution and society outcomes – user satisfaction and the impact of library services on users at the local institution and society  quantitative data  qualitative data from the perspective of:  users  funding bodies  librarians group work

8 Library quality assessment - what is required? set of quality criteria library’s goals and objectives set of quality criteria library’s goals and objectives set of performance indicators set of performance indicators national library statistics system national library statistics system standardisation on local, national and international level standardisation on local, national and international level but...

9 “ The level of quality cannot be defined once and for all, since both the criteria and evaluation methods, as well as the assessment of the results achieved, may change. This is caused by various factors, for example technological, political, economic, as well as the ones connected with the community in which and for whom information services work. It is crucial that these varying criteria and methods, as well as the dissimilarity in the level of the quality achieved by a given country, are taken into account when aiming at adopting the international standards on the quality of information work. Countries differ in living standards they achieve, in the level of education, and the way they introduce innovations. Setting the goal is of fundamental importance, since its accomplishment or failure to reach it may be treated as a basis for quality assessment”. W. Pindlowa

Activities conducted by the Group for Standardisation for Polish Research Libraries, results and plans for the future CASLIN Seminar, June 2006

11 The Group for Standardisation for Polish Research Libraries coordinated by the Adam Mickiewicz University in Poznań  Lidia Derfert-Wolf (ATR, Bydgoszcz)  Ewa Dobrzyńska-Lankosz (AGH, Krakow)  Mirosław Górny ( Adam Mickiewicz University, Poznan)  Elżbieta Górska (Warsaw Public Library)  Marek Górski (Cracow University of Technology)  Artur Jazdon ( Adam Mickiewicz University, Poznan )  Dariusz Pawelec (Silesian University, Katowice)  Anna Sokołowska-Gogut (Academy of Economics, Krakow)  Teresa Wildhardt (Pedagogical University, Krakow)

12 Objectives  to gather libraries' statistical data for a computer database  to select a set of performance indicators and standards for library performance (quantity, quality and effectiveness)  to conduct a comparative research  to prepare and publish yearly reports  to define methods for the assessment of Polish research libraries

13 Project for the Analysis of Polish Research Libraries

14 Some facts as of May 2006 Libraries registered – academic libraries (42 state-owned and 11 non state-owned) 3 public libraries 2 special libraries Questionnaires for 2004 completed 31 libraries Questionnaires for 2003 completed 29 libraries Questionnaires for 2002 completed 17 libraries

15 Software for the collection and analysis of data - general requirements  on-line access to the questionnaire (submission, modification)  selected performance indicators automatically calculated and presented  automatic control and verification of the accuracy of submitted data  access to analysing functions for individual libraries  possibility to conduct a multi-aspect comparative analysis of selected data and performance indicators

16 Web-based Application for: General users Library directors Analysts Administrators

17 General user General information about the Project Instructions Registration Rules for filling up questionnaires Data analyses Questionnaires (patterns in doc. format) Example results Useful links Useful links

18 General information Instructions Questionnaires Results Useful links handouts 1

19

20 Library Director Filling up questionnaires; Analysis of the own library data sorted by years, question categories or indicators; Access to calculated automatically indicators Comparison of the own library data with average results of other libraries in the same category or in the country (data published in the form of tables). Establishment and modification of access rights for the library staff

21 Login

22 For directors of registered libraries Users Questionnaires Analyses Send Change password Logout

23 Questionnaire  48 questions of various types  refer to easily accessible or computable data (e.g. size of collection, number of users etc.)  closed questions about the services offered (e.g. on-line reservation: Yes/No)  88 performance indicators  19 calculated by librarians  69 calculated automatically handouts 2

24 Questionnaire addresses all the elements of a library system, its environment, library processes and services divided into chapters: Staff Collection Budget Infrastructure Circulation Information services Didactics Publications and data bases created by the library Library cooperation Organisation of library events Professional activity of library staff,

25 Examples of performance indicators required to complete the questionnaire  library’s total expenditures  expenditures for library materials/books  ratio of library budget to the budget of its parent university  time required for the technical processing of a document  collection on the computer system as a % of the whole collection of the library  percent of catalogue descriptions acquired from outside resources

26 Examples of performance indicators calculated automatically  library expenditure per student/user  expenditures for library materials/books per student/user  library registered users as a percent of potental users  space of the library per user  collection on the computer system as a % of the whole collection of the library  number of user training hours per one staff member

27 Survey form No of libraries in the university library and information system (incl. the main library) For university libraries data concern main library university library and information system Staff No of library staff commentary categories education age

28 Collection Acquisition

29 Budget

30 Infrastructure Circulation

31

32 Standard lending period Information services Didactics Library’s own publications and databases Electronic sources usage

33 Library’s own publications and databases Interlibrary cooperation, staff professional activities

34 Selected performance indicators

35 Once the questionnaire is validated the indicators are calcullated automaticly Indicators calculated automatically Admin can add other indicators based on the data gathered - no action by director required

36 Adding and modifying users

37 values statistical data Director can analyse performance of his/her own library for selected years

38 Tyle kategorii ile pytan i wskaznikow wpisanych i automatycznych categories for the analysis Number of categories = number of data and indicators (entered and calculated automatically)

39 Total library space per student No of loans per staff membber No of loans per registered user

40 No of years analysed: 3 Total library space per student No of loans per one registered user No of loans per staff member

41 Analysis Questionnaires Access to all the questionnairesAccess to all the questionnaires No modification rightsNo modification rights Analyst

42 Selection of the library categories and years Type: academic Other types Source of funding Analyse the years Sort by (years, types)

categories for the analysis found, please select Staff Collection

44 Indicators calculated automaticaly Total library space per one user No of loans per one registered user

45 Total library space per user No of loans per one registered user Research libraries performance in 2003

46 No of loans per one registered user No of loans per staff member Research libraries performance in 2004

47

48 Administartor’s Module Libraries logout Database statistics Change password Send Outstanding questionnaires Forms Questionnaires Users

49 Registration form Send via snail mail to….

50 Sortowanie Add / modify library Sort by Source of funding Type Name

51 Select the library

52 List of Libraries with outstanding questionnaires missing not validated Questionnaires validated by director Validated by administrator

53 The analysis of data – examples (no complete data for 2005 yet)

54

55

56 Structure of budget in state university libraries in the years 2002 and 2003 collection salaries space hardware &software administration other

57 Structure of budget in state technical universities libraries in the years 2002 and 2003 collection space hardware &software administration other salaries

58 Selected indicators for state academic libraries No of books per one user [vols] all academic libraries technical universities universities

59 Selected indicators for state academic libraries No of new documents (books) per user [vols.]

60 Selected indicators for state academic libraries Expenditures per user Expenditures for new collection per user All academic libraries Technical universities universities All academic libraries Technical universities Universities

61 rok 2003 Selected indicators for state academic libraries rok 2002 Budżet biblioteki jako procent budżetu uczelni - rok 2002 Library budget as percent of parent university budget

62 Selected indicators for state academic libraries Pracownicy z wyższym wykształceniem bibliotekarskim jako odsetek pracowników działalności podstawowej Staff with MA degree in librarianship as a percent of the whole library staff

63 Feedback loop

64 lack of statistical data required to complete the questionnaire lack of comparable data on the use of electronic resources (incl. differences in usage statistics generated by various providers) differences in library structure and budgeting within university difficulties with validation – mistakes (e.g. wrong ratio) need correction, misunderstanding of data requirements, wrong interpretation of questions Problems arisen when receiving data

65 productimprovementstesting problems noticed by librarians and analysts Verivfication of data required Verification of indicators Software tools to control data in the fields Detailed commentaries...and solving them

66 Selection of performance indicators – the reasons for measurement What? Why? At what cost? Part of reporting mechanism (statistics) Part of internal assessment Support for decision making (locally, within parent organisation, at the national level) External requirements

67 Library performance indicators in the ISO 11620:1998 Standard

68 The purpose of library performance indicators To function as tools to assess the quality and effectiveness of activities and services provided by a library To assess the efficiency of resources allocated by the library to its activities and services

69 Required features of performance indicators Informative content Reliability Validity Appropriateness Practicality Comparability

70 Informative content The indicator has to be informative as a tool for measuring activity, for identyfying achievements and problems in the performance of the library so that action can be taken to remedy this. It should provide information for decision-making, e.g. goalsetting, budget allocation, prioritizing services and activities.

71 Reliability A performance indicator should be reliable in the sense that it consistently produces the same results when used repeatedly under the same circumstances.

72 Validity A performance indicator must be valid, that is, it must measure what it is intended to measure.

73 Appropriateness A performance indicator must be appropriate for the purpose to which it is to be put. That is, the units and scale must be suitable and the operations necessary to implement the process of measurement should be compatible with the library’s procedures, physical layout, etc.

74 Practicality A performance indicator has to be practical in the sense that it uses data that can be made available by the library with a resonable amount of effort in terms of staff time, staff qualifications, operational costs and user’s time and patience.

75 Comparability A performance indicator allows comparison between libraries if the same score, making allowance for the accuracy of the score, means the same level of quality of services or the same level of efficiency in the libraries to be compared.

76 Some definitions – group work Performance Indicator Evaluation Accessability Effectiveness Efficiency Reliability Validity Loan Recurrent expenditure Resources Target population Handouts 3 and 4 afterwards

77 Performance indicators – a descriptive framework Name Objective Scope Definition Method Interpretation Factors affecting the indicator Related indicators Handout 5

78 Uses of performance indicators The quality and effectiveness of the services of the library as well as the efficiency of the uses of the resources are evaluated against the mission, goal and objectives of the library itself. The quality and effectiveness of the services of the library as well as the efficiency of the uses of the resources are evaluated against the mission, goal and objectives of the library itself. Performance indicators should be linked to systematic library planning and evaluation. Performance indicators should be linked to systematic library planning and evaluation. Indicators are useful for comparison over time within the same library. Comparison between libraries is possible but careful interpretation is required.

79 Comparability of performance indicator data Purposes of using library performance indicators: self-diagnosis (within the same library) e.g. comparisons of one year’s performance with another; comparison accross different libraries in full recognition of the limitations of such comparisons and with respect for each library’s: mission, goals and objectives resources user groups governance / funding structure procedures

80Limitations It is impossible to achieve optimum scores simultaneously on all performance indicators (eg. user satisfaction vs. expenditure per user) „The scores on performance indicators must be interpreted in the light of what the library intends to accomplish, not simply in terms of optimizing scores on particular indicators.” Degree of accuracy Degree of accuracy Sampling errorsSampling errors Subjective aspects of the measuring processSubjective aspects of the measuring process Inadequate time or resource for measuring processInadequate time or resource for measuring process Users skills vs. library performance perception Users skills vs. library performance perception Linking resources to services, management approaches, staff skills etc. Linking resources to services, management approaches, staff skills etc.

81 What factors would you consider when making decision for registering your library to the library project for comparable measures? - group work

82 Relevancy to to the services being analysed Strategic purpose of the measurement activity Staff involved in the measurement process and their awarness of the fact that measurment process is part of their normal flow of work Accuracy Reliability Consistency over time Practical aspect Consider

83 Success conditions Simplicity versus complexity (goals!) Simplicity versus complexity (goals!) The effort involved in data collection versus the expected outcomes The effort involved in data collection versus the expected outcomes Possibility to integrate data collection with the already carried work Possibility to integrate data collection with the already carried work Trends are (usually) more important than figures Trends are (usually) more important than figures For some measures (e.g. time taken to handle information queries) setting aside certain periods throughout the year and extrapolating findings to a full year For some measures (e.g. time taken to handle information queries) setting aside certain periods throughout the year and extrapolating findings to a full year Representative samples Representative samples Do not collect data for the sake of it Do not collect data for the sake of it

84 Thank you!