Collection Analysis: Overview Sponsored by ALCTS CMDS Measures & Education Committees Peggy Johnson, Associate University Librarian University of Minnesota

“Culture of Assessment/Evaluation” Way to demonstrate Way to demonstrate RelevanceRelevance ValueValue ImpactImpact Considered from the view of Considered from the view of UsersUsers StakeholdersStakeholders Amos Lakos & Shelley Phipps – “Culture of Assessment” John Crawford – “Culture of Evaluation”

“ Those who fail to move in the direction of systematic assessment will be unable to cope with the increasingly difficult questions that promise to confront collection officers in years to come.” Mark Sandler, Univ. of MI

Why do we do it? As part of good management. Accountability: To demonstrate to funders and clients that the service is delivering the benefits expected when the investment was made Accountability: To demonstrate to funders and clients that the service is delivering the benefits expected when the investment was made Decision-making: To ensure that resources are being used efficiently and effectively (an internal control mechanism) Decision-making: To ensure that resources are being used efficiently and effectively (an internal control mechanism) Marketing: To report success and accomplishments (public relations) Marketing: To report success and accomplishments (public relations)

Collection assessment assumes that the criteria for success are defined and understood by those doing the assessment and those to whom it is being reports.

What is it? A mechanism to determine: If the collection is meeting its objectives If the collection is meeting its objectives How well it is serving its users How well it is serving its users In which ways or areas it is deficient, and what remains to done to develop the collection In which ways or areas it is deficient, and what remains to done to develop the collection If selectors are performing their responsibilities effectively If selectors are performing their responsibilities effectively How to allocate collections/access funds How to allocate collections/access funds

How is assessment different from evaluation? Evaluation determines how well the collection supports the goals, needs, and curriculum of the parent organization. Evaluation determines how well the collection supports the goals, needs, and curriculum of the parent organization. Assessment examines or describes collections either in their own terms or relative to other collections and checklists. Assessment examines or describes collections either in their own terms or relative to other collections and checklists.

Who is the audience ? Accreditation agencies Accreditation agencies Parent organization (administration, board, senior management) Parent organization (administration, board, senior management) Library administration Library administration CDM supervisor CDM supervisor Selector Selector User community or communities User community or communities Consortial partners Consortial partners

How can we do it well? Simple Simple Practical Practical Repeatable Repeatable Clear focus Clear focus Understandable results Understandable results Meaningful results Meaningful results Results lead to action Results lead to action

Collection-based Measures Look at: Size Size Growth Growth Coverage (depth, breadth, balance) Coverage (depth, breadth, balance)

Collection-based Measures Checking lists, catalogs, bibliographies Checking lists, catalogs, bibliographies Evaluating the collection directly Evaluating the collection directly Compiling comparative statistics Compiling comparative statistics Application of collection standards Application of collection standards

Use- and User-based Measures Look at: Who is using the collection? Who is using the collection? How often? How often? What are users’ expectations? What are users’ expectations? What are user’s needs? What are user’s needs? What are their perceptions? What are their perceptions?

Use- & User-based Measures Circulation studies Circulation studies In-house use studies In-house use studies Survey of users Survey of users Shelf availability studies Shelf availability studies Analysis of online usage of electronic resources Analysis of online usage of electronic resources Analysis of ILL statistics Analysis of ILL statistics Citation studies Citation studies Document delivery tests Document delivery tests Cost-per-use Cost-per-use

Quantitative Measures Count things Count things UseUse ExpendituresExpenditures TitlesTitles Physical itemsPhysical items

Quantitative Measures Titles Titles Circulation transactions Circulation transactions Expenditures Expenditures E-metrics E-metrics ILL transactions ILL transactions Ratios (monographs/serials; volumes/students; expenditures/degree programs; electronic/print) Ratios (monographs/serials; volumes/students; expenditures/degree programs; electronic/print)

E-Metrics Online sessions Online sessions Documents downloaded Documents downloaded Records downloaded Records downloaded Virtual visits Virtual visits Turn-aways Turn-aways Alert usage Alert usage Personal profile users Personal profile users Remote versus onsite usage Remote versus onsite usage

Qualitative Research “A process of inquiry that draws data from the context in which events occurs... using induction to derive possible explanations based on observed phenomena” Gorman and Clayton, Qualitative Research for the Information Professional: A Practical Handbook, 2 nd ed. (London: Facet, 2005)

Qualitative Measures Look at: Strengths Strengths Weaknesses Weaknesses Non-strengths Non-strengths

Qualitative Measures Provide the context Provide the context Offer a way to understand the attitudes that inform the statistics Offer a way to understand the attitudes that inform the statistics

Quantitative Measures Focus groups Focus groups Online or printed surveys Online or printed surveys Interviews (structured or unstructured) Interviews (structured or unstructured) Observation Observation

Collection Analysis Methods Use- or User-based Collection-based Quantitative ILL statistics ILL statistics Circulation statistics Circulation statistics In-house use statistics In-house use statistics Document delivery statistics Document delivery statistics Shelf availability statistics Shelf availability statistics E-metrics E-metrics Collection size and growth Collection size and growth Materials budget size and growth Materials budget size and growth Collection size standards and formulas Collection size standards and formulas Expenditures by subject Expenditures by subject Ratios Ratios Qualitative User opinion surveys User opinion surveys User observation User observation Focus groups Focus groups Interviews Interviews List checking List checking Verification studies Verification studies Citation analysis Citation analysis Direct collection checking Direct collection checking Collection mapping (assigning conspectus levels) Collection mapping (assigning conspectus levels)

Steps in a Collection Analysis Project Do StudyAct Plan

Where to start? Define the question or problem Define the question or problem Determine metrics to use Determine metrics to use Decide: Decide: Where to locate the informationWhere to locate the information Who will collect the informationWho will collect the information Who will analyze and report the informationWho will analyze and report the information Who will act on the informationWho will act on the information

Remember Chose measures that matter Chose measures that matter Chose an approach that is simple Chose an approach that is simple Don’t aim for perfection—good ‘nuff is OK Don’t aim for perfection—good ‘nuff is OK Don’t do it once and never again Don’t do it once and never again Know your audience Know your audience Present data in a context—explain what it means Present data in a context—explain what it means