Forging Forward: Using Evaluation as a Stepping Stone Joe Matthews SLA – San Diego Fall Seminar October 30, 2015

Evaluation is the process of … Identifying and collecting data Focusing on specific services or activities Establishing criteria to gauge success Determining the degree to which stated goals and objectives are achieved

Types of Evaluation Activities Program or service planning Program or service monitoring Impact assessment (outcomes) Economic efficiency

Evaluation Questions How much?How many?How economical?How prompt? Magnitude % of change % of overall change Costs Magnitude Change Resources used Units processed Cycle times Turnaround time Anticipatory Library Controls

Evaluation Questions How valuable?How reliable?How accurate?How well? Effort expended Cost-benefit obtained Dependability Access Accuracy Currency Completeness Comprehensiveness Accuracy Expertise Performance meeting expectations Library & Customer Decide

Evaluation Questions How courteous?How responsive?How satisfied? Welcoming Attentive Anticipatory Helpful Empathetic Expectations met Materials obtained Personal interaction Ease of use Equipment used Facilities Willingness to return Customers Decide

Evidence-Based Library and Information Practice 1.Formulate a clearly defined question 2.Find the best evidence to answer the question 3.Critically appraise the evidence 4.Combine the appraisal of the evidence with professional judgment to make a decision 5.Evaluate the effectiveness of the results 6.Disseminate the results

Library Metrics Resources CapabilityOutputsOutcomes Impacts Input Measures Process Measures Output Measures Outcome Measures Impact Measures

Library Metrics National Center for Education Statistics Academic Library Survey Integrated Postsecondary Educational Data System (IPEDS) ACRL Academic Library Trends and Statistics survey ACRLMetrics IMLS Public Libraries Survey PLAMetrics

Perspectives & Tools LibraryCustomer QualitativeExpert opinion Checking lists Ethnographic QuantitativeUsage analysis Peer comparisons Citation studies ILL analysis Loss rate analysis Shelf availability Document delivery Curriculum analysis Collection development Life-cycle costs Surveys Value

Internal Evaluation Facilities Technologies Library Personnel Collections Services Other Input, Process, and Output Measures

External Evaluation Library’s Perspective Demographics and use Collection strengths Library personnel Library deployed technologies Facilities Services provided Unique programs and services

External Evaluation Customer Perspective – Outcomes ACRL – Megan Oakleaf’s Value report ACRL – Assessment in Action The LibValue Project – Carol Tenopir PLA – Project Outcome

Measuring Satisfaction Customer Satisfaction In-person interview Focus group interview Exit interview Comment cards Surveys Complaint/ Compliment analysis Mystery shopping

Satisfaction Surveys Gaps Model of Service Quality – LibQUAL –Affect of service –Information control –Library as place Counting Opinions – LibSat DIY

Customer Satisfaction Customer comments Net Promoter Score (NPS) Opportunity index Resolution rate

Value of the Library Value of the Library Personal & Organizational Benefits Economic Benefits Direct Indirect Nonuse Indirect

Value of the Special Library Cognitive results Affective results Meeting expectations Task accomplishments Time aspects Money aspects

Value of the Academic Library StudentFacultyInstitution Recruitment/Enrollment Retention & Graduation Success Achievement Learning Experience, Attitude & Perception of Quality Research Productivity Grants Teaching Reputation & Prestige

Value of the Public Library Generic Learning Outcomes Knowledge & Understanding Attitudes & Values Activity, Behavior & Progression Enjoyment, Inspiration & Creativity Skills

Communicating the Results Library staff members Friends groups Prospective donors Board of Trustees Community governing boards Institutional stakeholders Library administrative agencies State legislatures Manager/Partner Customers –Faculty –Graduate students –Undergraduate students –Others

1.Accountability is part of librarianship 2.Austerity increases the need for evaluation 3.Librarians are engaged in organizational change management 4.Evaluation is not brain surgery

5.Meaningful evaluation focuses on customers and demonstrates the value of the library to them 6.The hardest thing is to start – “Just do it” 7.Play with the various concepts

8. Make evaluation a part of your regular work 9. Share the results 10. Implement a results reporting system

Joe AT JoeMatthews.Org