Lessons Learned: How to Survive the Development of Outcome Measures Cherie McCraw, M.L.S., Library Consultant, Division of Library and Information Services Linda B. Schrader, Ph.D., Florida State University

Overview u Overview of the four-year implementation process u Development of an outcome evaluation system for the State Library of Florida u Examination of lessons learned and challenges encountered

History u 1993 Government Performance Results Act –Requires federal agencies to establish specific objective, quantifiable, and measurable performance goals for each program u 1996 reauthorization of Library Services and Technology Act (LSTA)

LSTA Grant Background u State-based program u Focuses on information access through technology and information empowerment through special services u Emphasizes public libraries, but encourages all types of libraries to apply u Emphasizes accountability and evaluation

Institute of Museum and Library Services (IMLS) u Turned to outcome-based evaluation for LSTA grants –United Way model –Looks at the effect of an institution's activities and services on the people it serves –Powerful tool for identifying successes and communicating their value

IMLS Plan u Five states were selected to implement outcomes evaluation of LSTA grants early and develop model programs –Florida –Minnesota –North Carolina –Maryland –Washington

Floridas Four Year Implementation Plan u Development of outcome model and standardized outcomes u Marketing of outcomes evaluation process u Planning for staff training and technical assistance u Conducting outcome evaluations

Implementation Activities u Administrative and management tools u Senior level buy in u State Library and field training

1998 IMLS selects states for pilot projects 1998Consultant hired to help develop model 1998 State Library staff receive United Way Outcomes Evaluation training Oct 1998 Outcomes Evaluation concept presented at annual Public Library Directors Conference 1999 Standardized outcomes for 6 LSTA categories developed 1999 OPPAGA Review favorable toward outcome concept Oct 1999 Outcomes Evaluation program presented at annual Public Library Directors Conference Nov 1999 Born to Read Capacity Building Workshop (outcome plans developed) July 1999 Floridas LSTA Outcome Evaluation Plan published Fall 1999 Workshops on outcome evaluation presented Jan 2000LSTA Grants workshop featured new guidelines Mar 2000 LSTA applications required outcome plan (not scored) Sep 2000 Workbook: Outcome Measurement of Library Programs published Oct 2000Panel discussion of outcomes at annual Public Library Directors Conference Nov 2000 LSTA Capacity Building Workshop (outcome plans refined) Oct 2000 Outcomes based LSTA grants began 2000LSTA categories changed from 6 to 2/new standardized outcomes created Mar 2001LSTA applications required outcome plan (scored) Jul 2001Outcomes based evaluation workshops conducted Nov 2001LSTA Capacity Building Workshop (evaluation strategies and tools) Nov 2001LSTA annual reports reflected outcomes evaluation Dec 2001LSTA Grants workshop included session on outcomes State Library of Floridas Outcomes Timeline

Development of Outcome Model u Hired consultant u Adapted United Way Outcome Model u Identified standardized state outcomes u Revised rules, guidelines and forms

Outcome Model Components u Inputs u Activities u Outputs u Outcomes u Evaluation Indicators u Evaluation Sources/Methods

Standardized State Outcomes u Characteristics –Broad scope, relevant –Easily understandable –Focus on skills/behaviors –Can include quantitative and qualitative indicators –Tailored to local programs

Standardized State Outcomes u Access for persons having difficulty using libraries –Persons having difficulty using library services use services or information that were not previously available u Library Technology Connectivity and Services –Public uses technology to get information –Public learns to use technology

Access for persons having difficulty using libraries u Golden Gateways u Library Family Learning Centers u Small Business Information Resource Center u REACH: REmote ACcess to the Homebound u Parents Plus u Born to Read u Juniors to Seniors: Hillsborough Remembers u Library Elderly Outreach (LEO) Project

Library Technology Connectivity and Services u Development of the Everglades Information Center u Electronic Library: Community Training and Outreach u Seniors Jacksonville Public Library u Mobile Training Lab u Osceola Internet Improvement u Library Automation: Moving Legal Information Toward the Public u Mi Servicio de Biblioteca

Marketing the Outcome Process u State leadership vanguard for change –Paradigm shift in what gets measured u Senior level buy in of new evaluation process u Continuous communication throughout the four year period u Ongoing support

Staff Training u Developmental approach to training u Technical assistance provided by regional library consortiums and state library consultants u Statewide conferences

Conducting Outcome Evaluations u Phased in requirements for outcome evaluations in grant process u Continued training and support to field staff u Initial review of annual reports

Lessons Learned A State Library assumed leadership role B Implementation phased in over time C Standardized the outcomes D Measurement tools could be the same E Better communication between libraries and grant partners F Multiple training opportunities

Challenges A Resistance from State and field staff B Cannot be implemented quickly C Adapting local programs to the standardized outcomes D Tailoring existing data collection tools E Libraries and grant partners interpret outcomes evaluation differently F Different levels of knowledge about outcomes

Where Do We Go From Here? u Continue training field librarians u Promote/pilot Outcomes Model for other library programs u Develop new Long Range Plan using outcome measurement