A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Charles R. McClure Jordon Andrade College of Information Florida State Tapping the vast reservoir of human knowledge --Louis Round Wilson, founder, 1931

Tapping the vast reservoir of human knowledge --Louis Round Wilson, founder, 1931

Research questions 1.What methodologies are used in LSTA evaluations? 2.How do states’ goals map to LSTA goals? 3.What, if any, correlations are there between methodologies used and states’ and LSTA goals? 4.How successful are different methodologies in providing useful evaluation data about library programs?

Background on IMLS and LSTA 2008: IMLS awarded ~$161 M under Grants to States program LSTA funds < 15% of total state library funding But ≈ 97% of all federal funding to state libraries IMLS encourages grantees to use Outcome Based Evaluation methods

LSTA goals 1.Expand services for learning and access to information and educational resources in a variety of formats, in all types of libraries, for individuals of all ages; 2.Develop library services that provide all users access to information through local, state, regional, national, and international electronic networks; 3.Provide electronic and other linkages between and among all types of libraries; 4.Develop public and private partnerships with other agencies and community-based organizations; 5.Target library services to individuals of diverse geographic, cultural, and socioeconomic backgrounds, to individuals with disabilities, and to individuals with limited functional literacy or information skills; and 6.Target library and information services to persons having difficulty using a library and to underserved urban and rural communities, including children from families with incomes below the poverty line.

Rendon’s LSTA goal categories A: Establish or enhance electronic linkages among or between libraries. B: Electronically linking libraries with educational, social, or information services. C: Assisting libraries in accessing information through electronic networks. D: Encouraging libraries in different areas, & encouraging different types of libraries to establish consortia & share resources. E: Paying costs for libraries to acquire or share computer systems and telecommunications technologies. F: Targeting library & information services to persons having difficulty using a library & to underserved urban & rural communities, including children from families with incomes below the poverty line. The Rendon Group (2003). National Profile: Analyses of the Five-Year Evaluations submitted to the Institute of Museum and Library Services by the State Library Administrative Agencies under the Grants to States program of the Library Services and Technology Act. Washington, DC: Institute of Museum and Library Services.

Methodology Collected 5-year plans & evaluation reports for 28 states for & From 5-year plans: 1.State’s 5-year goals, 2.Connection between state’s goals and LSTA’s goals. From evaluation reports: 1.Methodologies and data collection instruments used, 2.Stakeholder groups that provided data, 3.Accomplishment of LSTA goals, 4.Recommendations.

Methods used in LSTA evaluations Methodologies used Number of methods used

Data collected from whom

States’ goals mapping to Rendon

Accomplishment of states’ goals

Accomplishment of Rendon goals

Methods used to evaluate goals

Discussion Quality and readability of eval reports varied considerably. Need more explicit mapping between state & LSTA goals, and eval methodologies. Only 39% of goals accomplished completely.

Recommendations To libraries: – Need for clearer goals & measurable objectives. – Need for more appropriate methods. To IMLS: – Not just OBE. – Simplify the process. To both: – Need for more consistent use of terms. – Need for planning pre-eval & re-program implementation.

Answers to research questions 1.What methodologies are used in LSTA evaluations? A lot of surveys. 2.How do states’ goals map to LSTA goals? Not very well. 3.What, if any, correlations are there between methodologies used and states’ and LSTA goals? Not many. 4.How successful are different methodologies in providing useful evaluation data about library programs? Not very.

Thank you! Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Charles R. McClure Jordon Andrade College of Information Florida State Tapping the vast reservoir of human knowledge --Louis Round Wilson, founder, 1931