1. Batchloading: Current Practices and Future Challenges Rebecca L. Mugridge Pennsylvania State University Libraries American Library Association January 22, 2012

2.  Batchloading of bibliographic records for aggregated e-resource and microform collections into the online catalog has become a very time consuming workflow for Penn State  We were interested in how other large research libraries were addressing challenges related to batchloading Why this topic?

3.  Survey the members of the ALCTS Technical Services Directors of Large Research Libraries Interest Group (i.e., “Big Heads”)  Likely that this group would be very familiar with these issues  Likely that this group also very involved in these activities  Likely that this group would also be very interested in the research results Survey proposal

4.  24 largest ARL libraries in the U.S. and Canada  1 non-ARL library (Stanford)  2 public libraries  3 national libraries Who are the Big Heads?

5.  Ten sections; 57 questions addressing:  Demographics  Staffing  Budgets devoted to batchloading activities  Scope of batchloading activities  Management  Workflow  Quality standards  Collaborative efforts  Information technology support  Assessment Survey structure

6.  Survey reviewed by the university Office for Research Protections  It was determined that the survey did not need to be submitted for review because it did not collect information about human subjects Institutional review

7.  Survey distributed in December 2010  Low initial response rate (23%)  Followup personalized emails resulted in a 60% response rate Timeline

8.  18 respondents  Integrated library systems:  Ex libris  SirsiDynix  Innovative Interfaces  Sizes of online catalogs range from one million to more than 12 million records Demographics

9.  Overall staff size varied widely:  Librarians (45-266)  Staff (60-488)  Staff devoted to batchloading:  Librarians averaged 1.66  Other professionals averaged.897  Support staff averaged 1.51 Staff involved with batchloading activities

10.  All respondents had redefined existing positions to add responsibility for batchloading  21% had redefined existing positions to be dedicated to batchloading  21% had created new positions to be dedicated to batchloading  66% of respondents expected to devote more staff to batchloading in the next 5 years Staff, cont’d

11.  24% of responding libraries had a dedicated budget to support batchloading activities  Amounts spent on records ranged from $1,000 to >$100,000  Not necessarily a correlation between amount spent and number of records purchased Budgets

12.  88% Collections  65% Operations  18% Special funds  18% Endowments  6% Grant monies (Respondents could select multiple funding sources) Source of funding

13.  Within the past three years  12% loaded 100,000 – 200,000 records  18% loaded 200,000 – 500,000 records  70% loaded more than 500,000 records Scope of batchloading activities

14.  Records deleted per year  18% deleted fewer than 100,000 records  29% deleted 1,000 – 5,000 records  24% deleted 5,000 – 10,000 records  12% deleted 10,000 – 50,000 records  12% deleted 50,000 – 100,000 records  6% deleted over 100,000 records Scope, cont’d

15.  Cancellation of subscription to online resource  Withdrawal of physical item  Invalid URLs  Errors found in records  Routine maintenance Reasons for deleting records

16.  Cataloging (76%)  IT/Systems (53%)  Acquisitions (41%)  Collection development (18%)  Public services (6%)  Other  E-Resources Management Section  E-resources, Serials, and Database Management  Scholarly Resource Integration  Knowledge Access & Resource Management Services What unit is responsible for managing batchloading activities?

17.  Cataloging (71%)  IT/Systems (41%)  Acquisitions (12%)  Other (24%)  E-Resources Management Section  E-resources, Serials, and Database Management  Scholarly Resource Integration  Knowledge Access & Resource Management Services Primary responsibility for managing batchloading activities

18.  Length of time it takes to load a file:  2 libraries reported: within one week  One library reported: 2 – 3 weeks  24% reported: within a month  24% reported: 2 – 3 months  35% reported: > 3 months  65% reported the time to be unacceptable Workflow

19.  59% of respondents use alternatives to batchloading for at least some electronic resources:  WorldCat Local  SFX Find It deep linking  Primo Central  Summon  MetaLib Alternatives to batchloading records into the online catalog

20.  65% of respondents provide access to Google books through their ILS (92% of them only for titles that they own(ed) in print)  59% used the Google API  24% used other methods:  In-house script written by IT staff  Persistent URLs added to print record  Open URL resolver  Selective searching of Google books by subject Google books

21.  35% of respondents provide access to materials in HathiTrust through their ILS  Of those six libraries, only one reported that they provided access to materials for which they did not own the print version  5 libraries reported use of the HathiTrust API  1 library batchloaded records for their HathiTrust materials HathiTrust

22.  56% of respondents batchload records for freely- available web resources  Selected by subject specialists/selectors (47%) Freely-available web resources

23.  82% reported that batchloading has lowered their metadata standards  18% reported that batchloading did not impact their metadata standards  No libraries reported that metadata standards were improved by their batchloading activities Quality standards

24.  100% visual review (sampling)  53% automated validation using MarcEdit or other software How is quality assessed?

25.  94% edit records using MarcEdit or a similar program  53% reported accepting and loading records “as is”  29% reported editing records manually  12% reported not loading records with poor quality metadata How do libraries deal with problematic metadata

26.  76% reported rejecting records based on metadata quality issues  Some reasons cited:  Lack of authorized headings or subject access  Incomplete fields  Character encoding errors  Technical limitations  Records lacked unique identifiers  Created duplicates in the online catalog Rejected records

27.  Two approaches:  Collaboration among libraries within consortia or other groups  Collaboration between libraries and vendors to improve vendor services  Collaboration is opportunistic and not repeated or programmatic Collaborative efforts

28.  82% reported that IT support is necessary to perform batchloading activities  IT needs include:  Record customization  Programming  Troubleshooting  Running system reports  Batch deleting  Managing staff authorizations  Creating FTP scripts for file management Information Technology

29.  88% reported that their ILSs presented technical challenges to effective management of batchloading activities:  An inability to "mark" records for deletion  Lack of sophistication and customizability of ILS loaders  Inability to effectively match and replace or overlay records  Limit of loading a certain (relatively small) number of records at one time  Extremely limited global edit options  And many more! Technical challenges of ILSs

30.  Usage data (50%)  End user feedback (41%)  Faculty review (12%)  Focus group (6%)  Other:  Review by libraries-wide task force  Review by bibliographers  Quality assurance testing  Staff review  Error reports Assessment of batchloading activities

31.  All but one library informed one or more constituents when batchloading activities are completed:  71% inform all library staff  53% inform subject specialists or selectors  3 libraries inform appropriate academic departments Communication

32.  Inconsistent record quality (82%)  Staffing (76%)  Ongoing maintenance (59%)  Vendor technical support (53%)  Local technical support (47%)  Funding purchase of records (35%) Current batchloading challenges

33.  Inconsistent record quality (82%)  Staffing (82%)  Ongoing maintenance (65%)  Vendor technical support (65%)  Local technical support (47%)  Funding purchase of records (65%) Future batchloading challenges

34.  Notification from vendors (82%)  Feedback from library patrons (76%)  Feedback from subject specialists (71%)  Periodic reviews by staff (35%)  Others included:  Re-loading of batchloaded records  Subscribing to OCLC update services  URL checker Maintenance

35.  Website for reporting problems (88%)  E-mail (82%)  Telephone (47%)  Other:  Questionpoint or other chat reference  Online catalog error report functionality  Electronic discussion list  In-person reporting at service point Error notifications

36.  62% never check URLs  19% check URLs irregularly  One library checks them on a quarterly basis  One library checks them on a monthly basis  One library runs a link checker on resources included in a separate database, but never against the entire ILS  One library uses the SFX link resolver service URL maintenance

37.  Improving the functionality of the ILS  Collaborative efforts  Working with vendors earlier in the process  Consortial efforts  Establishment of “best practices” (e.g., MARC Record Guide for Monograph Aggregator Vendors, 2nd Edition)  Better communication Opportunities for improvement

38.  Quality of records  Assessment methodology  Impact of collaborative efforts Areas for future research

39. Rebecca L. Mugridge Head, Cataloging and Metadata Services Pennsylvania State University Libraries 126 Paterno Library University Park, PA 16802 iym6@psu.edu Phone: 814-865-1850 Fax: 814-863-7293 Jeff Edmunds Monographs Cataloger and Batchloading Specialist Pennsylvania State University Libraries 126 Paterno Library University Park, PA 16802 jhe2@psu.edu Phone: 814-867-0798 Fax: 814-863-7293 For more information: