Batchloading: Current Practices and Future Challenges Rebecca L. Mugridge Pennsylvania State University Libraries American Library Association January 22, 2012

 Batchloading of bibliographic records for aggregated e-resource and microform collections into the online catalog has become a very time consuming workflow for Penn State  We were interested in how other large research libraries were addressing challenges related to batchloading Why this topic?

 Survey the members of the ALCTS Technical Services Directors of Large Research Libraries Interest Group (i.e., “Big Heads”)  Likely that this group would be very familiar with these issues  Likely that this group also very involved in these activities  Likely that this group would also be very interested in the research results Survey proposal

 24 largest ARL libraries in the U.S. and Canada  1 non-ARL library (Stanford)  2 public libraries  3 national libraries Who are the Big Heads?

 Ten sections; 57 questions addressing:  Demographics  Staffing  Budgets devoted to batchloading activities  Scope of batchloading activities  Management  Workflow  Quality standards  Collaborative efforts  Information technology support  Assessment Survey structure

 Survey reviewed by the university Office for Research Protections  It was determined that the survey did not need to be submitted for review because it did not collect information about human subjects Institutional review

 Survey distributed in December 2010  Low initial response rate (23%)  Followup personalized s resulted in a 60% response rate Timeline

 18 respondents  Integrated library systems:  Ex libris  SirsiDynix  Innovative Interfaces  Sizes of online catalogs range from one million to more than 12 million records Demographics

 Overall staff size varied widely:  Librarians (45-266)  Staff (60-488)  Staff devoted to batchloading:  Librarians averaged 1.66  Other professionals averaged.897  Support staff averaged 1.51 Staff involved with batchloading activities

 All respondents had redefined existing positions to add responsibility for batchloading  21% had redefined existing positions to be dedicated to batchloading  21% had created new positions to be dedicated to batchloading  66% of respondents expected to devote more staff to batchloading in the next 5 years Staff, cont’d

 24% of responding libraries had a dedicated budget to support batchloading activities  Amounts spent on records ranged from $1,000 to >$100,000  Not necessarily a correlation between amount spent and number of records purchased Budgets

 88% Collections  65% Operations  18% Special funds  18% Endowments  6% Grant monies (Respondents could select multiple funding sources) Source of funding

 Within the past three years  12% loaded 100,000 – 200,000 records  18% loaded 200,000 – 500,000 records  70% loaded more than 500,000 records Scope of batchloading activities

 Records deleted per year  18% deleted fewer than 100,000 records  29% deleted 1,000 – 5,000 records  24% deleted 5,000 – 10,000 records  12% deleted 10,000 – 50,000 records  12% deleted 50,000 – 100,000 records  6% deleted over 100,000 records Scope, cont’d

 Cancellation of subscription to online resource  Withdrawal of physical item  Invalid URLs  Errors found in records  Routine maintenance Reasons for deleting records

 Cataloging (76%)  IT/Systems (53%)  Acquisitions (41%)  Collection development (18%)  Public services (6%)  Other  E-Resources Management Section  E-resources, Serials, and Database Management  Scholarly Resource Integration  Knowledge Access & Resource Management Services What unit is responsible for managing batchloading activities?

 Cataloging (71%)  IT/Systems (41%)  Acquisitions (12%)  Other (24%)  E-Resources Management Section  E-resources, Serials, and Database Management  Scholarly Resource Integration  Knowledge Access & Resource Management Services Primary responsibility for managing batchloading activities

 Length of time it takes to load a file:  2 libraries reported: within one week  One library reported: 2 – 3 weeks  24% reported: within a month  24% reported: 2 – 3 months  35% reported: > 3 months  65% reported the time to be unacceptable Workflow

 59% of respondents use alternatives to batchloading for at least some electronic resources:  WorldCat Local  SFX Find It deep linking  Primo Central  Summon  MetaLib Alternatives to batchloading records into the online catalog

 65% of respondents provide access to Google books through their ILS (92% of them only for titles that they own(ed) in print)  59% used the Google API  24% used other methods:  In-house script written by IT staff  Persistent URLs added to print record  Open URL resolver  Selective searching of Google books by subject Google books

 35% of respondents provide access to materials in HathiTrust through their ILS  Of those six libraries, only one reported that they provided access to materials for which they did not own the print version  5 libraries reported use of the HathiTrust API  1 library batchloaded records for their HathiTrust materials HathiTrust

 56% of respondents batchload records for freely- available web resources  Selected by subject specialists/selectors (47%) Freely-available web resources

 82% reported that batchloading has lowered their metadata standards  18% reported that batchloading did not impact their metadata standards  No libraries reported that metadata standards were improved by their batchloading activities Quality standards

 100% visual review (sampling)  53% automated validation using MarcEdit or other software How is quality assessed?

 94% edit records using MarcEdit or a similar program  53% reported accepting and loading records “as is”  29% reported editing records manually  12% reported not loading records with poor quality metadata How do libraries deal with problematic metadata

 76% reported rejecting records based on metadata quality issues  Some reasons cited:  Lack of authorized headings or subject access  Incomplete fields  Character encoding errors  Technical limitations  Records lacked unique identifiers  Created duplicates in the online catalog Rejected records

 Two approaches:  Collaboration among libraries within consortia or other groups  Collaboration between libraries and vendors to improve vendor services  Collaboration is opportunistic and not repeated or programmatic Collaborative efforts

 82% reported that IT support is necessary to perform batchloading activities  IT needs include:  Record customization  Programming  Troubleshooting  Running system reports  Batch deleting  Managing staff authorizations  Creating FTP scripts for file management Information Technology

 88% reported that their ILSs presented technical challenges to effective management of batchloading activities:  An inability to "mark" records for deletion  Lack of sophistication and customizability of ILS loaders  Inability to effectively match and replace or overlay records  Limit of loading a certain (relatively small) number of records at one time  Extremely limited global edit options  And many more! Technical challenges of ILSs

 Usage data (50%)  End user feedback (41%)  Faculty review (12%)  Focus group (6%)  Other:  Review by libraries-wide task force  Review by bibliographers  Quality assurance testing  Staff review  Error reports Assessment of batchloading activities

 All but one library informed one or more constituents when batchloading activities are completed:  71% inform all library staff  53% inform subject specialists or selectors  3 libraries inform appropriate academic departments Communication

 Inconsistent record quality (82%)  Staffing (76%)  Ongoing maintenance (59%)  Vendor technical support (53%)  Local technical support (47%)  Funding purchase of records (35%) Current batchloading challenges

 Inconsistent record quality (82%)  Staffing (82%)  Ongoing maintenance (65%)  Vendor technical support (65%)  Local technical support (47%)  Funding purchase of records (65%) Future batchloading challenges

 Notification from vendors (82%)  Feedback from library patrons (76%)  Feedback from subject specialists (71%)  Periodic reviews by staff (35%)  Others included:  Re-loading of batchloaded records  Subscribing to OCLC update services  URL checker Maintenance

 Website for reporting problems (88%)  (82%)  Telephone (47%)  Other:  Questionpoint or other chat reference  Online catalog error report functionality  Electronic discussion list  In-person reporting at service point Error notifications

 62% never check URLs  19% check URLs irregularly  One library checks them on a quarterly basis  One library checks them on a monthly basis  One library runs a link checker on resources included in a separate database, but never against the entire ILS  One library uses the SFX link resolver service URL maintenance

 Improving the functionality of the ILS  Collaborative efforts  Working with vendors earlier in the process  Consortial efforts  Establishment of “best practices” (e.g., MARC Record Guide for Monograph Aggregator Vendors, 2nd Edition)  Better communication Opportunities for improvement

 Quality of records  Assessment methodology  Impact of collaborative efforts Areas for future research

Rebecca L. Mugridge Head, Cataloging and Metadata Services Pennsylvania State University Libraries 126 Paterno Library University Park, PA Phone: Fax: Jeff Edmunds Monographs Cataloger and Batchloading Specialist Pennsylvania State University Libraries 126 Paterno Library University Park, PA Phone: Fax: For more information: