Tools and Techniques to Clean Up your Database Meghan Weeks Systems Librarian Loyola Marymount University Southern California IUG Meeting Oct. 10, 2012

Overview Why is it important to have a clean database? Where to start with a database cleanup project? How does bad data get into the database? How do you find bad data? How do you fix the problems? How do you prevent issues in the future?

Why is it important to have a clean database? OPAC searching issues OPAC display issues Statistics not accurate Web Management Reports not accurate Create lists not accurate

Where to start? Millennium Database Control Working Group Formed in Oct. 0f 2010 Biweekly meetings Members from Acquisitions, Cataloging, Circulation, Media & Reserves, Special Collections, and Systems Systematically go through each record type and review all of the fixed length and variable length fields Clean out obsolete fields or values If needed, create new fields or values Document what the fields are used for and by whom

Document, Document, Document!

How does bad data get into the database? A fixed length field code value is deleted from the table but some records still contain that code Templates contain invalid entries or no entries

How does bad data get into the database? Valid code with blank label Valid code but it is a space Valid codes used incorrectly Bib location in an item record Old codes still in the system and being used

How do you find bad data? Patrons and/or staff may report issues that they find in the OPAC Wrong material type icon Limiting/Scoping not working as expected Staff report checkouts giving incorrect due dates Invalid or wrong patron type, item type, or location Staff report holds being placed on items that should not circulate Wrong status or item type

How do you find bad data? Use field statistics for all record types

Field Statistics Reports Look for Bad Code Null ` ` Unknown

Field Statistics Reports – Fixing Bad Code Use Create Lists and Rapid Update

Field Statistics Reports – Fixing Bad Code Use Create Lists and Rapid Update

Field Statistics – Call Numbers not in SCAT

Gap in SCAT Table Review File After you run field statistics on bib records a report is automatically generated for call numbers not in SCAT table View report by selecting an empty review file and choose copy Scroll down toward the bottom of the list and select Bibs: call numbers not in SCAT Why are call numbers not in the SCAT table? Adjust the SCAT table or fix records with invalid call numbers

How do you find bad data? Web Management Reports Circulation statistics – stats group issues

How do you find bad data? Headings Reports (Cataloging) Duplicate entries Item records – barcodes, ISSN, ISBN Patron records – barcodes Blind references Subject authority records Name authority records Other headings reports

How do you prevent issues in the future? Edit Preferences (in Login Manager) for each user and make sure invalid text is highlighted

How do you prevent issues in the future? Follow best practices for deleting: Location codes (107738) Fund records (100710) Vendor records (105790) Check templates for all record types for bad code and missing entries and add prompts Periodically review load tables Periodically review fixed length field code values – no blank labels and don’t use a space as a valid code

How do you prevent issues in the future? Check the manual regarding the fixed length field code values that you want to delete or alter. Source: http://csdirect.iii.com/documentation/badcode.shtml

How do you prevent issues in the future? Utilize automatic link maintenance or manually run link maintenance every day Make sure the location mapping table is accurate Command line interface (aka telnet) - A, L, E Make sure the MARC Validation table is up-to-date A, A, S, O ,D ,1 Review scope rules for accuracy Delete staff accounts when they leave the organization Periodically review staff authorizations

References and Useful Links Fixing Bad Codes FAQ on CSDirect http://csdirect.iii.com/documentation/badcode.shtml Cataloging Clean up Projects http://csdirect.iii.com/documentation/catcleanprojects.shtml Getting Started with Millennium Statistics http://csdirect.iii.com/documentation/startmilstats.shtml Create and Interpret Reports in Millennium Statistics http://csdirect.iii.com/documentation/milstats.shtml Cleaning Up Your Database Presentation by Amy Homick, 2012 IUG Conference, Chicago, IL

Thank you! Meghan Weeks Meghan.Weeks@lmu.edu Questions? Thank you! Meghan Weeks Meghan.Weeks@lmu.edu