Checklist for Digital/Hybrid Library Web Sites Based on Pitchmann's (2001) article Includes content as well as interface design and underlying technology Expandable to metrics which relate content, operations/processes and systems/technological performance E. O’Brien Digital Libraries Term Project Due May 13, 2002

Libraries Evaluated: MIT – University of Delaware – UIUC – Rice (Fondren) – Trinity U. (Coates) – Georgetown U. – Rutgers U. – E. O’Brien Digital Libraries Term Project Due May 13, 2002

Sample Criteria - Content Is there one formal selection policy statement for the whole library on each website? Is there any statement on the web site that helps users understand that the services offered are selective and quality controlled? Does the library identify whether or not the collection is: –Comprehensive –Research –Study –Basic –Minimal E. O’Brien Digital Libraries Term Project Due May 13, 2002

Sample Criteria – Operations/Processes Is the research process leading to the creation of the content described? Is there evidence that the intended audience has been identified? Does the site adhere to recognized standards and appropriate technologies? What level of user support is available? (FAQs only, interactive support, help screens, contact information) E. O’Brien Digital Libraries Term Project Due May 13, 2002

Sample Criteria – Systems/Technological Performance Are there process or technical criteria? ( Technical features that measure the integrity of the site and availability of content reported to be provided) Information integrity ( Age of site, frequency of updates, frequency of revisions, version control, link maintenance) System integrity ( Server stability, frequency of downtime, response time, traffic volume and delays, downtime announcements, projected availability time and date announcements) What metadata are used at each web site? E. O’Brien Digital Libraries Term Project Due May 13, 2002

Content, process, technology and metrics Components of electronic libraries (Lakos, 1997) –Infrastructure –Services –Contents –Management –Ownership Performance measurement compares what library is doing (actual performance) with what it means to do (mission) and what it wants to achieve (goals) E. O’Brien Digital Libraries Term Project Due May 13, 2002

Collection development policy statement AcquisitionCataloging LinkingDigitizing ServingRetrievingUsing E. O’Brien Digital Libraries Term Project Due May 13, 2002 Connecting the components:

Total !Total ?Total - MIT30813 UIUC28718 Rice27915 UDEL24918 Trinity RU18924 GU91527 Results:

Highlights: Rice/UIUC had excellent collection development policy descriptions Only Trinity U describes mission, objectives and progress assessment UIUC documents library staff operations yr/yr and compares data to other libraries. Only GU and RU did not have info about quality control and selectivity of services Only RU announced anticipated downtimes All libraries described how they identified different user populations MIT, UDEL, UIUC had process criteria to measure integrity of site and availability of content E. O’Brien Digital Libraries Term Project Due May 13, 2002

Conclusions: Pitchmann's criteria can be adapted to evaluation of academic digital libraries There may be better sources for evaluation (Brophy, 1998 & Lakos, 1997, others) Libraries don’t plan for evaluation & performance measurement Plenty of room for future work in this area but it can be done! E. O’Brien Digital Libraries Term Project Due May 13, 2002

Specific recommendations for Rutgers University Libraries are: Define AAU criteria that libraries are to help RU meet. State how libraries will contribute. Include measurements of progress toward goals and when and to whom results will be reported. Create mission or vision statement for libraries that includes mission, goals, current performance. (See UIUC, MIT, Trinity U. for examples.) Develop metrics for infrastructure, services, content, management and ownership of libraries that link the impacts of performance in each area. (See Lakos article.) Provide information on RU libraries’ web site about service and collection quality. (See MIT, UDEL, Rice and Trinity U. as examples.) Update all pages on web sites that deal with mission statement, metrics, collection development. Add information to web sites about how interests and needs of users are explored and how often their input is sought. (See MIT, UIUC as examples.) E. O’Brien Digital Libraries Term Project Due May 13, 2002