1. www.digiqual.org DigiQUAL™ DigiQUAL™: a Digital Library Evaluation Service Presented at 7 th Northumbria by Martha Kyrillidou, ARL Bruce Thompson, Texas A&M South Africa, August 14, 2007

2. What are some of the current developments with library assessments efforts? ARL StatsQUAL ™ E-Metrics LibQUAL+ ® DigiQUAL ™ MINES for Libraries ™ Where are the most critical needs and opportunities? What are the lessons learned? Library Assessment in an Electronic Era www.arl.org

3. www.digiqual.org DigiQUAL™ NSF Funding Building on the LibQUAL+ ® experience Secures feedback on user’s perceptions of library’s web site Five questions on services, functionality, and content Goal is to determine utility, reliability, and trustworthiness

4. Background: ServQUAL  LibQUAL+ ®  DigiQUAL ™ LibQUAL+ ® Dimensions of Service Quality: Affect of Service Information Control Library as Place Developing DigiQUAL ™ Survey Items DigiQUAL™ 12 themes of service quality: Accessibility Navigability Interoperability Collection building Resource Use Evaluating collections DL as community for users DL as community for developers DL as community for reviewers Copyright Role of Federations DL Sustainability www.digiqual.org

5. Evaluating the NSF National Science Digital Library Collections Categories and Themes from DLESE and MERLOT focus Groups

6. I. Web attractiveness; Design features “I think the homepage is too cluttered.” “I really appreciate no pop-ups.” “I would want a site that is truly simplistic; there’s a lot of initial information on the homepage.”

7. II.Accessibility; Navigability “…there’s a lot of links, you know, like that– [unusable, without a username]..” “…I think another step is that the interfacing needs to be designed in such a way that the data is taken advantage of in simple, easy-to-use, and intuitive.” “Vocabulary is an issue.”

8. III.Interoperability of the Sites “Yes and no. [Interoperability] depends on the learning object that you access.” “I think there’s a large group of educators out there that are certainly capable, knowing content, but actually using the computer, using things in that domain—it’s very difficult for them. Again, it’s something new to them; not that they’re stupid or something like that.”

9. Evaluating the NSF National Science Digital Library Collections Social and Psychological Aspects of the Digital Libraries and their Collections

10. IV.Library and Digital Library as “Community” “I’m beginning to understand now that this whole notion of community is incredibly important— probably far more important that us coming up any ‘grand schema’, if you will.”

11. Library and Digital Library as “Community” continued “I think [community] brings to mind two very important things: 1) the notion of community within and between MERLOT…and 2) I was just kind of blown away by the fact that [the physicists] would completely ignore any kind of ‘knowledge’ that we obviously have in the library community on how things should be done, and they built this thing along the lines of how they actually think!”

12. V.Collection Building “Collection building…we will forever be in a collection-building phase as far as I can tell.” “It was previously in its rapid growth stage and trying to figure out who we are and how to tell it to people. And now we’re into what don’t have and why can’t people find what they want.”

13. VI.Meta-data issues Sample Responses: “The way around meta-data issues, is ‘cross- walking’.” “…and the critical thing is we’re still developing a control vocabulary for the library.”

14. Meta-data issues (cont’d) “My impression if that users want to have resources described in a way that anticipates the things that they want in that resource, whether it is a Website or a learning object, how granular it is [that is, is it an image, or is it an entire website?].”

15. VII.Copyright Issues; Ownership of Materials “But [the copyright/ownership issue] does suggest something about ownership and about the way we now view knowledge as a multi-person constructed set of constructions.” “The people who create it…have ownership.” “Who owns the content? You own your own content, but we own the meta-data. I mean, we can have the meta-data…The meta-data can be harvested by NSF.”

16. VIII.How Digital Library Resources are Used “And what I found is that they [teachers who utilize the digital libraries] really sought the animations, the quick things that they could take into classrooms 10 minutes before class.” “I use MERLOT basically to show all my faculty how to find online course objects that they can use in conjunction with their regular classroom instruction.”

17. How Digital Library Resources are Used (cont’d) “So I use both a search and a harvest. I have to tell you that I also harvest stuff. There are tools out there…some of them very specialized, some of them not so specialized. And I must tell you that I shamelessly harvest stuff, and I’m sure I’m committing all kinds of copyright violations on a daily basis…But I do that because I’m fearful that the stuff will sometimes disappear, and so I harvest that. So I’m both a searcher and a harvester. And that’s how I use MERLOT.”

18. X.Issues of Sustainability, Financing, Support “I would ask [NSF] for the strategy of sustainability. Because when it comes down to it, if it can’t be sustained, then it really doesn’t matter.” “And when you talk about sustainability, Chris, are you talking about how are we going to keep the Federal government funding it? Or are you talking about how do we keep growing the collection and the community?” “Both”

19. IX.Evaluation Issues for Users, Developers, Reviewers and Others “Mainly, the broken links are the problem for me.” “I guess because there’s so much information there, it wasn’t as intuitive as a lot of Websites that I’ve been to before. And I went to it a lot.

20. Evaluation Issues continued “The one thing I’ve discovered with mine [my teachers], just having real-time data isn’t enough…It’s not just the data, you have to have the documentation; and it doesn’t have to be a full lesson plan, Just enough so that they can know the lesson plan.”

21. Technical Composition Human/System Interaction Content Community User/Creator Vetting Process  Access  Reliability  Trustworthiness/A ccuracy  Scope & Sequence  Active Links  Browsability  Organized  Navigability  Self-sufficiency  Trustworthiness (feels “right)  Usability  Fulfills purpose/useful As a social network Takers and givers Preservation Propagation DIGITAL LIBRARY ENVIRONMENT

22. Pilot Testing Survey Items and Implementation www.digiqual.org

23. Building a Survey Review and select items Issue: aligning items to individual DL needs & users – vocabulary and content www.digiqual.org

24. Customize Survey Issue: Flexibility vs. Standardization Building a Survey, cont. www.digiqual.org

25. Implementing Survey – Notification Methods Links on site Newsletters Next to resources Issues: no pop-ups, no individual emails www.digiqual.org

26. Implementing Survey - Incentives Issues: must be easily transferable, requires email address - clear IRB www.digiqual.org

27. (mis)Interpreting results from individual digital libraries in the context of other sites Sites reluctant to share data and results Analysis and Reporting - Issues www.digiqual.org

28. Outstanding Issues and Challenges Unique DLs: niche market, critical mass, both? Balance: – custom vs. generic content  results – flexible vs. standard implementation  scaling Mixed methods – Preserving user privacy – Collecting truly useful data Moving target: digital libraries as… it depends. www.digiqual.org

29. Effective, Sustainable and Practical Assessment Findings Assessment protocols that had the greatest impact in the last five years: –LibQUAL® –Usability Testing

30. Managing the Total Customer Experience Offering products or services is no longer enough Organizations must provide their customers with satisfactory experiences Competing on this dimension means orchestrating the ‘clues’ that people detect The first step is recognizing the clues an organization is sending to its customers Organizations must manage the emotional component of experiences with the same rigor they manage product and service functionality Organizations that simply tweak design elements or focus on the customer experience in isolated pockets of their business will be disappointed in the results –Berry et al, Spring 2002, MIT Sloan Management Review