User evaluation and the ‘Simplifying resource discovery and access in academic libraries’ JISC Project Annette Coates, Service Manager (Digital Library Services), Northumbria University 25 th May 2010

What my presentation will cover Northumbria’s ‘search history’ & Summon implementation progress Joint Huddersfield & Northumbria Summon JISC project User evaluation

1998 onwards

2005 onwards - Webfeat federated search

Summon Summon implementation began in December 2009 Summon instance delivered in March 2010 Testing and further set up – April / May 2010 Parallel running – June 2010 Summon service replaces WebFeat – September 2010

JISC project: Simplifying resource discovery and access in academic libraries – #summon4hn #jisclms Enhancing library management systems – Theme A – Use of new resource discovery interfaces 09/12/1209lms.aspx 6 month project from April to October 2010 Joint case study of the implementation, evaluation and launch of Summon at Huddersfield and Northumbria Project blog - Twitter hashtags above

JISC project: Simplifying resource discovery and access in academic libraries – #summon4hn #jisclms Key aims of the project: Report on objectives for implementation, challenges encountered & any issues raised during the technical implementation Gather detailed feedback from a representative range of users in order to refine and improve the system Update information skills training programmes to include the use of Summon – any generic materials to be made available to the wider community via JORUM Recommend any future enhancements that we feel would be necessary to improve the system Share lessons learned from the project outcomes with the JISC and the wider community

User evaluation – Why is it important? Is the service improving the user’s search experience? / How can we improve it further? NORA user survey in 2006 & feedback gathered since then from different sources Our joint JISC project aim: “detailed feedback from a representative range of users will be sought in order to refine and improve the system”

User evaluation – What are we doing? Online survey Focus groups One to one sessions – face to face or via phone By any means at our disposal!

User evaluation - Focus groups Importance of timing / timescale! Getting people involved – users & Library staff Incentives Capturing the session – audio recording / scribblepads / staff observers Positive impact of a focus group that goes well!

Online survey “had no idea what it was and I just had a play about it was very easy” “No - it is very intuitive” “Not at all, it's a very simple system to use.” “No help needed - very simple and clear to use” “VERY SIMPLE TO USE - GREAT RESULTS” Focus group Thought the simple search was easy to use Felt that new students might need help with advanced search Thought that we should explain the facets when showing people how to use it – particularly the fact that changing some of them automatically updates the results set Ongoing online survey & focus group feedback comparison Did you feel you need any help to use Summon?

Online survey “More references than I have been able to find before on the subject” “So far but will have to spend more time exploring!” “Couldn't access the text. As usual with the online service it just didn't work. Please keep Metalib!” “The exact journal was not available but a got a similar article to what I was searching for” Focus group Found more than in the previous NORA Much better & more subject specific than before More relevant to their subject Ongoing online survey & focus group feedback comparison Did you find what you were looking for?

Online survey “Full text articles found straight away” “It is very easy to use and much easier than Metalib.” “It was so much easier to use than metalib. I found that I used to waste so much valuable time on there.” “I like the format, it is just like using an advance search engine such as google.” “very like Google Scholar” Focus group Previously couldn’t tell whether full-text was available & now can Liked the abstract preview feature Liked the fact that it is easier to refine your search by subject Liked the ‘did you mean’ feature “The fantastic thing about this new page is we can call it the Google page of the University.” Ongoing online survey & focus group feedback comparison What did you like BEST about Summon?

Online survey “when they took it off line last week as i was lost without it” “Everything is out of order. Books followed by newspaper articles” “The amount of information! without refining your search, it can be a bit mind boggling!” “It's google like operationality. I preferred metalib” Focus group Struggled to think of anything they didn’t like when asked the question directly! But didn’t like… The fact their saved items disappeared Not being able to tell which source the results had come from The fact that the results list wasn’t numbered The fact it isn’t available yet - “CAN’T YOU START RUNNING IT NEXT MONTH?” Ongoing online survey & focus group feedback comparison What did you like LEAST about Summon?

User evaluation – What will we do with it? Triangulation of data gathered via different methods to ensure depth Use it to improve the way the system works – feed positives & negatives back to Serials Solutions Use it to improve the way we show others how to use it Use it as a marketing tool – internally & externally Comparative analysis of Huddersfield and Northumbria data Disseminate to the wider community through our joint JISC project so that everyone benefits from it

Contact details & more information Northumbria University Library website – Enhancing library management systems – Theme A – Use of new resource discovery interfaces 09/12/1209lms.aspx Summon JISC project blog -