Evaluation of a Chat Reference Service from the User’s Perspective Lili Luo Jeffrey Pomerantz School of Information & Library Science UNC Chapel Hill <luo, pomerantz>@unc.edu Tapping the vast reservoir of human knowledge --Louis Round Wilson, founder, 1931

Brief Outline Research Context: NCknows Research Questions & Theoretic Framework Study Design & Methodology Results Discussion & Future Work Evaluation = 18-month NCknows pilot phase LSTA = Library Services and Technology Act 2001-2002: SLNC funded planning phase to focus gather info on level of interest & knowledge of: virtual reference services among the NC librarians in North Carolina’s libraries developing a common and increased knowledge of virtual reference services among the librarians on exploring strategies for developing virtual reference services 15 volunteer librarians formed a Virtual Reference Advisory Committee to plan NCknows.

NCknows Collaborative state-wide chat service in North Carolina (www.ncknows.org) 18-month pilot phase launched in February 2004, with 19 libraries participating. Librarians in all 19 libraries staffed the service during the pilot. Participants: public, research university, community college, government libraries; urban and rural, large and small. Pilot phase is completed. Service had its final launch in July 2005. Evaluation = 18-month NCknows pilot phase LSTA = Library Services and Technology Act 2001-2002: SLNC funded planning phase to focus gather info on level of interest & knowledge of: virtual reference services among the NC librarians in North Carolina’s libraries developing a common and increased knowledge of virtual reference services among the librarians on exploring strategies for developing virtual reference services 15 volunteer librarians formed a Virtual Reference Advisory Committee to plan NCknows.

Evaluation 4 Components to the Evaluation Baseline: Statistics from the 24/7 reference chat application Quality control: Peer reviews of chat transcripts Administration: Interviews with librarians Use & user satisfaction: Exit surveys & follow-up interviews Evaluation = 18-month NCknows pilot phase LSTA = Library Services and Technology Act 2001-2002: SLNC funded planning phase to focus gather info on level of interest & knowledge of: virtual reference services among the NC librarians in North Carolina’s libraries developing a common and increased knowledge of virtual reference services among the librarians on exploring strategies for developing virtual reference services 15 volunteer librarians formed a Virtual Reference Advisory Committee to plan NCknows.

Research Questions Motivation: What motivates users to use a chat reference service? Use: How do users use the information provided by the chat reference service? Satisfaction: What is the users’ level of satisfaction with chat reference service? Evaluation = 18-month NCknows pilot phase LSTA = Library Services and Technology Act 2001-2002: SLNC funded planning phase to focus gather info on level of interest & knowledge of: virtual reference services among the NC librarians in North Carolina’s libraries developing a common and increased knowledge of virtual reference services among the librarians on exploring strategies for developing virtual reference services 15 volunteer librarians formed a Virtual Reference Advisory Committee to plan NCknows.

Dervin’s Sense-Making Theory situation gap motivation to seek assistance from chat reference service bridging fulfillment use of information provided by chat reference service Evaluation = 18-month NCknows pilot phase LSTA = Library Services and Technology Act 2001-2002: SLNC funded planning phase to focus gather info on level of interest & knowledge of: virtual reference services among the NC librarians in North Carolina’s libraries developing a common and increased knowledge of virtual reference services among the librarians on exploring strategies for developing virtual reference services 15 volunteer librarians formed a Virtual Reference Advisory Committee to plan NCknows.

Study Design Exit Survey Email & Telephone Interviews an exit survey popped up for the user at the conclusion of a chat session Email & Telephone Interviews semi-structured interviews were conducted two to three weeks after users’ chat sessions

Results Prior to Use of the Service The Point of Service Use of Other Reference Services Discovery of NCknows Motivation for Using the Service Motivation for Asking the Question The Point of Service Satisfaction Levels Role of User Use of the Information Provided by the Service

How did you find out about the NCknows chat service? (Follow-up interview − 68 responses) Four routes by which users learned about the service emerged: 1. The local library, or library-produced publicity materials (69%). 2. Online searching (12%). 3. School system (12%). 4. Friends and colleagues (7%).

Reference Services Respondents Have you ever used any of your library’s other reference services? Which? (Follow-up interview − 67 responses) Reference Services Respondents desk reference service 48% e-mail reference service 19% telephone reference service 33% never used any other reference services did not answer the question directly 7% librarians or library staff

Why did you use the chat service to get this question answered rather than any of these other services? (Follow-up interview − 68 responses) Convenience (47%). Other means of seeking information were not helpful (15%). Curiosity (13%). Serendipity (12%). Recommended by others (7%). Personal characteristics/habits (7%). Other reference services were not available (1.5%).

(Follow-up interview − 72 responses) Can you tell me about what motivated the question that you asked the NCknows chat service? (Follow-up interview − 72 responses) To satisfy an inquiry about the library itself, 3% To help others look for information, 3% Others, 3% To locate a known item, 8% To answer a question arising from the user’s personal life, 32% To answer a work/school related question, 51%

Business-related: “I’m a small business owner and I need the information to negotiate with other type of business people to do a joint venture.” School-related: “I was doing a research paper on homeschooling. Want some hard stats numbers from the website about how homeschool students do academically.” Genealogy: “Wanted to find passenger ships of 1800 that had been lost at sea, trying to find one that may have been carrying wife and children of my great grandfather.” Vocational or academic change: “doing research on school that meets the criteria that I’m looking for. I want to go back to school.” Known-item search: “I was looking for an audio book and couldn’t understand the new system.” On behalf of others: “It was my son’s schoolwork questions in Geography class. He had found the answers to many of his geography questions but we had searched and searched using Google and also used Ask Jeeves but were not successful in finding the answers to a couple of questions.”

How satisfied were you with the …? (Follow-up interview − 72 responses) Completeness of the answer Speed that the librarian answered your question Helpfulness of the librarian Ease of use of the software Very satisfied 68.5% 67.7% 81.2% 82.7% Satisfied 23.55% 24.9% 13.1% 14.2% Dissatisfied 5.4% 5.6% 3.4% 1.8% Very dissatisfied 2.6% 2.1% 2.4 1.3% Total: n = 387 381 382

Would you recommend this chat service to a friend? (Exit survey − 385 responses) Recommend Percentage Very likely 87.5 Maybe 10.1 Unlikely 1.6 Never 0.8 Total 100%

You asked this question in your role as a(n): (Exit survey − 379 responses) Role Percentage Other 29.7 Student: Undergraduate 23.2 Student: Graduate 13.5 Student: K-12 10.3 Parent 7.8 Librarian 7.7 Medical Professional 6.3 Teacher: K-12 3.6 Higher Ed Faculty 3.5 Adult Educator 3.1 Administrator 2.7 Teacher: Pre-school 0.8 Policymaker 0.0 Politician

Have you had a chance to use the information that NCknows provided to you? (Follow-up interview − 67 responses)

Use: “It was utilized as background information for my lecture.” “I used the information in the negotiation with other business owners. Through the knowledge gained, we have (hopefully) brought another business to Fayetteville. At least we are talking now.” Partial Use: “NCknows told me the person to call at my local library who said they can't provide that information. I was going to use it in my graduate school thesis. Alas…” “I am still working on this matter but was provided good leads from your service.” No use: “The librarian did give me a website for on loan tapes, but they took to long to receive.” “I plan to call the number I got from the service in a couple of time.”

Discussion Target User Groups Campaigns to market chat reference services should be targeted to user groups that are likely to have a need for the service: for example, in grade schools, colleges, and universities; businesses and other corporate settings; and social organizations. Expand Service Hours The availability of the service at all hours of the day and night is one important factor in the convenience of the service. Identify Appropriate Reference Medium for Users Encourage users to use a reference service in the most appropriate medium. Research question: How to identify the most appropriate medium for a user / for a question?

Limitations Future Work Non-response bias Self-selection bias Methodological enhancements: alternative subject recruiting methods Further the exploration of service value: what users gain from using the information provided by the service

<luo, pomerantz>@unc.edu Questions & Comments? <luo, pomerantz>@unc.edu