1. E-Valuating Virtual Viewpoints: User, Non-User, and Librarians Perspectives on Live Chat-Based Reference Marie L. Radford, Ph.D. Associate Professor, Rutgers, The State University of New Jersey Lynn Silipigni Connaway, Ph.D. Senior Research Scientist, OCLC Panel: E-Valuating E- Reference: Transforming Digital Reference through Research and Evaluating ASIST Annual Meeting October 24-29 2008 Columbus, OH

2. $1,103,572 project funded by Institute of Museum and Library Services Rutgers, The State University of New Jersey OCLC, Online Computer Library Center, Inc. Four phases: Focus group interviews Analysis of 850 QuestionPoint live chat transcripts Online surveys 176 VRS librarians 184 VRS non-users 137 VRS users Telephone interviews 283 Total Seeking Synchronicity: Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives

3. Critical Incident Technique  Flanagan, 1954  Qualitative technique  Focuses on most memorable event/experience  Allows categories or themes to emerge rather than be imposed

4. Online Survey CI Questions Librarians & Users Think about one experience in which you felt a chat reference encounter achieved (or did not achieve) a positive result Non-users Think about one experience in which you felt you achieved (did not achieve) a positive result after seeking library reference services in any format

5. Interpersonal Communication Analysis: Results Relational Facilitators Interpersonal aspects of chat conversation that have a positive impact on librarian-client interaction & that enhance communication. Relational Barriers Interpersonal aspects of chat conversation that have a negative impact on librarian-client interaction & that impede communication.

6. Relational Theory & Approach to Interpersonal Communication Every message has dual dimensions – both content & relational (Watzlawick, Beavin, & Jackson, 1967)

7. Librarian Demographics Gender Female132 Male 42 Age 21-3034 31-4039 41-5050 51-6041 61+10 Ethnicity Caucasian152 African American 5 Other 5 Asian or Pacific Islander 2 Hispanic/Latino 1 Native American 0

8. Librarian Demographics Location Urban94 Suburban52 Rural26 Library Type Academic104 Public54 Special 7 Consortium 2 School 0

9. Librarians: Positive Result (CI N=142) Number %  Both Relational & 85 60% Content  Primarily Content 54 38%  Primarily Relational 3 2%

10. Librarians: Positive Result (CI N=142) Relational Themes* Number %  Attitude69 49%  Relationship quality33 23%  Familiarity 3 2% *The percentages do not total to 100% because each CI can be coded into more than one theme

11. Librarians: Positive Result (CI N=142) Content Themes * Number %  Providing information 12085%  Providing instruction 49 35%  Demonstrating knowledge 1410%  Convenience/multi- 10 7% tasking/ time or money saving *The percentages do not total to 100% because each CI can be coded into more than one theme

12. Librarians: Negative Result (CI N=124) Number %  Primarily Relational 53 43%  Primarily Content 40 32%  Both Relational & 31 25% Content

13. Librarians: Negative Result (CI N=124) Relational Themes* Number %  Attitude67 54%  Relationship quality28 23%  Impact of technology 7 6%  Approachability 1 1% *The percentages do not total to 100% because each CI can be coded into more than one theme

14. Librarians: Negative Result (CI N=124) Content Themes* Number %  Lack of information 64 52%  Lack of knowledge 15 12%  Task unreasonable 1 1% *The percentages do not total to 100% because each CI can be coded into more than one theme

15. User Demographics Age 12-14 7 15-18 19 19-2823 29-35 18 36-4533 46-55 21 56-65 12 65+ 4 Gender Female85 Male52

16. User Demographics Location Suburban85 Urban38 Rural 13 Ethnicity Caucasian107 Asian or Pacific Islander 11 African American 11 Other 5 Hispanic/Latino 3 Native American 0

17. Users: Positive Result (CI N=129) Number %  Primarily Content 79 61%  Both Relational & 33 26% Content  Primarily Relational17 13%

18. Users: Positive Result (CI N=129) Content Themes * Number %  Providing information 91 71%  Convenience/multi- 36 28% tasking/time saving/ money saving  Providing instruction 14 11%  Demonstrating knowledge 7 5% *The percentages do not total to 100% because each CI can be coded into more than one theme

19. Users: Positive Result (CI N=129) Relational Themes* Number %  Attitude 36 28%  Relationship quality 21 16%  Impact of technology 1 1% *The percentages do not total to 100% because each CI can be coded into more than one theme

20. Users: Negative Result (CI N=68) Number %  Primarily Content 46 68%  Primarily Relational 15 22%  Both Relational & 7 10% Content

21. Users: Negative Result (CI N=68) Content Themes* Number %  Lack of information 48 71%  Lack of knowledge 8 12% *The percentages do not total to 100% because each CI can be coded into more than one theme

22. Users: Negative Result (CI N=68) Relational Themes* Number %  Relationship quality17 25%  Attitude13 19%  Approachability 1 1%  Impact of Technology 1 1% *The percentages do not total to 100% because each CI can be coded into more than one theme

23. Non-user Demographics Age 12-1418 15-1842 19-2862 29-3511 36-4518 46-5519 56-6510 65+ 4 Gender Female125 Male 59

24. Non-user Demographics Location Suburban107 Urban 67 Rural 10 Ethnicity Caucasian131 Asian or Pacific Islander 30 African American 10 Other 7 Hispanic/Latino 4 No Response 0

25. Non-users: Positive Result (CI N=154) Number %  Primarily Content79 51%  Both Relational & 48 31% Content  Primarily Relational27 18%

26. Non-users: Positive Result (CI N=154) Content Themes * Number %  Providing information 75 49%  Providing instruction 35 23%  Demonstrating knowledge 21 14%  Convenience/multi- 18 12% tasking/time saving/ money saving *The percentages do not total to 100% because each CI can be coded into more than one theme

27. Non-users: Positive Result (CI N=154) Relational Themes* Number %  Attitude 51 33%  Impact of FtF assisting 32 21% relationship development  Relationship quality 25 16%  Impact of phone/Email 5 3% assisting information seeking process  Approachability 4 3%  Familiarity 1 1% *The percentages do not total to 100% because each CI can be coded into more than one theme

28. Non-users: Negative Result (CI N=100) Number %  Primarily Content 52 52%  Primarily Relational 33 33%  Both Relational & 15 15% Content

29. Non-users: Negative Result (CI N=100) Content Themes* Number %  Information 60 60%  Lack of knowledge 24 24%  Instruction 9 9%  Task unreasonable 4 4% *The percentages do not total to 100% because each CI can be coded into more than one theme

30. Non-users: Negative Result (CI N=100) Relational Themes* Number %  Attitude47 47%  Relationship quality24 24%  Approachability 3 3%  Impact of technology 2 2% *The percentages do not total to 100% because each CI can be coded into more than one theme

31. Implications: Librarians  Value  Delivery of accurate answers/ information  Polite, interested users  Find rude or impatient users disruptive to chat success

32. Implications: Users & Non-Users  Value  Accuracy of answers/information  Delivery of specific content  Knowledge of sources & systems  Positive attitude  Good communication skills  Younger VRS users  Impatient & want info delivered quickly - no fuss  Not as concerned as librarians w/ instruction

33. Recommendations  Provide  Specific info  Variety of formats  Friendly & courteous service  Marketing to non-users  User education needed for more realistic expectations  Do not force instruction unless wanted

34. Future Directions Online survey results informed 283 telephone interviews  Collected more critical incidents  Analysis in progress

35. Future Directions  Write, write, write!

36. Special Thanks Rutgers University & OCLC Grant Project Team Project Managers: Jocelyn DeAngelis Williams Timothy J. Dickey Research Assistants: Patrick A. Confer David Dragos Jannica Heinstrom Vickie Kozo Mary Anne Reilly Lisa Rose-Wiles Susanna Sabolsci-Boros Andrea Simzak Julie Strange Janet Torsney

37. End Notes This is an updated version of a presentation given at ALISE 2008 This is one of the outcomes from the project Seeking Synchronicity: Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives Funded by IMLS, Rutgers University, & OCLC Online Computer Library Center, Inc. Slides available at project web site: http://www.oclc.org/research/projects/synchronicity/ http://www.oclc.org/research/projects/synchronicity/