E-Valuating Virtual Viewpoints: User, Non-User, and Librarians Perspectives on Live Chat-Based Reference Marie L. Radford, Ph.D. Associate Professor, Rutgers, The State University of New Jersey Lynn Silipigni Connaway, Ph.D. Senior Research Scientist, OCLC Panel: E-Valuating E- Reference: Transforming Digital Reference through Research and Evaluating ASIST Annual Meeting October Columbus, OH

$1,103,572 project funded by Institute of Museum and Library Services Rutgers, The State University of New Jersey OCLC, Online Computer Library Center, Inc. Four phases: Focus group interviews Analysis of 850 QuestionPoint live chat transcripts Online surveys 176 VRS librarians 184 VRS non-users 137 VRS users Telephone interviews 283 Total Seeking Synchronicity: Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives

Critical Incident Technique  Flanagan, 1954  Qualitative technique  Focuses on most memorable event/experience  Allows categories or themes to emerge rather than be imposed

Online Survey CI Questions Librarians & Users Think about one experience in which you felt a chat reference encounter achieved (or did not achieve) a positive result Non-users Think about one experience in which you felt you achieved (did not achieve) a positive result after seeking library reference services in any format

Interpersonal Communication Analysis: Results Relational Facilitators Interpersonal aspects of chat conversation that have a positive impact on librarian-client interaction & that enhance communication. Relational Barriers Interpersonal aspects of chat conversation that have a negative impact on librarian-client interaction & that impede communication.

Relational Theory & Approach to Interpersonal Communication Every message has dual dimensions – both content & relational (Watzlawick, Beavin, & Jackson, 1967)

Librarian Demographics Gender Female132 Male 42 Age Ethnicity Caucasian152 African American 5 Other 5 Asian or Pacific Islander 2 Hispanic/Latino 1 Native American 0

Librarian Demographics Location Urban94 Suburban52 Rural26 Library Type Academic104 Public54 Special 7 Consortium 2 School 0

Librarians: Positive Result (CI N=142) Number %  Both Relational & 85 60% Content  Primarily Content 54 38%  Primarily Relational 3 2%

Librarians: Positive Result (CI N=142) Relational Themes* Number %  Attitude69 49%  Relationship quality33 23%  Familiarity 3 2% *The percentages do not total to 100% because each CI can be coded into more than one theme

Librarians: Positive Result (CI N=142) Content Themes * Number %  Providing information 12085%  Providing instruction 49 35%  Demonstrating knowledge 1410%  Convenience/multi- 10 7% tasking/ time or money saving *The percentages do not total to 100% because each CI can be coded into more than one theme

Librarians: Negative Result (CI N=124) Number %  Primarily Relational 53 43%  Primarily Content 40 32%  Both Relational & 31 25% Content

Librarians: Negative Result (CI N=124) Relational Themes* Number %  Attitude67 54%  Relationship quality28 23%  Impact of technology 7 6%  Approachability 1 1% *The percentages do not total to 100% because each CI can be coded into more than one theme

Librarians: Negative Result (CI N=124) Content Themes* Number %  Lack of information 64 52%  Lack of knowledge 15 12%  Task unreasonable 1 1% *The percentages do not total to 100% because each CI can be coded into more than one theme

User Demographics Age Gender Female85 Male52

User Demographics Location Suburban85 Urban38 Rural 13 Ethnicity Caucasian107 Asian or Pacific Islander 11 African American 11 Other 5 Hispanic/Latino 3 Native American 0

Users: Positive Result (CI N=129) Number %  Primarily Content 79 61%  Both Relational & 33 26% Content  Primarily Relational17 13%

Users: Positive Result (CI N=129) Content Themes * Number %  Providing information 91 71%  Convenience/multi % tasking/time saving/ money saving  Providing instruction 14 11%  Demonstrating knowledge 7 5% *The percentages do not total to 100% because each CI can be coded into more than one theme

Users: Positive Result (CI N=129) Relational Themes* Number %  Attitude 36 28%  Relationship quality 21 16%  Impact of technology 1 1% *The percentages do not total to 100% because each CI can be coded into more than one theme

Users: Negative Result (CI N=68) Number %  Primarily Content 46 68%  Primarily Relational 15 22%  Both Relational & 7 10% Content

Users: Negative Result (CI N=68) Content Themes* Number %  Lack of information 48 71%  Lack of knowledge 8 12% *The percentages do not total to 100% because each CI can be coded into more than one theme

Users: Negative Result (CI N=68) Relational Themes* Number %  Relationship quality17 25%  Attitude13 19%  Approachability 1 1%  Impact of Technology 1 1% *The percentages do not total to 100% because each CI can be coded into more than one theme

Non-user Demographics Age Gender Female125 Male 59

Non-user Demographics Location Suburban107 Urban 67 Rural 10 Ethnicity Caucasian131 Asian or Pacific Islander 30 African American 10 Other 7 Hispanic/Latino 4 No Response 0

Non-users: Positive Result (CI N=154) Number %  Primarily Content79 51%  Both Relational & 48 31% Content  Primarily Relational27 18%

Non-users: Positive Result (CI N=154) Content Themes * Number %  Providing information 75 49%  Providing instruction 35 23%  Demonstrating knowledge 21 14%  Convenience/multi % tasking/time saving/ money saving *The percentages do not total to 100% because each CI can be coded into more than one theme

Non-users: Positive Result (CI N=154) Relational Themes* Number %  Attitude 51 33%  Impact of FtF assisting 32 21% relationship development  Relationship quality 25 16%  Impact of phone/ 5 3% assisting information seeking process  Approachability 4 3%  Familiarity 1 1% *The percentages do not total to 100% because each CI can be coded into more than one theme

Non-users: Negative Result (CI N=100) Number %  Primarily Content 52 52%  Primarily Relational 33 33%  Both Relational & 15 15% Content

Non-users: Negative Result (CI N=100) Content Themes* Number %  Information 60 60%  Lack of knowledge 24 24%  Instruction 9 9%  Task unreasonable 4 4% *The percentages do not total to 100% because each CI can be coded into more than one theme

Non-users: Negative Result (CI N=100) Relational Themes* Number %  Attitude47 47%  Relationship quality24 24%  Approachability 3 3%  Impact of technology 2 2% *The percentages do not total to 100% because each CI can be coded into more than one theme

Implications: Librarians  Value  Delivery of accurate answers/ information  Polite, interested users  Find rude or impatient users disruptive to chat success

Implications: Users & Non-Users  Value  Accuracy of answers/information  Delivery of specific content  Knowledge of sources & systems  Positive attitude  Good communication skills  Younger VRS users  Impatient & want info delivered quickly - no fuss  Not as concerned as librarians w/ instruction

Recommendations  Provide  Specific info  Variety of formats  Friendly & courteous service  Marketing to non-users  User education needed for more realistic expectations  Do not force instruction unless wanted

Future Directions Online survey results informed 283 telephone interviews  Collected more critical incidents  Analysis in progress

Future Directions  Write, write, write!

Special Thanks Rutgers University & OCLC Grant Project Team Project Managers: Jocelyn DeAngelis Williams Timothy J. Dickey Research Assistants: Patrick A. Confer David Dragos Jannica Heinstrom Vickie Kozo Mary Anne Reilly Lisa Rose-Wiles Susanna Sabolsci-Boros Andrea Simzak Julie Strange Janet Torsney

End Notes This is an updated version of a presentation given at ALISE 2008 This is one of the outcomes from the project Seeking Synchronicity: Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives Funded by IMLS, Rutgers University, & OCLC Online Computer Library Center, Inc. Slides available at project web site: