Online Virtual Chat Library Reference Service: A Quantitative and Qualitative Analysis Dr. Dave Harmeyer Associate Professor Chair, Marshburn Memorial Library Azusa Pacific University

Outline 1.Purpose of the Study 2.Research Questions 3.Methodology 4.Variables (I.V., D.V.) 5.Significant Findings 6.Conclusions 7.Questions & Answers

Purpose of the Study  Virtual chat reference augments face-to-face reference interview  Library reference literature lacks research- based findings to back up recommended practices  This study fills the void with a theoretical conceptual model based on an empirical study of chat reference transactions

Research Questions 1. What measurable indicators are found for virtual chat reference transactions, looking exclusively at data created from the chat reference transcripts? 2. Do published reference interview guidelines from RUSA, a set of other strategies and the nature of the query contribute to an accurate answer? 3. What conceptual model of best practices can be suggested by an analysis of the data?

Methodology  Two-and-a-half years of archived academic library chat transcripts using Krippendorff’s (2004) content analysis  333 random transcripts from 2,500  Analyzing 16 independent variables and their relationship with one dependent variable of an accurate reference answer  Pearson correlations and ANOVA variance tests  120 virtual librarians at 43 American institutions  320 remote patrons accessing the service through one Southern California undergraduate/masters university

Methodology (cont.)

Variables  Research Question 1 answered:  16 independent variables  1 dependent variable - question accuracy  Influence on question accuracy  Observed in content analysis of chat transcripts  Derived from RUSA guidelines  Derived from literature review

Quantitative IVs 1. Librarian’s initial contact time (hold time, in seconds) 2. Total time of transaction (service time, in seconds) 3. Longest time gap by librarian (in seconds) 4. Number of URLs co-browsed with the patron 5. Keystrokes by librarian 6. Keystrokes by patron 7. Keystrokes by both

Qualitative IVs 1. The question’s difficulty (seven-point scale) 2. Response to a patron’s “are you there” statements (scored as present, not present, not applicable or ambiguous when coders disagreed) 3. Librarian’s friendliness 4. Lack of jargon 5. Use of open-ended questions 6. Use of closed and/or clarifying questions 7. Librarian maintains objectivity 8. Asking if the question was answered completely 9. The type of question (seven categories: ready reference, research question, library technology, request for materials, bibliographic verification, other and ambiguous for disagreements among coders)

Dependent Variable Coders Qualitative JudgmentsService Quality Coders Qualitative JudgmentsService Quality 8 Librarian gave (or referred) patron to a single 8 Librarian gave (or referred) patron to a single source with an accurate answer Excellent source with an accurate answer Excellent 7 Librarian gave (or referred) patron to more than 7 Librarian gave (or referred) patron to more than one source, one of which provided an accurate one source, one of which provided an accurate answer Very good answer Very good 6 Librarian gave (or referred) patron to a single 6 Librarian gave (or referred) patron to a single source which does not lead directly to an source which does not lead directly to an accurate answer but did serve as a preliminary accurate answer but did serve as a preliminary source Good source Good 5 Librarian gave (or referred) patron to more 5 Librarian gave (or referred) patron to more than one source, none of which leads directly than one source, none of which leads directly to an accurate answer but one which served as to an accurate answer but one which served as a preliminary source Satisfactory a preliminary source Satisfactory

Dependent Variable (cont.) Coders Qualitative JudgmentsService Quality Coders Qualitative JudgmentsService Quality 4No direct accurate answer given, 4No direct accurate answer given, referred to another person or institution Fair / poor 3No accurate answer (or referral) given 3No accurate answer (or referral) given (e.g., “I don’t know”) Failure 2Librarian gave (or referred) patron to a 2Librarian gave (or referred) patron to a single source which did not answer the question Unsatisfactory 1Librarian gave (or referred) patron to 1Librarian gave (or referred) patron to more than one source, none of which answered the question Most unsatisfactory (Richardson and Reyes, 1995)

Significant Findings Summary  Research Question 2 answered: yes  30 significant relationships (p <.05)  From 9 of 16 variables  5 found in RUSA guidelines  4 found in other strategies or nature of online chat

Significant Findings Answer Accuracy Answer Accuracy as Judged by Coders (N=331) Criteria Point Frequency %Cum. % Accurate Answer (single source) Excellent (1/4) Accurate Answer (mult. sources) Very good (1/2) Preliminary Source (single source) Good (2/3) Preliminary Source (mult. sources) Satisfactory (3/4) No Accurate Answer, referred Fair / poor “I don’t know,” no referral Failure Not Accurate (single source) Unsatisfactory Not Accurate (multiple sources) Most unsatisfactory

Significant Findings Best Practices Research Question 3 answered: yes A Conceptual Model for Reference Chat Accuracy minor plus est (less is more) 1. Keep time gaps between sending responses to patrons to no more than one-and-a-half minutes 2. Maintain a total chat transaction time of eight minutes or less 3. Keep total keystrokes per transaction to within six and-a-half lines of text (or 480 characters). 4. Expect to type twice as many characters as the patron

Significant Findings Best Practices (cont.) 5. Be careful about beginning the question negotiation segment of the reference interview with an open question unless the nature of the patron’s question explicitly calls for one. 6. Ask closed or clarifying questions when appropriate 7. At the end of the reference transaction, ask “Does this completely answer your question?” 8. Even moderately difficult questions decrease answer accuracy and not just the medium to high difficult questions

Significant Findings 1. Gaps  Keep time gaps between sending responses to patrons to not much more than one-and- a-half minutes  Reinforces RUSA’s interest guideline (2.6), time away from the patron short, maintain “word contact” (RUSA, June 2004)  Anything nearing two minutes or higher is likely to decrease answer accuracy

Significant Findings 1. Gaps Longest Librarian Gap Quartiles (min.) Acc. Mean Sig. of Diff. (p)_____ 1st nd 1.87 – (1st & 2nd) 3rd 2.85 – (1st & 3rd, no sig.) 4th (1st & 4th) Diff=.71 Diff=.65 Diff=.71

Significant Findings: 2. Service Time  Maintain a total chat transaction time of eight minutes or less  Average = 16.0 minutes (n = 331)  7 minutes more than Richardson’s (2002) 8.9 minutes (n = 20,000)  However, similar to six f2f studies with mean service time ranging from 10 to 20

Significant Findings 2. Service Time Service Time of Transactions Quartiles (min.) Accuracy Mean Sig. of Diff. (p) 1st 0 – nd 8.32 – (1st & 2nd) 3rd 13.1 – (1st & 3rd) 4th (1st & 4th) Diff=.78 Diff=.70 Diff=.80

Significant Findings: 3. Keystrokes  Keep total keystrokes per transaction to within six and-a-half lines of text (or 480 characters)  Application to virtual software vendors (add a timer)  Anything over 15 lines of text will decrease accuracy

Significant Findings 3. Keystrokes Keystrokes Keystroke Quartiles Accuracy Mean Sig. of Diff. (p) Librarian 1st 0 – 480 (6.5 lines)*6.58 4th 1128 (15 lines) (1st & 4th) Patron 1st 0 – 188 (2.5 lines)6.65 4th 545 (7.5) (1st & 4th) Both Librarian & Patron 1st 0 – 690 (9 lines)6.63 4th 1668 (22.5 lines) (1st & 4th) *measured at 74 keystrokes per line of text

Significant Findings 4. Twice the Typing  Expect to type twice as many characters as the patron  Appeared across all four quartile segments between librarian and patron.

Significant Findings 5. Open-ended Questions   Be careful about beginning the question negotiation segment of the reference interview with an open question unless the nature of the patron’s question explicitly calls for one Frequency of Open-ended Questions Category Frequency Percent Present Absent (but should) Not Applicable Ambiguous

Significant Findings 5. Open-ended Questions Open-ended Questions Category Accuracy Mean Sig. of Diff. (p) Not Applicable6.72 Present (3 & 1)

Significant Findings: 6. Closed-ended Questions   Ask closed or clarifying questions when appropriate Frequency of Closed-ended and/or Clarifying Questions Category Frequency Percent Present Absent Not Applicable Ambiguous4413.2

Significant Findings 6. Closed-ended Questions Closed and/or Clarifying Questions Category Accuracy Mean Sig. Of Diff. (p) 3. Not Applicable Absent (3 & 2, not sig.) Ambiguous filtered 3. Not Applicable Absent (3 & 2)

Significant Findings 7. Follow-up Question   At the end of the reference transaction, ask “Does this completely answer your question?” Frequency of the Librarian Asking If the Question Had Been Answered Completely Category FrequencyPercent Present Absent Not Applicable Ambiguous

Significant Findings 8. Question Difficulty Question Difficulty Criteria Point Frequency %Cum. % Low / /4 Medium % High

Significant Findings 8. Question Difficulty Question Difficulty and Accuracy (reporting only significance) Criteria Points Accuracy Mean Sig. of Diff. (p) Low (1.0 & 2.0) (1.0 & 2.5) Medium (1 & 3.5) (1.0 & 4.0) (1.0 & 4.5) (1.0 & 5.0) (1.0 & 5.5) High (1.0 & 6.0) Low (1.5 & 2.5) Medium (1.5 & 3.5) (1.5 & 4.0) (1.5 & 4.5) (1.5 & 5.5) High (1.5 & 6.0)

5. Conclusions  Virtual reference lacks a statistically sound conceptual model to guide the library profession toward improving the reference interview through empirical studies which informs best practices in professional training and assessment.  This study addresses that knowledge void by its discovery of several statistical relationships between nine behavioral factors and an acurate answer in the reference interview.  It is hoped that the suggested eight-point rubric and other results of this project can be a catalyst for practical application toward improving the practice of the global community of professionals and stakeholders in the field of library and information studies.

6. Questions & Answers