1. Enhancing Quality of Library Instruction Programs through Delayed Assessment 6th Northumbria International Conference on Performance Measures in Libraries and Information Services, Durham, UK. August 22-25, 2005 Gabrielle Wong Reference Librarian Hong Kong University of Science and Technology

2. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 2 Presentation Outline 1.Introduction 2.Background of the Survey 3.Survey Design and the Questionnaire 4.Sampling 5.Key Findings 6.Future Assessments

3. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 3 Introduction A survey was conducted by HKUST Library to assess the enduring impact of its library instruction program Questionnaires were sent 4-8 weeks after library classes in Fall 2004 The survey provides data that demonstrates the value of the program, and helps librarians to better understand the characteristics and needs of users It was a valuable learning process for HKUST librarians; this experience serves as the basis for future assessment endeavour

4. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 4 HKUST Founded in 1991 One of the three research universities in Hong Kong Undergraduates: 5,500; Postgraduates: 3,000; Faculty: 440 4 Schools: Science, Engineering, Business, Humanities & Social Sciences

5. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 5 Library Instruction at HKUST Library Serves the whole university community: students, staff and faculty Over 200 library sessions every year, reaching 4,000 attendees One-shot sessions, 50-80 minutes Wide variety: database workshops, course-specific library sessions, topic/subject specific instruction (e.g. job hunting, microforms, web searching, research workshops for postgraduates)

6. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 6 Background of the Survey Evaluation Practice since 1996 Evaluation forms were distributed at the end of sessions Measured users' satisfaction Instructors' summary for future improvement Generally, over 80% attendees agreed that the class was worth recommending to their peers Voluntary evaluation Results were not analyzed systematically

7. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 7 Background of the Survey The Needs for a Formal Assessment Do the library sessions help the attendees in their research and studies? What do they think after a period of time, during which they are expected to apply the skills for practical purposes? How can we do better?

8. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 8 Background of the Survey The University Librarian, Dr. Samson Soong, recommended the Reference Department to launch a study to answer these questions The survey aimed to measure the enduring impact of the library program, and produce figures that he may use to demonstrate such value to the university administration It should also expose the strengths and weaknesses of the program, thus facilitating improvements

9. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 9 Survey Design: Major Considerations first attempt of such kind simple, straightforward and effective cover as many classes as possible (programmatic) librarians' workload demand on users' side assess the enduring/long term value Methodology adopted: a delayed perception survey

10. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 10 Survey Design: Delayed Assessment The survey was administered 4-8 weeks after the sessions The purpose of the delay period was to allow attendees to apply the skills on their assignments and projects Skills may be reinforced, proved insufficient or simply forgotten A mean to assess the enduring value of the classes

11. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 11 Survey Design: The Questionnaire 2 pages, 14 questions, should take less than 5 minutes to complete composed mostly of structured questions rather than open questions to encourage feedback provided options for open comments whenever possible

12. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 12 Survey Design: The Questionnaire Perception Questions: Q.2, Assessing impact: how the class helped/changed the attendees Q.3, Specific skills learned Q.4, Skill retention Q.5, Overall class rating Q.6. Overall instructor rating Q.7 Suggestions for improvement

13. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 13 Survey Design: The Questionnaire Other Questions: Library use habit  information search confidence  no. of library classes attended previously  research frequency Demographic  status  place of permanent residence  gender

14. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 14 Sampling Selection criteria:  class size > 15  if it was a course-specific session, there should have been assignment or project associated with it No intention to mimic the university population 25 sessions selected: 15 Course-Specific, 10 Open Workshops

15. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 15 About Class Types Course-Specific (CS)Open Workshop (OW) library session serves the information need of the associated credit course, e.g. FINA221 library session was "compulsory" all students enrolled in the course were to be surveyed well-defined group, we could use print and anonymous questionnaire distributed by the professors high response rate library session opens for all interested users to register by email attendees were naturally more motivated composed of a higher proportion of PGs and staff, therefore a more mature group we have no choice but to survey them with emails lost anonymity much lower response rate

16. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 16 Responses 688 questionnaires sent out Received (Print): 388 (90.7%) Received (E-mail): 78 (30.0%) Total: 466 (67.7%) 89% of respondents attended the library sessions 11% had not attended the sessions, but answered the demographic and library use habit questions

17. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 17 Findings: Impact Question 2, rate on a 4-point scale ("1" Strongly Disagree to "4" Strongly Agree) : As a result of the class, I a. learned about sources to find needed information. b. learned about search methods to find needed information. c. learned how to find needed information more quickly. d. was more confident in researching information. e. have increased my interest in using the Library.

18. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 18 Findings: Impact Mean ScoreAgree %Disagree % a. sources 3.1595.64.4 b. search method 3.1293.07.0 c. efficiency 3.0485.314.7 d. confidence 2.8574.125.9 e. interest 2.6861.238.8

19. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 19 Findings: Impact The differences of these mean scores are statistically significant (p<.05). a sources b search method c efficiency d confidence e interest CS – mean3.093.062.982.802.59 Agree %95.291.884.772.257.9 Disagree %4.88.215.327.842.1 OW – mean3.523.503.383.163.21 Agree %98.310098.186.081.0 Disagree %1.7010.914.019.0

20. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 20 Findings: Skills Learnt Question 3: Please indicate the skills in which this class has helped you (you may check more than one item): a. Choose the best databases for my information need b. Form better search strategies c. Select more appropriate keywords to search d. Export or save my results more effectively e. Find more relevant internet resources f. Use printed resources more effectively g. Others

21. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 21 Findings: Skills Learnt Average no. of choices per respondent: 2.06 Top 3 choices:  Form better search strategies  Find more relevant internet resources  Select more appropriate keywords to search

22. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 22 Findings: Skills Retention CSOWAll Yes23339272 No10718125 Valid %68.5%68.4%68.5% Question 4: Have you continued to use the skills that you learned in this class?

23. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 23 Findings: Skills Retention Written Comments – "Yes" Continued to use the skills learned to find information - general remarks 17 - specified "finding more relevant information" 5 - specified efficiency 5 - specified particular information sources 10 37 Used the skills to find better Internet resources or use search engines more effectively 11 Used the library catalog 5 Used databases 40 Applied the search skills learned 27 Used the skills learned to achieve particular goals (projects, assignments, etc.) 64 Used the subject guide learned in class 1

24. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 24 Findings: Skills Retention Written Comments – "No" No chance or need to apply the skills 22 Forgot what they learned in the classes 8 No time 7 Preferred or were satisfied with using the Internet 3 Low interest in using the Library resources 4 Considered the class not useful 2

25. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 25 Findings: Overall Ratings Question 5 and 6: 5.What is your OVERALL rating of the class: 6.What is your OVERALL rating of the instructor: Very good Good Above average Average Below average Bad Very bad

26. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 26 Findings: Overall Ratings

27. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 27 Findings: Overall Ratings

28. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 28 Findings: Overall Ratings To benchmark with the grading scale in the University's credit course evaluation: Very bad=0, Bad=16.7 Below average=33.3, Average=50, Above average=66.7 Good=83.3, Very good=100 Mean ScoresCSOWAll University average scores in the same semester Class Score66.3484.2168.8270.2-75.1 Instructor Score69.1985.3871.4972.4-78.5

29. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 29 Findings: Suggestions for Improvement Question 7: How could this class be improved? (you may check more than one item) a. I already find the class satisfactory b. Shorter time c. Longer time d. Cover fewer topics, please specify e. Cover more topics, please specify f. Cover some topics in more detail, please specify g. Cover some topics in less detail, please specify h. Instructor should speak more slowly i. Instructor should speak more clearly j. Offer students more hands-on practice k. Have a smaller class l. Others, please specify

30. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 30 Findings: Suggestions for Improvement Average no. of choices per respondent: 1.48 Top Three Choices: CSOW Shorter time (31%) Satisfactory (55%) Satisfactory (27%) Shorter time (20%) Hands-on (22%) Hands-on (18%)

31. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 31 Findings: Suggestions for Improvement Written Comments: Fewer topics: 2 commented on the flood of information in the class More topics: 7 comments, 4 called for more coverage on databases More details: 14 comments on various issues: 4 about search skills, 2 called for more examples, 2 for databases Others: 15 various comments, 6 asked for more and better handouts

32. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 32 Other Observations Class size:  Bigger classes gave lower class ratings, instructor ratings and lower skill retention Information search confidence:  Respondents with higher confidence in information search rated higher in these three key indicators

33. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 33 What the Findings show Us It demonstrates positive value of the Library's bibliographic program by yielding these figures:  After a period of time, over 85% of users remain affirmative that the class was useful  Almost 70% of users retain the skills learnt  The overall ratings are comparable to credit courses in the University It tells us what users want:  Smaller class  More hands-on  Better class handouts

34. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 34 What we Learnt from the Survey There are significant differences between the Open Workshop Group and the Course-Specific Group Why users did not use the skills:  did not see the need/did not have the chance  forgot what they learnt Users with higher confidence in information search are more receptive and appreciative of library instruction Class size does matter

35. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 35 Where do We Go from Here? Nurture a culture of assessment at HKUST Library Next assessment in 2006 Administer a comparable immediate evaluation attempting to quantify the "delay" effect Two separate schemes for CS and OW? Focus on the outcomes of one particular class instead of a programmatic assessment? Consider testing attendees' library skills instead of measuring perception?

36. Thank you!