1. Department of Information & Communications Online Information Literacy Audits: a longitudinal study Bob Glass & Jill Griffiths

2. Department of Information & Communications The activities undertaken address the following questions: 1. Is there value in administering online information literacy audits? 2. What do the results of these audits tell us? 3. Does intervention make a difference in the results? 4. Is there a need in for a UK designed, generic, online Information Literacy Audit? A three year study of two undergraduate cohorts The testing is based around an online web based Information Literacy Test from Steven Wise & Lynn Cameron (2004). The test is hosted by The Institute for Computer Based Assessment, James Madison University (2006)

3. Department of Information & Communications Who are the students? Information & Communications 65 students on 5 degree routes all studying the same six first year units Economics 19 students from1 degree route all studying the same six first year units A very mixed group socio economically, ethnically and chronologically Some students were working librarians Background

4. Department of Information & Communications What is the test? An Online, web based, multiple choice, using radio buttons for answer selection Asks 65 questions (60 live plus 5 trial questions) Uses the “Adaptex” Assessment System (Wise, S. L. & Yang, S. 2003) to manage the responses. Measures four of the five ACRL standards Testing takes around an hour to complete, no time limit set Students receive a score on completion of the test Tutors receive detailed performance data in a range of formats Devised by library and psychology practitioners

5. What is the test? ACRL Objectives – designed around ACRL standards for information literacy: ACRL1 – Determines the nature and extent of the information needed ACRL2 – Accesses needed information effectively and efficiently ACRL3 – Evaluates information and its sources critically and incorporates selected information into his or her knowledge base and value system ACRL4 – Assessed differently (essay) ACRL5 – Understands many of the economic, legal and social issues surrounding the use of information and access and uses Department of Information & Communications

6. What is the test? Format:

7. Department of Information & Communications Is there value in administering online information literacy audits? can provide a cohort baseline for tutors/ practitioners can provide interaction, information & feedback for the students can highlight areas for development with students can highlight areas for development in the curriculum can highlight areas for development for library practitioners

8. Department of Information & Communications What do the results tell us? 1.the majority of students scored well above the score required for “Proficient” 2. the mean score for ICO students was in the “Advanced” classification the mean score for Economics was almost identical 3. Only one in every 18 of the students failed to reach the Proficient grade (4 ICO students and 1 Economics) 4. just over a quarter of students were classed as Proficient two thirds were classed as Advanced

9. Preliminary results Department of Information & Communications

10. Preliminary results Department of Information & Communications

11. Preliminary results Department of Information & Communications

12. 1: can the student define and articulate the nature and extent of the information needed. 2: can the student access needed information effectively and efficiently. 3: can the student evaluate information and its sources critically and incorporate selected information into his or her knowledge base and value system. 5: can the student understand many of the ethical, legal and socio-economic issues surrounding information and information technology. What is the test? Standards assessed : Standard 4 (written work) cannot be assessed by this type of test

13. Preliminary results Department of Information & Communications

14. Preliminary results Department of Information & Communications

15. Preliminary results Department of Information & Communications

16. Preliminary results Department of Information & Communications

17. Preliminary results Department of Information & Communications

18. Preliminary results Department of Information & Communications

19. What do the results tell us? 1.the majority of students were able to answer questions concerning defining and articulating the nature and extent of the information needed correctly 2. around half were not able to answer questions concerning accessing needed information effectively and efficiently correctly 3. two thirds were able to answer questions concerning evaluating information and its sources critically and being able to incorporate selected information into his or her knowledge base and value system 4. a similar number were able to answer questions concerning understanding many of the ethical, legal and socio-economic issues surrounding information and information technology correctly.

20. Department of Information & Communications What do the results tell us? ACRL2:Students struggled with answering questions about: Selecting the most appropriate methods and strategies for accessing the needed information, for example: Did not understand about different types of database and what a library catalogue was Could not determine the provenance of the information or sources Constructing and implementing effectively designed search strategies Did not know how to expand searches or apply them to different databases Refining the search strategy if necessary Unable to use boolean operators, truncation, phrase searching Extracting records and managing the information and its sources Could not recognise different item formats from bibliographic search results presented to them from databases

21. Department of Information & Communications The answers to our remaining questions Q. Does intervention make a difference in the results? Q. Is there a need in for a UK designed, generic, online Information Literacy Audit? Stage two testing completed 14 th March 2008 Stage two analysis completed by the end of May 2008 Stage three testing commences November 2008 In progress

22. Department of Information & Communications References The Association of College and Research Libraries (2000). Information Literacy Competency Standards for Higher Education. ACRL Wise, S. L. & Yang, S. (2003). The Adaptex Assessment System (Version 1.5). U.S. Department of Education Fund for the Improvement of Postsecondary Education. Wise,S,.L. Cameron, L.,Yang, S. & Davis, S. (2005) Information Literacy Test: Test Development and Administration Manual. Harrisonburg: Institute for Computer-Based Assessment; Center for Assessment & Research Studies, James Maddison University. The Institute for Computer Based Assessment, James Madison University(2006) Information Literacy Test (ILT) [Online: http://www.jmu.edu/icba/prodserv/instruments_ilt.htm accessed 21st February 2008) http://www.jmu.edu/icba/prodserv/instruments_ilt.htm