1. Ada Emmett and Judith Emde University of Kansas Libraries Assessing Chemical Information Literacy Skills using the ACRL Standards ALA 2006 Annual Meeting, New Orleans STS Research Forum Sunday June 25 th

2. 2 Overview  Chemistry bibliography course  Assessment tool development to measure learning outcomes  Assessment results and observations  ACRL standards: advantages

3. 3 Bibliography of Chemistry : CHEM 720  One hour credit offered for graduate students  Major chemistry and biomedical research tools  Grading: satisfactory/unsatisfactory

4. 4 Questions  What should students learn?  Does teaching produce the desired learning outcomes?  How can we assess student learning?

5. 5  Used to develop:  Learning outcomes  Teaching activities  Assessment tool  An “information literate” student ACRL’s Information Literacy Competency Standards for Higher Education

6. 6  Alternatives to ACRL literacy standards  ACS  STS

7. 7 ACRL’s Information Literacy Competency Standards for Higher Education 1. Determine the extent of information needed 2. Access the needed information effectively and efficiently 3. Evaluate information and its sources critically and incorporate selected information into his or her knowledge base and value system. 4. Use information effectively to accomplish a specific purpose 5. Understand the economic, legal, and social issues surrounding the use of information, and access and use information ethically and legally http://www.ala.org/ala/acrl/acrlstandards/informationliteracycompetency.htm

8. 8 Year One: 2004  Methodology  Developed assessment tool  Conducted pre- and post-course assessment interviews  Used control group  Quantified data

9. 9 Year One: 2004  Results Average score Pre-Test Score Post-Test Score PossiblePercent Improvement Points Improved Students in Class N=26 45657244%20 “Control” N=4 48567217%8

10. 10 Year One: 2004 Pre- Possible Post-

11. 11 Year Two: 2005  Assessment tool: redesign and refinement  SPSS Data Entry Builder to enter data  One-on-one interviews  Pre- and post-test given  students 16  small control group 5

12. 12 Year Two: 2005  Results Average score Pre-Test Score Post- Test Score PossiblePercent Improvement Points Improved Students in Class N=16 619912062%38 “Control” N=5 738512016%12

13. 13 Year Two: 2005  Results

14. 14 Year Three: 2006  Used “backward design”– starting with ACRL learning outcomes  Develop assessment questions  Design course lectures/assignments  Create tool – web-based questionnaire  Run as pre- and post-test  No controls

15. 15 Year Three: 2006  Results  Points Improved: 27  Percent Improved: 57% Pre- Post- Possible

16. 16 Comparison by Assessment Questions

17. 17 2004-2006 Improvement  2004: 44% improvement (“control” 16%)  2005: 62% improvement (“control” 17%)  2006: 57% improvement

18. 18 Assessment questions grouped by standards Standard #2: The information literate student accesses needed information effectively and efficiently. Performance indicator #5: The information literate student extracts, records, and manages the information and its sources. Outcome “e”: Uses various technologies to manage the information selected and organized. Assessment questions:  Select among the following tool(s) that manage references or citations and therefore assist in writing research papers.  Describe two features of a software program that assist in managing citations.

19. 19 Assessment questions grouped by standards

20. 20 Students’ perceptions

21. 21 Observations  Mapping standards is subjective  Consultation on mapping questions to standards  Some standards such as 3 and 4 are difficult to assess  Strong subject expertise needed (faculty/instructor collaboration)  Mechanism of delivering assessment tool needs improvement  Ongoing review of assessment tools’ strengths/weaknesses

22. 22 Observations  Assessing all learning outcomes difficult  Keep it simple to several specific outcomes.  Did the class and the test incorporate most important learning outcomes for that student group?  Faculty/instructor collaboration

23. 23 ACRL standards: advantages  Provides mechanism to assess information literacy skills via learning outcomes (using “backward design”)  Assists in the development of course content through backward design

24. 24 Discussion, Questions and Comments

25. 25 Contact information Ada Emmett aemmett@ku.edu 785-864-8831 Judith Emde jemde@ku.edu 785-864-4931 Supplemental information at http://www.people.ku.edu/~jemde/ http://www.people.ku.edu/~jemde/

26. 26

27. 27 Mapping lectures / exercises to standards  Potential learning outcome of lecture and exercise mapped to standards at beginning of semester  Comparison of lectures/exercises to results of post assessment by grouped standards.

28. 28 Mapping lectures / exercises to standards