1. THE ‘UNDERSTANDING LIBRARY IMPACTS’ PROTOCOL: DEMONSTRATING ACADEMIC LIBRARY CONTRIBUTIONS TO STUDENT LEARNING OUTCOMES IN THE AGE OF ACCOUNTABILITY Derek Rodriguez Doctoral Candidate, School of Information and Library Science The University of North Carolina at Chapel Hill Program Officer, Triangle Research Libraries Network 9th Northumbria International Conference on Performance Measurement in Libraries and Information Services, August 23, 2011, York, England

2. Undergraduate education in the age of accountability Concerns Access and cost Attainment Competencies of graduates Metrics Retention Graduation rates Student learning outcomes

3. Locally defined learning outcomes General education and the academic major Broad abilities and cross-discipline skills e.g. Critical thinking, inquiry and analysis, information literacy, quantitative literacy, written communication VALUE rubrics (Rhodes, 2010) Discipline specific knowledge and skills By academic major, e.g. History, Chemistry, Nursing, etc. Tuning projects (Gonzalez & Wagenaar, 2005; ICHE, 2010) - Rhodes, T., ed. 2010. Assessing Outcomes and Improving Achievement: Tips and Tools for Using Rubrics. Washington, DC: Association of American Colleges and Universities. - González, J. and Wagenaar, R. (eds.) (2005). Tuning Educational Structures in Europe II. Bilbao, ES: University of Deusto. - Indiana Commission for Higher Education. Tuning USA Final Report: The 2009 Indiana Pilot, 2010.

4. The challenge for libraries Library use Student effort Student Learning Assessment Program reviews External frameworks ? - Connecting library use to student learning outcomes VALUE & Tuning

5. ‘Understanding Library Impacts’ - Explores library use as a component of student effort Library use Learning Activities Student Learning Assessment Program reviews Student effort External frameworks VALUE & Tuning

6. A focus for library assessment Capstone experience General education Academic major Expectations Student effort

7. Design for pilot study (spring 2011) Population: undergraduate history majors enrolled at 2 sites, a liberal arts college and a liberal arts university in the U.S. (N=74) Project: 20 – 40 page research paper using evidence from primary sources Step 1: Constructed a learning activities crosswalk using syllabi and rubrics for research papers Step 2: Students completed a web-based critical incident survey after turning in papers Step 3: Reports were distributed to study sites via secure, database-driven web-sites

8. Understanding Library Impacts - Explores library use as a component of student effort Library use Learning Activities Student Learning Assessment Program reviews Critical incident survey Learning activities crosswalk External frameworks ‘Capstone’ project VALUE & Tuning

9. Activities Getting oriented Choosing a topic Developing a thesis Gathering evidence Finding other sources Creating a bibliography Writing Discipline-specific skills -Locates secondary and primary sources -Distinguishes among types of sources Evidence and analysis -Evaluates and interpret primary sources -Uses secondary sources to provide context -Demonstrates an understanding of the methods of history Thesis and argument -Develops original thesis statement -Advances argument in support of thesis using evidence from primary sources Writing and Citing -Communicates argument in a coherent, well- written paper -Follows disciplinary style and citation standards Capstone-related outcomes Learning activities crosswalk

10. Critical incident survey - Developed and refined in qualitative studies (2006, 2007) Library use - Electronic resources - Traditional resources - Services - Facilities / Equipment Top-ranked e-resource Top-ranked traditional resource Top-ranked service Top-ranked facility Learning activities Helps and problems Open ended questions and probes Demographics and affect Academic challenge Local questions Deliverables and outcomes

13. Outcomes related to the activity ‘Gathering evidence’ -Locates secondary and primary sources -Distinguishes among types of sources -Evaluates and interprets primary sources -Advances argument in support of thesis using evidence from primary sources

14. Outcomes related to the activity ‘Writing’ -Uses secondary sources to provide context -Advances argument in support of thesis using evidence from primary sources -Communicates argument in a coherent, well-written paper -Follows disciplinary style and citation standards

15. Factors of use (helps and problems) Derived from qualitative studies and LIS literature Gathered for ‘ranked uses’ only Categorized for analysis Example category: ‘Help finding information’ “I learned about information sources for my project” (help) “I learned new skills” (help) “helped me when I got stuck” (help) “the assistance I received wasn't helpful” (problem) “it was difficult to find someone to help me” (problem)

17. “Overwhelmed by various themes and layers of my research” (A-2) “Putting all of my information together into a cohesive paper” (A-5) “Time management” (A-8) “Several of the books I needed were being rebound…” (B-3) “Uncertainty about where to start” (B-10) “I was worried about finding primary source material” (B-15) “At a somewhat early point in the project, I realized that the scope of my project was much too wide …” (B-14) Challenges faced during the project Site A, n=17Site B, n=22

18. Most important library uses, by type of use, n=37 “… those databases, JSTOR specifically, really helped me.” (B-4) “Library space to work on my thesis both writing and research” (B-2) “being able to get books from off campus” (B-3) Alternate activity, if first choice hadn’t been available, n=29

19. Understanding Library Impacts protocol - Contributions and implications Library use Learning Activities Student Learning Assessment Program reviews External frameworks - Credible connections between library use and expectations for student learning - Generates rich data useful for advocacy and improvement - Places library in important campus conversations about student learning ‘Capstone’ project VALUE & Tuning

20. Next steps Refining instruments; conducting reliability and validity testing Exploring “patterns and predictions” Continue evaluation with history majors at 4 more sites (fall 2011) Future work Evaluate protocol with other disciplines and settings Explore integration with assessment systems and analytics projects Questions? Contact: Derek_Rodriguez@unc.edu

21. Crosswalk: Broad abilities and discipline- specific outcomes VALUE Rubrics (Rhodes, 2010) Critical thinking Inquiry and analysis Information literacy Written Communication Tuning Outcomes for History (ICHE, 2010) Historical knowledge Thinking and analytical skills Communication skills Personal motivation Rhodes, T., ed. 2010. Assessing Outcomes and Improving Achievement: Tips and Tools for Using Rubrics. Washington, DC: Association of American Colleges and Universities. Indiana Commission for Higher Education. Tuning USA Final Report: The 2009 Indiana Pilot, 2010.

22. Related capstone outcomes -Locates secondary and primary sources -Distinguishes among types of sources -Evaluates and interprets primary sources -Advances argument in support of thesis using evidence from primary sources

23. Outcomes related to ‘gathering evidence’ Activity Gathering evidence Tuning Outcome Formulate and test plausible historical hypotheses and marshal an argument. Project outcome Advances argument in support of thesis using evidence from primary sources VALUE Inquiry and Analysis Analysis - Organizes and synthesizes evidence to reveal insightful patterns, differences, or similarities.