1. Imperfect but Authentic and Invaluable
Information Literacy Assessment by Applying Rubrics to Anonymized Student Papers
Dana Longley
Assistant Director for Library Instruction and Information Literacy
Slides: 

2. Not Your Typical College
Non-traditional students
18,600+ primarily part-time, working adults (ave age: 35)
50% blended, 50% fully online
7 overlapping terms schedule
No “campus” (35 regional centers in NYS + overseas)
Highly individualized study
Customizable degree programs
One-on-one mentoring
Narrative feedback (tests not generally used) 
No required courses
Students design their own degree programs in the only required course: Degree Planning – so basically no place to hit all students.

3. Online Library Small staff (4 PEs)
No director, student workers, or support staff
Fully online collections (no physical library)
No student info lit requirements (infused Info Mgt Gen Ed)

4. Learning Assessment Timeline
1980s: Area of Study reviews (degree portfolios)
: SUNY-wide Gen. Ed. assessments
: Assessments in the Major
2012: College Learning Goals Policy drafted. Includes:
“Information and Digital Media Literacy: Critically access, evaluate, understand, create and share information using a range of collaborative technologies to advance learning, as well as personal and professional development.“

5. Info Lit Assessments Official:
Gen. Ed. Info Management: via sample of student papers
Unofficial:
Assessment in the Major: student papers
Synchronous Library Workshops: pre- & post-tests (voluntary)
Research Skills self-paced tutorial (voluntary)
2 credit Info Lit course (voluntary)

6. Information Management
Competency: Ability to locate and use information from a variety of sources
Outcomes:
Determine the quality of information needed
Locate an appropriate variety of information
Evaluate information and resources critically
Use information effectively to accomplish a specific purpose
Attribute external information sources and ideas correctly

7. Uses a zero to three scale.

8. Pre-Assessment Process
Managed/owned by Office of Decision Support
Appropriate advanced level courses and assignments ID’d by faculty
Assessment team volunteers sought and verified
Papers + assignment instructions put into Adobe and all personal info scrubbed
Team reviews/discusses rubric and conducts norming session
This entire process is owned/managed by Office of Decision Support, not library. 
Advanced level =junior/senior equivalent
Team usually equals handful of faculty plus librarian (me)
Norming session: to come to shared understanding of what rubric means.

9. Team Assessment Sessions
2-day formal assessment session: 
4-6 assessors
Most samples rated by 2 members 
Ratings input into Survey Monkey 
Analysis & inter-rater reliability via SPSS
Team discusses follow-up recommendations
Decision Support crafts report
Report shared internally
Report drafted, sent to team for initial comment/changes,then shared college-wide(but not made publicly available)
Report is geared to internal audience and mostly for administrative purposes (I.e.,SUNY reporting)

10. Pros and Cons? Pros: Authentic student work
Faculty and librarian input
Cons:
Workload intensive when done
Uses online courses only (not representative?)
Some assignments don't fit rubric well
Small team sizes 
Subjectivity of rubric interpretation & scores?
No formal mechanism for implementing recommendations
Most assignment instructions don't highlight/require all elements from rubric.

11. Results - Trends
Out of a 0-3 scoring scale.

12. More Trends
More than 50% of papers consistently score below Meets

13. More Trends

14. Possible broad conclusions?
Not great!
>50% samples = “does not meet” or “approaches”
Perception of info lit concepts growing among faculty?
Students have trouble especially with:
Synthesis
Evaluation
Attribution

15. Quality

16. Variety

17. Evaluation

18. Use

19. Attribute

20. 2016 Results by Outcome

21. 2016 Results by Score

22. Other IL-related Assessments
Many Major rubrics include an entry for "research" (or similar) - usually vaguely defined

23. Credit Course Data Self-assessment quiz at start:
Students struggle with:
Topic formation
ID of non-standard citations
Search strings
Understanding what lit reviews and primary sources are
In-text citations
Terminology (library jargon): journal vs journal article; database
Students already have some knowledge of:
ID of journal & book citations
Peer-review
Plagiarism basics
Basic publication types
Quiz: small number of responses so far (new)
Non-standard citations: book chapter
Search strings: tend to enter entire research question/sentence or put it all in quotes

24. Looking Forward Ongoing or near future:
Update Information Management rubric
Better ID of relevant research assignments
Info Lit self-assessment in online orientation
Increased faculty awareness of IL assessment results & course and curricula remediation options
Longer term:
Similar paper assessment between 4 yr periods (by library?)
Update rubric for clarity of rating and better align with Framework and College Learning Goals
Use papers from same courses assignments over time as better way to gauge?

25. Questions?
Slides: 
@disobedientlib (twitter)