Imperfect but Authentic and Invaluable Information Literacy Assessment by Applying Rubrics to Anonymized Student Papers Dana Longley Assistant Director for Library Instruction and Information Literacy Slides: 

Not Your Typical College Non-traditional students 18,600+ primarily part-time, working adults (ave age: 35) 50% blended, 50% fully online 7 overlapping terms schedule No “campus” (35 regional centers in NYS + overseas) Highly individualized study Customizable degree programs One-on-one mentoring Narrative feedback (tests not generally used)  No required courses Students design their own degree programs in the only required course: Degree Planning – so basically no place to hit all students.

Online Library Small staff (4 PEs) No director, student workers, or support staff Fully online collections (no physical library) No student info lit requirements (infused Info Mgt Gen Ed)

Learning Assessment Timeline 1980s: Area of Study reviews (degree portfolios) 2002-03: SUNY-wide Gen. Ed. assessments 2007-08: Assessments in the Major 2012: College Learning Goals Policy drafted. Includes: “Information and Digital Media Literacy: Critically access, evaluate, understand, create and share information using a range of collaborative technologies to advance learning, as well as personal and professional development.“

Info Lit Assessments Official: Gen. Ed. Info Management: via sample of student papers Unofficial: Assessment in the Major: student papers Synchronous Library Workshops: pre- & post-tests (voluntary) Research Skills self-paced tutorial (voluntary) 2 credit Info Lit course (voluntary)

Information Management Competency: Ability to locate and use information from a variety of sources Outcomes: Determine the quality of information needed Locate an appropriate variety of information Evaluate information and resources critically Use information effectively to accomplish a specific purpose Attribute external information sources and ideas correctly

Uses a zero to three scale.

Pre-Assessment Process Managed/owned by Office of Decision Support Appropriate advanced level courses and assignments ID’d by faculty Assessment team volunteers sought and verified Papers + assignment instructions put into Adobe and all personal info scrubbed Team reviews/discusses rubric and conducts norming session This entire process is owned/managed by Office of Decision Support, not library.  Advanced level =junior/senior equivalent Team usually equals handful of faculty plus librarian (me) Norming session: to come to shared understanding of what rubric means.

Team Assessment Sessions 2-day formal assessment session:  4-6 assessors Most samples rated by 2 members  Ratings input into Survey Monkey  Analysis & inter-rater reliability via SPSS Team discusses follow-up recommendations Decision Support crafts report Report shared internally Report drafted, sent to team for initial comment/changes,then shared college-wide(but not made publicly available) Report is geared to internal audience and mostly for administrative purposes (I.e.,SUNY reporting)

Pros and Cons? Pros: Authentic student work Faculty and librarian input Cons: Workload intensive when done Uses online courses only (not representative?) Some assignments don't fit rubric well Small team sizes  Subjectivity of rubric interpretation & scores? No formal mechanism for implementing recommendations Most assignment instructions don't highlight/require all elements from rubric.

Results - Trends Out of a 0-3 scoring scale.

More Trends More than 50% of papers consistently score below Meets

More Trends

Possible broad conclusions? Not great! >50% samples = “does not meet” or “approaches” Perception of info lit concepts growing among faculty? Students have trouble especially with: Synthesis Evaluation Attribution

Quality

Variety

Evaluation

Use

Attribute

2016 Results by Outcome

2016 Results by Score

Other IL-related Assessments Many Major rubrics include an entry for "research" (or similar) - usually vaguely defined

Credit Course Data Self-assessment quiz at start: Students struggle with: Topic formation ID of non-standard citations Search strings Understanding what lit reviews and primary sources are In-text citations Terminology (library jargon): journal vs journal article; database Students already have some knowledge of: ID of journal & book citations Peer-review Plagiarism basics Basic publication types Quiz: small number of responses so far (new) Non-standard citations: book chapter Search strings: tend to enter entire research question/sentence or put it all in quotes

Looking Forward Ongoing or near future: Update Information Management rubric Better ID of relevant research assignments Info Lit self-assessment in online orientation Increased faculty awareness of IL assessment results & course and curricula remediation options Longer term: Similar paper assessment between 4 yr periods (by library?) Update rubric for clarity of rating and better align with Framework and College Learning Goals Use papers from same courses assignments over time as better way to gauge?

Questions? Slides:  dana.longley@esc.edu @disobedientlib (twitter)