Practical Strategies for Assessing Student Learning in Departments and Programs: A Utilization-Focused Approach Jo Beld St. Olaf College

Guiding questions Who are the users of the evidence?Who are the users of the evidence? How might the evidence be used?How might the evidence be used? How will you gather the evidence?How will you gather the evidence? How will you animate the evidence?How will you animate the evidence?

Theoretical perspectives Utilization-focused assessment (Patton, 2008): Focus on intended uses by intended users

Theoretical perspectives Backward design (Wiggins & McTighe, 2005): “Beginning with the end in mind”

Theoretical perspectives Traditional instructional design: Choose texts Develop classroom activities Develop classroom activities Make up tests

Theoretical perspectives Backward instructional design: Identify intended learning outcomes Determine appropriate evidence Determine appropriate evidence Plan instruction and practice

Theoretical perspectives Traditional assessment design: Choose an assessment instrument Gather and summarize evidence Gather and summarize evidence Send a report to someone

Theoretical perspectives Backward assessment design: Identify intended users and uses Define and locate the learning Define and locate the learning Choose assessment instrument Choose assessment instrument

Who are the users? Faculty roles, commitments, and disciplinary identities offer both incentives and disincentives to engage assessment

What are the uses? What kind of outcome do you want to investigate… –Knowledge (“Understanding of…”) –Proficiencies (“The ability to…”) –Practices (“The habit of…”) –Values or attitudes (“A concern for…”)

What are the uses? …and why? Affirming current practicesAffirming current practices Tweaking the content of key coursesTweaking the content of key courses Extending a specific pedagogyExtending a specific pedagogy Enhancing “scaffolding”Enhancing “scaffolding” Piloting innovationsPiloting innovations Supporting grant applicationsSupporting grant applications Setting future assessment agendasSetting future assessment agendas

How will you gather evidence? Assessment is like any other kind of investigation; use strategies that fit the questions you are trying to answer

How will you gather evidence? “Direct” Assessment” “Direct” Assessment Evidence of what students actually know, can do, or care about “Indirect” Assessment Evidence of learning-related experiences or perceptions

How will you gather evidence? Common direct assessment “artifacts”  Papers, essays, abstracts  Presentations and posters  Oral or written examination items  Analytic journals  Responses to survey or interview questions that ask for examples of knowledge, practice, or value

How will you gather evidence? Common indirect assessment “artifacts”  Course-taking patterns or transcript analysis  Responses to survey or interview questions about experiences, perceptions, self-reported progress, or impact of program experiences  Reflective journals

How will you gather evidence? But wait!! Aren’t we observing student work all the time anyway? What’s the difference between grading and assessment?

How will you gather evidence? Gradingsummarizes many outcomes for one student Assessmentsummarizes one outcome for many students

How will you gather evidence? The purpose of an assessment instrument is to provide systematic, summarized information about the extent to which a group of students has realized one or more intended learning outcomes

How will you gather evidence? Options to consider:  Use an instrument developed by someone else  Adapt an existing instrument  Add to something you’re already doing  Connect to institutional-level evidence  Invent something new

How will you gather evidence? Where possible, pair indirect observations of processes and perceptions with direct observations of outcomes (pp ).

How will you gather evidence? The dual goal of sampling: RepresentativenessandManageability

How will you gather evidence? Examples involving comprehensive sampling:  Survey of all senior majors  Application of rubric to all research abstracts in all seminars  Application of rubric to all work submitted for senior art show

How will you gather evidence? Examples involving selective sampling:  Application of rubric to randomly-selected subset of final papers in capstone course  Pre/post administration of locally-developed quiz in required sophomore methods course  End-of-course survey in one introductory and one senior-level course  Aggregation of results on selected items in an evaluation form for student work

How will you animate the evidence? Evidence never speaks for itself; be intentional in summarizing, interpreting, and responding to the findings

How will you animate the evidence? Plan intentionally for summarizing and interpreting findings: Delegate and distributeDelegate and distribute Enlist help from staff and studentsEnlist help from staff and students Use technology to save timeUse technology to save time Balance individual and collective workBalance individual and collective work Dedicate time for discussionDedicate time for discussion

How will you animate the evidence? Plan intentionally for responding to the findings: Borrow strategies from past successes in collective departmental actionBorrow strategies from past successes in collective departmental action Focus reporting on planned actions, not on the evidence itselfFocus reporting on planned actions, not on the evidence itself Weight Watchers trumps The Biggest LoserWeight Watchers trumps The Biggest Loser Dedicate resources for actionDedicate resources for action

A final thought…. Less really is more!