1. Assessing Co-curricular Learning
Robert Mundhenk
Visiting Scholar
The Higher Learning Commission

2. Traditional Approaches to Assessment of Learning
Rescheduled for Jan. 19 at 3:30 – 5:00 p.m. YU 204
Co-curricular Assessment workshop presented by Bob Mundhenk                 Yellowjacket Union 203  While most institutions have devoted effort and resources to the effectiveness of the co-curricular services and programs they offer, only recently have they started to assess the learning these areas produce. This session will explore ways to assess the impact of co-curricular efforts on student learning and development.
Targeted Audience:  Faculty and staff in non-academic units interested in assessment.     
Ask Institutional Research about graduation, retention, GPAs, and the like
Ask faculty about their teaching and the learning it produces—but not necessarily whether they know they’re producing learning
Ask faculty how they know they’re producing learning
Traditional Approaches to Assessment of Learning

3. The Institutional Mismatch
Traditional wisdom: Learning outcomes need to be aligned at course, program, and institutional levels
But where are many general education goals, like “tolerance” and “teamwork” and “the ability to function in an increasingly diverse world” and “inclination” taught and assessed?
Or is “taught” the right word???
The Institutional Mismatch

4. Shifting Perspectives
What happens if we substitute the word “learned” for the word “taught”?
What are the implications of “Where are learning outcomes learned and assessed”?
Emphasis on student demonstration, not topic- covering
Ability to do or apply supersedes knowing
Responsibility for learning is shared
Site of learning becomes less specific, and boundaries become more fungible
Shifting Perspectives

5. After Learning Reconsidered
“Learning” is not exclusively classroom- based
Many valued outcomes are not taught exclusively in the classroom
Many valued outcomes are the result of processes outside the classroom
“Learning” is a process based on three interdependent student experiences:
Understanding academic content and processes
Student development
Identity formation
After Learning Reconsidered

6. After Learning Reconsidered
Responsibility for “learning” exists outside the classroom
Responsibility for “learning” doesn’t always take the same form; some entities on campus produce it, some facilitate it, some support it
Responsibility for assessing learning exists outside the classroom as well
After Learning Reconsidered

7. Some Post-LR Examples of Learning
Civic Responsibility
AA: Service learning
SA: Student government, voter registration, student judicial boards
Think and Engage as a Global Citizen
AA: Language courses, Anthropology, Sociology
SA: International experiences, culture days, residence halls
Some Post-LR Examples of Learning

8. UW- Superior’s Five Institutional Goals
Ability and inclination to:
Think and make connections across disciplines
Express oneself in multiple forms
Analyze and reflect upon multiple perspectives to arrive at a perspective of one’s own
Think and engage as a global citizen
Engage in evidence-based problem-solving
UW- Superior’s Five Institutional Goals

9. Ability and Inclination
How is ability made evident?
Where and how is ability developed?
Where is ability assessed?
What is “inclination”?
How is it made evident?
Where and how is it developed?
Where is it assessed?
Ability and Inclination

10. How Do These Outcomes Connect with Undergraduate Learning Outcomes?
Campus Life learning outcomes:
Recognizing and enhancing leadership potential
Developing an appreciation of human differences
Seeking opportunities to engage in the campus community and beyond
Expanding desire for life-long learning
How Do These Outcomes Connect with Undergraduate Learning Outcomes?

11. What Ability or Inclination Do They Develop?
Campus Life learning outcomes:
Recognizing and enhancing leadership potential
Developing an appreciation of human differences
Seeking opportunities to engage in the campus community and beyond
Expanding desire for life-long learning
What Ability or Inclination Do They Develop?

12. How Do We Know They’re Achieved?
Campus Life learning outcomes:
Recognizing and enhancing leadership potential
Developing an appreciation of human differences
Seeking opportunities to engage in the campus community and beyond
Expanding desire for life-long learning
How Do We Know They’re Achieved?

13. Co-curricular Outcomes
Need to be intentional
Need to be planned
Need to be part of the structure of a student’s experience
Need to be assessed
Co-curricular Outcomes

14. BEING INTENTIONAL

15. Traditional Co-curricular Assessment
Efficiency models:
Focus on process
How well is this office/service functioning?
Focus on numbers:
Clients served
Graduation rates
Tutorial visits
Attendance at activities
Student/staff ratios
Traditional Co-curricular Assessment

16. Newer Co-curricular Assessment
Effectiveness Models: Indirect
Based on surveys and other indirect indicators, like NSSE
Often rely on student self-reporting
Tend to skew positively on outcomes, if not always on the processes that led to them
Newer Co-curricular Assessment

17. Newer Co-curricular Assessment
Effectiveness Models: Direct
Focuses on student performance
Can be based on observation or objective measures
Require carefully designed and consistent measuring practices
Newer Co-curricular Assessment

18. How to Assess Co-curricular Learning
Apply external standards, like CAS
Use surveys and questionnaires
Develop direct measurement strategies
All of the above
How to Assess Co-curricular Learning

19. CAS Standards Standards for 40 functional areas
Thirteen component parts:
Mission
Program
Leadership
Organization and management
Human resources
Financial resources
Facilities, technology, and equipment
Legal responsibilities
Equity and access
Campus and community relations
Diversity
Ethics
Assessment and evaluation
CAS Standards

20. CAS’s Six Outcome Domains
Knowledge acquisition, construction, integration, and application
Cognitive complexity
Intrapersonal Development
Interpersonal competence
Humanitarianism/Civic Engagement
Practical Competence
CAS’s Six Outcome Domains

21. Can easily document the efficiency of processes and organization
Can be used as well (through an emphasis on the domains) to chart the effectiveness of outcome development efforts—depending on outcome definition and quality of evidence
Using CAS Standards

22. Career Services: As a result of interactions with the Career Services Office, students and alumni will:
Identify their skills, abilities, and strengths in order to make knowledgeable career decisions
Have the necessary resources and skills to prepare for life-long post-graduate experiences
How are these outcomes connected to institutional learning outcomes?
What are “resources and skills”?
Outcome Definition

23. College Unions: As a result of experiences in the Yellowjacket Union, students will:
Identify and utilize the opportunities and services available to them
Demonstrate a sense of ownership for the campus community and civic involvement
Interact with and value individuals from diverse backgrounds and lifestyles
How do you know you’ve achieved the second and third outcomes?
How do they overlap with institutional outcomes?
Outcome Definition

24. Outcome Definition and Evidence
Clearly defined outcomes help determine the nature of evidence to be collected
Clearly defined outcomes focus on student performance and development, thus calling for both direct and indirect forms of evidence
Outcome Definition and Evidence

25. Direct and Indirect Measures
DIRECT: uses performance or product, created by students, that can be compared to expected outcomes --Capstone courses, projects, internships, clinical experiences, etc. INDIRECT: uses information that does not directly link the learning to the outcomes --graduation rates, grades, surveys, “usage” rates
Direct and Indirect Measures

26. Questions about Evidence
Is it relevant to the area’s stated mission and function?
Does it measure what we want it to measure?
Does it deal in some way with outcomes?
Is the information derived useful?
Can the information be used to improve either function or learning?
Questions about Evidence

27. Surveys and Questionnaires: Some Sample Questions
Did you accomplish what you hoped to accomplish in your meeting with your advisor?
How well did your experience at X prepare you for employment?
As a result of this First-Year program, do you feel better prepared for college?
Write a short essay in which you describe the ways in which your attitudes and values have changed as a result of your semester in Argentina.
Surveys and Questionnaires: Some Sample Questions

28. Surveys and Questionnaires: Indirect Evidence
Traditional, indirect source of information on effectiveness
Limitations:
Self-reporting
Unvalidated opinion
Response rates
Opportunistic data
Skewed samples
Surveys and Questionnaires: Indirect Evidence

29. Surveys and Questionnaires
Kinds:
Satisfaction
Reflective
Post-experience experience (alumni and employers)
Value:
True “customer” response
Can indicate areas for improvement and ratification
Provides data for planning and review
Surveys and Questionnaires

30. Surveys and Questionnaires
Making them tools to assess learning:
Use learning outcomes as basis for at least some questions
Validate by cross-referencing outcomes with different populations (employers, alumni, graduate, current students)
Emphasize the learning outcomes in design and analysis of surveys and questionnaires
Surveys and Questionnaires

31. Direct Effectiveness Measures
If Learning Reconsidered made the case for cross-campus responsibility for learning, then assessment of learning outcomes is also a cross-campus responsibility
Adaptation of practices and devices already in use in academic settings
Standardized
Judgment-based
Direct Effectiveness Measures

32. Developing Direct Measures of Effectiveness
Intentional Planning:
Determine areas of responsibility: what office/function might be a logical place to contribute to particular learning outcomes?
Plan the outcome-based purpose of the activity
Aim at the appropriate level of Bloom’s taxonomy
Design non-passive activities (watching a film plus discussion; International Days as more than food, costumes, and dance)
Design outcome-focused opportunities for processing
Developing Direct Measures of Effectiveness

33. Understanding the Outcomes
What is “inclination” and how is it developed in co-curricular activities?
How does a student “express oneself in multiple forms”?
What is a “global citizen”?
What does it mean to “make connections across academic disciplines” and how might co-curricular activities have a role in developing this outcome?
Understanding the Outcomes

34. Choose one of the outcomes below and determine how your co-curricular area might have some responsibility for developing it. Name specific activities that might help develop the outcome and specify what their effect on the student should be.
Think and make connections across academic disciplines
Express oneself in multiple forms
Analyze and reflect upon multiple perspectives to arrive at a perspective of one’s own
Think and engage as a global citizen
Engage in evidence-based problem-solving
Being Intentional

35. Planning for the Long Term
Mapping:
If the learning outcome is important, single exposure isn’t enough
How do first-year experiences differ from last- year ones—or what difference is expected in student response?
How to assure student’s development of outcomes from first year to last?
Planning for the Long Term

36. One-shot assessment produces haphazard results that are usually insufficient for planning improvement
Tie assessments to logical stages of development, based on an outcome map
Be consistent in approach to assessing
Options:
Standardized instruments
Self-generated tools
Regular Assessment

37. Self-generated Tools Observations Expert judgments
Student self-reflection
Employer/supervisor judgments
Self-generated Tools

38. Using Self-Generated Tools
Consistency across observers is crucial, so a rubric of some kind is essential
Holistic rubrics: broad judgments (Acceptable/Not Acceptable/Needs Improvement)
Descriptive rubrics: defined criteria and measures
Using Self-Generated Tools

39. First, determine the aspects of student performance that would indicate he/she has achieved an outcome (e.g., one aspect of a “social justice” outcome might be “the student’s writing demonstrates sensitivity to issues of class and power”)
Second, define the specific things a student would have to do to show he/she has mastered that aspect (e.g., “Clear understanding of the ways in which economic status affects behavior.” (Criteria)
Finally, describe degrees of achievement for each criterion (Measures)
Descriptive Rubrics

40. Using the outcome and the functional area you chose earlier, develop a rubric to measure student achievement, defining one performative aspect of the outcome, one criterion for measuring that aspect, and a system (holistic, descriptive, whatever) for describing degrees of attainment
Developing a Rubric

41. Institutional Assessment
Assessments done across campus should ideally use the same rubrics or measures
When using the same tools is not possible, it is essential that there be a way to extract information that is usable at the institutional level while still serving the needs of the functional area
Institutional Assessment

42. To conclude:
Understand the meaning of the desired outcomes and your role in developing them
Separate efficiency from effectiveness
Plan experiences and assessments carefully
Focus on using assessment information to improve learning, not to justify your existence
Collect information that is relevant, meaningful, and useful
Design systems that are reasonable and manageable

43. ROBERT MUNDHENK