1. Developing and Assessing Program-Level Learning Outcomes Peter Wolf pwolf@uoguelph.ca April 17, 2013

2. Agenda ► ► ‘Ideal’ Curriculum Development Process ► ► Developing Program Outcomes ► ► Assessing Program Outcomes

3. The Five-Minute University http://www.youtube.com/watch?v=kO8x8eoU3L4

4. Program Outcomes Foundational Content ‘Ideal’ Curriculum Development Process University Learning Objectives Program Structure Educational Experiences Required & Elective Courses

5. Key Question What are the knowledge, skills & values that we want to foster in our students?

6. Attributes of the ‘Ideal Undergraduate Graduate’ ► University Learning Outcomes ► Existing Program Outcomes ► Professional Organizations/Accreditation ► Other Similar Programs ► Disciplinary Norms and Language

7. Ontario Council of Academic Vice Presidents Degree Level Expectations ► Depth and Breadth of Knowledge ► Knowledge of Methodologies ► Application of Knowledge ► Communication Skills ► Awareness of Limits of Knowledge ► Autonomy and Professional Capacity http://www.cou.on.ca/content/objects/Undergrad%20Degree%20Expectations%20FINALen1.pdf

8. University of Guelph Learning Objectives (2013) ► Critical and Creative Thinking  Inquiry and Analysis, Problem Solving, Creativity, Depth and Breadth of Understanding ► Literacy  Information Literacy, Quantitative Literacy, Technological Literacy, Visual Literacy ► Global Understanding  Global Understanding, Sense of Historical Development, Civic Knowledge and Engagement, Intercultural Competence ► Communicating  Oral Communication, Written Communication, Reading Comprehension, Integrative Communication ► Professional and Ethical Behaviour  Teamwork, Ethical Reasoning, Leadership, Personal Organization and Time Management

9. Bachelors of Arts and Sciences Program Outcomes ► Pose & solve problems by drawing on & integrating protocols & methods from program ► Oral & written communications, for academic & general audiences ► Conduct traditional & electronic research, in humanities/social sciences & scientific contexts

10. Key Question What are the knowledge, skills & values that we want to foster in our students? Focus on the program-level

11. Taxonomy Creating (design, construct, generate ideas) Evaluating/Synthesizing (critique, judge, justify decision) Analyzing (compare, organize, differentiate) Applying (use in new situation) Understanding (explain, summarize, infer) Remembering/Knowing (list, describe, name) Anderson, L. W. and David R. Krathwohl, D. R., et al (Eds..) (2001) A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives Bloom’s (cognitive) Receiving (asks, describes, points to) Responding (answers, performs, practices) Valuing (demonstrates belief in, sensitive to) Organizing (relates beliefs, balances) Internalizing (acts, shows, practices) Bloom’s (affective)

12. Cite, count, define, draw, List, name, record define

13. Key Question Where in the curriculum are students encouraged to develop the program outcomes (knowledge, skills & values)?

14. Course Progression Maps Show student progression through curriculum to explore curriculum flow, balance of required core courses, electives, etc. Data collected from: ► Calendars ► Course outlines ► Curriculum committees

15. Course Progression Maps 15

16. Program Outcomes Map Goal: To match program outcomes with individual courses that intentionally foster the development of selected outcomes. Data collected from: ► Faculty ► Course outlines ► Curriculum committees

17. 17

18. 18

19. 19

20. 20

21. Course Survey: Instructional methods used Assessment approaches for students Which Knowledge, Skill & Value, if any, is intentionally fostered? At what level? How is each KSV taught and/or assessed? How are the marks distributed over the semester?

22. Research Methods All Instructional Approaches 22

23. Outcome: Research Methods 23

24. Written Literacy Numeracy Civic Engagement 24

25. Grades Allocated by Week 25

26. Key Question How do we know whether students have developed the knowledge, skills & values we intend?

27. Level 1 Reaction Level 2 Learning Kirkpatrick’s Four Levels of Evaluation Level 4 Results Level 3 Behavior Kirkpatrick, D. (1994). Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler

28. Level 1 Reaction Kirkpatrick’s Four Levels of Evaluation Key Question: What was the student reaction to learning environment? Timing: Usually done immediately or soon after the learning event(s)

29. Level 2 – Learning Kirkpatrick’s Four Levels of Evaluation Kirkpatrick, D. (1994). Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler Key Question: Did students achieve desired learning objective(s)? Timing: Usually done immediately or soon after learning

30. Kirkpatrick’s Four Levels of Evaluation Kirkpatrick, D. (1994). Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler Level 3 Behavior Key Question: Are newly acquired skills, knowledge, or attitude used by learner after learning event is completed? Timing: Usually done 1-3 months after learning

31. Kirkpatrick’s Four Levels of Evaluation Level 4 - Results Key Question: Did students achieve desired outcomes of program of study? Timing: Usually done 3 months – 2 years after learning experience

32. ParticipantsActivityHow oftenQuestions Current Students Focus Groups Annual What knowledge, skills and values do you think are most important to graduates of the programme? (compare with the current outcomes) Describe your most enjoyable learning experiences at Guelph to date. Comments on other aspects of your Guelph experience (e.g. awards, academic support)? What would you change about the curriculum? Please suggest changes to help us improve the program: what would you add/drop from the curriculum? Other changes? What advice would you give to an incoming student? Alumni/Employ ers Focus GroupsBi-annual  What knowledge, skills and values do you look for when hiring?  How well do our graduates meet those KSAs  What kinds of work would someone with an undergraduate degree be doing in your organization?  What advise would you give a student coming into this program regarding their educational options?  Where do you see new employees in 5 years? What are their opportunities for advancement? … Assessment Plan Sample

33. ParticipantsActivityWhenQuestions Secondary Documents Previous Review, Other programs 5 years  To provide an overview of “where we are” in relation to experiential insights gained from faculty and students. FacultyWeb survey followed by faculty retreat Every 2 years  Web survey – Solicit feedback on current programme outcomes by faculty  Retreat – review results of survey  Based on faculty perspectives, develop a list of programme (a) strengths (b) weaknesses (c) opportunities (d) limitations?  Compare faculty SWOT with the other stakeholder SWOT  Develop an action plan Assessment Plan Sample (cont’d)

34. Assessing Outcomes - Rubrics http://www.uoguelph.ca/vpacademic/avpa/outcomes/2012UniversityofGuelphLearningOutcomes_SenateDecember2012%28FINAL%29.pdf

35. Change… People Culture Shift Hiring Recognition & Rewards Infrastructure Teaching Space Scheduling Tools Curriculum Courses Assessment Methods

36. Data-informed Data-informed Faculty-driven Faculty-driven Student-engaged Student-engaged Stakeholder-informed Stakeholder-informed Culture- & context-specific Culture- & context-specific Rigorous & authentic Rigorous & authentic Continuous improvement of sustainable practices Continuous improvement of sustainable practices Resourced & recognized Resourced & recognized Recommended Practices A Guide to Developing and Assessing Learning Outcomes at the University of Guelph - http://www.uoguelph.ca/vpacademic/avpa/pdf/LearningOutcomes.pdf

37. Refining the blend of people, processes & resources Refining the blend of people, processes & resources Transition requires investment Transition requires investment Keep momentum Keep momentum Continuous improvement is at least as important as ‘reporting’ Continuous improvement is at least as important as ‘reporting’ Mistakes will happen Mistakes will happen Keep in Sight