Graduate Learning Outcomes: The challenge of assessment Serge Desmarais University of Guelph Learning Outcomes Conference October 17, 2014

Fostering a learning outcomes culture  U of G established 10 graduate learning objectives in 1987:  Inspirational and lofty  “desired characteristics of graduates”  Used to guide new grad and undergrad course and new program proposals  Since 2005, we re-engaged the community in an effort to develop a learning outcomes culture  2005: Provost’s White Paper  : 21 st Century Curriculum Committee  2008: UUDLEs & GDLES  2010: IQAPs at U of G & across the province

Fostering the Culture  First strategic steps  Introduced “Outcomes” as a priority for governance:  First addressed at Board of Undergraduate Studies and Program Committees  Following by introduction of same strategy at Board of Graduate Studies  Created a new Senate Committee:  Quality Assurance – new Senate committee; assist departments in meeting requirements under IQAP  Engaged Associate Deans with program-level responsibility in the discussion  Increased staff support: more teaching & learning staff

University Learning Outcomes  Two year process of broad consultation with faculty, staff and students  New learning outcomes combine: U of G’s learning objectives (1987) LEAP + Bases of Competence (Evers, Rush & Bedrow) +

5 University-wide Outcomes approved by Senate for both undergraduate (2012) and graduate (2013) programs:  Critical and Creative Thinking  Literacy  Global Understanding  Communicating  Professional and Ethical Behaviour University Learning Outcomes

Critical and Creative Thinking IntroduceReinforceMastery Inquiry and Analysis Problem Solving Creativity Depth and Breadth of Understanding... one applies logical principles... to solve problems with a high degree of innovation, divergent thinking and risk taking.... show evidence of integrating knowledge across disciplinary boundaries. Depth and breadth of understanding of disciplines essential to this outcome.

Critical and Creative Thinking IntroduceReinforceMastery Inquiry and Analysis Problem Solving Creativity Depth and Breadth of Understanding x  Depth and Breadth of Understanding  Level 3: Compares the merits of alternate hypotheses in many different disciplines. Demonstrates mastery of a body of knowledge and critically evaluates the limits of their own knowledge and how these limits influence analyses.

Assessing L.O.s in Undergrad programs  Collaboration with Desire2Learn to develop the assessment tools within our current LMS  Our eventual goal: an assessment method that will be used by all departments and programs  The initial plan engaged two programs:  Bachelor of Arts and Science  Bachelor of Engineering  Now working with the full B. Eng program and beginning process with DVM (Veterinary Medicine)

Lessons learned… so far  Software development is not the most complex aspect of the project  Decisions re: which outcomes to assess at what point in the curriculum progression is complex  Long term, challenge is to gain full departmental and faculty engagement:  Much resistance, paranoia, and hope for continued status quo  Identify your champions and the most engaged programs, and leverage their enthusiasm and persuasiveness

How to proceed?  A “one size fits all” model is not the right approach  Must respect disciplinary differences  Efforts should be programmatic, not only course by course  Graduate programs must have a clear sense of when and where specific learning outcomes will be addressed and assessed  Curriculum Mapping  Assessment must be documented and evidentiary based

So… how do we assess in grad programs?  Moving to a culture of evidence-based continuous improvement will be hard work  Requires a re-envisioning of the entire graduate program process  Requires a commitment and broad agreement on departmental and program strategy  Let’s work on a few examples….

Communicating Rubric Introduce 1 Reinforce 2 Master 3 Oral Communication Includes interpersonal skills and oral speaking Demonstrates the ability to present information in a comprehensive manner, clearly, and effectively. Demonstrates oral communication skills that are organized, and presented in a creative and interesting manner. The student speaks in a clear, loud voice. Demonstrates the ability to present information in that the receiving party can easily understand the information. The speaker is clear and shows confidence in public speaker. Has mastered oral communication skills. Written Communication The ability to express one’s ideas through writing The ability to write clearly, and demonstrate a general knowledge about describing an idea. Has the ability to write a clear message with good vocabulary and little grammatical, spelling or functional errors. The student has shown a breadth of vocabulary and can write in a sophisticated manner that clearly conveys the message of the speaker. Grammar, spelling, and functional errors are almost non- existent. Reading Comprehension The understanding of writing The student develops the key components in reading at their age and grade level The ability to read a wide vocabulary, and understand more sophisticated writing is evident. The ability to read and extract information from the text has developed. The student can answer questions, and generate ideas from writing. Listening Involves being attentive when others are speaking, and responding effectively to others’ comments during a conversation The ability to pay attention when a speaker is talking. The receiver shows evidence they are intently listening, occasionally asking questions. The receiver not only demonstrates bodily gestures indicating listening, but also engages with the speaker, including asking questions, and reiterating points the speaker states.

Information Literacy Rubric Introduce 1 Reinforce 2 Master 3 Collaborative Literacy The ability to know where there is a need for information Is able to locate and use materials from a variety of resources. Can not only locate materials, but also is able to understand when more information is needed and how to evaluate its effectiveness. This skill is developed where one can easily identify, locate, evaluate many resources and effectively and responsibly use the information discover new information. Quantitative Literacy Includes numeracy, and a comfort in working with numerical data Understands the need for evidence and use of numerical data, and is able to generally interpret this data. Is able to use quantitative data to one’s advantage by using it as evidence for a claim. Is comfortable with everyday situations with numeracy. Possesses the ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations, and from this, can develop sophisticated arguments supported by quantitative evidence. Is also able to create and communicate numerical data. Computing The ability to use technology and programs Uses technology to research and support communication of knowledge. Is able to use computers and technology to compliment knowledge and understanding, and has a large skill set in helpful programs. Has a breadth of knowledge in computer and technology skills which one uses to enhance presentations and knowledge. Uses technology in a sense that does not overshadow knowledge, but rather compliments it. Depth and Breadth of Understanding An ability to break disciplinary boundaries and bring together information Realizes the differences of information from a variety of disciplines. Is able to extract information, and understands the advantages of using a variety of disciplines. Is able to understand a variety of expressions from different disciplines, and can use information to ones advantage, i.e. many supporting perspectives on a claim enhances the reliability and objectivity.

Thank You Serge Desmarais Acting Provost and Vice-President (Academic)