Project Evaluation: Measuring Learning Impact Diane Salter, Vice Provost Teaching and Learning

Why measure ‘learning impact’? To determine whether a pedagogical intervention has influenced student learning as intended To monitor progress and change to enhance student learning To provide empirical data to support the pedagogical viability of the learning/teaching method

When might this be important? As part of a scholarly approach to teaching – ongoing revision and course/ programme review Projects/grants applications and reporting to show: Does this project/instructional strategy promote active learning-centred teaching? Does this project/instructional strategy lead to significant improvements in quality or outcomes of student learning? How might the approach to this subject be improved?

To measure impact we need to know: How do we measure impact on learning? What are the objectives/expected outcomes of the intervention? What are the indicators of success? How can these be observed/measured? What is the baseline (what was happening before the intervention) against which changes can be judged?

Measuring impact Level 1 – Reaction Level 2 – Knowledge and Skills Gained Level 3 – Behaviour Level 4 – Performance Competence *

Measuring impact Reaction – How did it feel ? Did they like it? Do they think it made a difference? Knowledge and Skills Gained – What can they do that they couldn’t do before, or do better than they could do before? Behaviour – What do they do now that they didn’t do before? Performance competence – Has their performance improved as a result of increased learning?

Examples of indicators Student Feedback Perceived usefulness and benefits Self report on motivation, engagement or study approaches Self-reported learning gains Instructor Feedback Observed changes in students’ motivation, engagement and study behaviour Teachers’ reflections, etc. Classroom Observations Nature/amount of classroom interactions Students’ engagement and participation Other classroom study behaviours Student performance Assessment grade/marks Quality of student work Performance in practice settings

How do we know our course design works? What was the effect of an experience the learners? (learning impact): Knowledge (e.g. know what, know how and know why) Skills (e.g. cognitive, meta-cognitive, social and psychomotor) Attitude (e.g. beliefs, disposition and values)

What’s in a name?  Scholarly Teaching  Scholarship of Teaching 9

Scholarly Teaching Evidence based approaches to teaching and course design based on best practice. 10

Scholarship of Teaching  Sharing  Presentations  Publications  Reflection  Revision 11

What is the difference? Research for scholarship and publication Ethical approval required Research/feedback for the purposes of improvement of studies and programs. See Article 2.5 Tri-council agreement 12

Tri-council policy statement The Tri-council policy statement on Ethical Conduct for Research Involving Humans guides research done at university and provides clear direction regarding doing research/gathering feedback for the purposes of improvement of studies and programs. See article 2.5 (next slide) 13

Tri-council policy statement Article 2.5 states: Quality assurance and quality improvement studies, program evaluation activities, and performance reviews, or testing within normal educational requirements when used exclusively for assessment, management or improvement purposes, do not constitute research for the purposes of this Policy, and do not fall within the scope of the REB. 14

More information needed? If you would like a seminar in your department about SoTL and measuring learning impact please contact: 15