1. Student Feedback on, and Evaluation of, their Learning Experience Michael Prosser Institute for Teaching and Learning The University of Sydney

2. 2 INTRODUCTION Some distinctions in the way we talk about and describe students learning experiences in higher education A.Consumer and client centred views of student learning experience - ratings of student satisfaction with teaching and services B.Learner centred views of student learning experience - indicators of student perceptions and experiences relate to learning approaches and outcomes In this presentation I wish to focus on the experience of students as learners – learner centred views

3. 3 A.Design of learning experiences: -focus on the way student experiences are designed by the teacher B.Student experiences of the design of student learning experiences: - focus on the way the student experiences the design

4. 4 It is not (just) the way we design our courses and programs in higher education that relates to the quality of student learning but how our students experience and understand those designs. Finding out, understanding and responding to the way students experience our courses and programs – the student experience of our courses and programs Working towards aligning the way our student experience our courses and programs with the way we design those courses and programs

5. 5 OVERVIEW OF THE STUDENT LEARNING PERSPECTIVE Figure 1: Model of Student Learning

6. 6 Student Approaches to Learning A key variation represented by: Surface Approach: Intention to reproduce rote memorise information needed for assessment failure to distinguish principles from examples treat tasks as external impositions focus on discrete elements without integration Deep Approach: Intention to understand meaningfully memorise information for later use relate new ideas to previous knowledge relate concepts to everyday experiences relate evidence to conclusions

7. 7 Students Experiences of The Learning Context Student experiences of: Quality of teaching – including quality of feedback Clearness of course goals and standards Workload - too high to understand Assessment - reproduction and not understanding are key aspects of student experiences found to relate to how they approaches their studies and to their learning outcomes Entwistle and Ramsden, 1983

8. 8 EXAMPLE FROM MEDICINE Quantitative Survey Study of over 8000 students in first year subjects around Australia, including 618 first year medical students in three different first year courses Amongst the data collected were students responses to: Ramsden's Course Experience Questionnaire Biggs Study Process Questionnaire 1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS

9. 9 Approaches to Study Surface Approach 32. Although I generally remember facts and details, I find it difficult to fit them together into an overall picture Deep Approach 28. I try to relate ideas in this subject to those in other subjects, wherever possible Biggs Study Process Questionnaire

10. 10 Student Experiences Of Learning Environment Good Teaching 15. The staff made a real effort to understand difficulties students might be having with their work. Clear Goals and Standards 1. It was always easy to know the standard of work expected 6. I usually had a clear idea of where I was going and what was expected of me in this subject. Appropriate Workload 25. The sheer volume of work in this subject meant that it couldn't all be thoroughly comprehended (-). Appropriate Assessment 8. To do well in this subject, all you really need is a good memory (-). Paul Ramsdens Course Experience Questionnaire

11. 11 Analyses showed that within each course: 1. A deep approach is associated with experiences that the teaching is good and the goals and standards are clear 2. A surface approach is associated with experiences that the workload is too high and assessment tests reproduction That is, variation in students experiences of the learning environment within subjects is associated with the approaches to study within subjects – within subject variation in experience is not measurement error.

12. 12 Factor Analysis of Experiences of Learning Context and Approach to Study ________________________________________________ ScaleSubject ASubject BSubject C FactorsFactorsFactors ________________________ 121212 ________________________________________________ Experiences Good teaching.84.77.79 Clear Goals & Stan.68.70.69 App. Workload-.73-.68-.69 App. Assessment-.70-.60-.60 Approaches Deep Approach.75.84.80 Surface Approach.62.64.72 ________________________________________________

13. 13 EXAMPLE FROM MATHEMATICS Qualitative Survey First year university mathematics students, large research intensive university, analyses of short open ended written statements by students identified: Key differences in the experiences of Mathematics: Fragmented Experience A.Maths as numbers, rules and formulae B.Maths as numbers etc with applications to problems Cohesive Experience C.Maths as a way of thinking D.Maths as a way of thinking for complex problem solving E.Maths provides insights for understanding the world

14. 14 Experience of Maths Quote A: NumbersMaths is the study of numbers and the application of various methods of changing numbers C: Logical system Maths is the study of logic. Numbers and symbols used to study life in a systematic perspective and requires the mind to think in a logical and precise way E: Complex / World Techniques for thinking about observable, physical phenomena in a quantitative way and also for thinking more abstractly with little or no relation to the directly observable universe

15. 15 Approach to Study of Maths Quote A: Learning by rote memorisation, with an intention to reproduce (surface) I liked calculus because I could remember formulas which is how I used to study. I would rote learn all the formulas and summarise all my theoretical notes E: Learning with an intention of gaining a coherent understanding of the theory and looking for situations where the theory will apply (deep) Read the relevant theory and try to get on the same wavelength as the person who actually discovered it. Before I attempt any problems I try to think where you can use the concept: i.e. what the concept was invented for. Then I attempt problems (on my own)

16. 16 Relationship between Experience of Mathematics and Approach to Studying Mathematics ______________________________________________ ExperienceApproach Total _______________ SurfaceDeep (A & B)(C, D & E) ______________________________________________ Fragmented (A & B)179 (91%)17 (9%)196 Cohesive (C, D, E) 4 (10%)36 (90%) 40 Total18353236 ______________________________________________ Chi-square=126, p<.001; Phi=.57; ES=2.1 Crawford, K., Gordon, S., Nicholas, J. and Prosser, M. (1998) Qualitatively different experiences of learning mathematics at university. Learning and Instruction, 8, 455-468.

17. 17 Scale ItemItem Fragmented Experience Cohesive Experience 9. Mathematics is figuring out problems involving numbers 17. Mathematics is a logical system which helps explain the things around us Defining Items from the Experiences of Mathematics Questionnaire

18. 18 Cluster Analysis of Experiences and Understandings _____________________________________________________________ VariablesCluster 1Cluster 2 (n=147)(n=127) _____________________________________________________________ Prior Experiences and Understandings Fragmented experience of Maths 0.31-0.46 Cohesive experience of Maths-0.39 0.45 Prior Surface Orientation 0.34-0.44 Deep Orientation-0.56 0.59 Academic Ranking (TER)-0.29 0.35 Experiences and Approaches Good Teaching-0.37 0.37 Clear Goals and Standards-0.23 0.24 Appropriate Workload-0.30 0.32 Appropriate Assessment-0.12 0.12 Surface Approach 0.44-0.45 Deep Approach-0.54 0.57 Post Experiences and Understanding Fragmented experience of Maths 0.34-0.41 Cohesive experience of Maths-0.27 0.27 Achievement (Final Mark in Maths)-0.34 0.40 ___________________________________________________

19. Cluster Analysis of Variables (medium to large differences) _____________________________________________________________ VariablesCluster 1Cluster 2 (n=147)(n=127) _____________________________________________________________ Prior Experiences and Understandings Fragmented experience of Maths HighLow Cohesive experience of MathsLowHigh Prior Surface OrientationHighLow Deep OrientationLowHigh Experiences and Approaches Good TeachingLowHigh Clear Goals and StandardsSmall difference Appropriate WorkloadLowHigh Appropriate AssessmentSmall difference Surface ApproachHighLow Deep ApproachLowHigh Post Experiences and Understanding Fragmented experience of MathsHighLow Cohesive experience of MathsLowHigh Achievement (Final Mark in Maths)LowHigh ___________________________________________________

20. 20 EXAMPLE FROM OXFORD UNIVERSITY (Honours: n=428, Approach: n=2320) _________________________________________________ ExperienceOutcomeApproach Honours_________________ (1,2a,2b)SurfaceDeep (+ve) ________________________________________________ Good teachingLargeLowHigh Clear Goals and StandardsLargeLowHigh Appropriate AssessmentMediumLowHigh Appropriate WorkloadMediumLowHigh _________________________________________________ http://www.learning.ox.ac.uk/iaul/pdf/OLCPFinal.pdf

21. 21 In summary: Relationships between students experiences of their course, their approaches to study and their learning outcomes View of student learning experiences: learner centred (learning) cf consumer / client centred (satisfaction) student experience of learning designs cf design of learning experiences

22. 22 EXAMPLE FROM THE UNIVERSITY OF SYDNEY Graduates: Graduate Course Experience Questionnaire - Program population survey of all coursework graduates Postgraduate Research Experience Questionnaire - Program population survey of all research graduates

23. 23 Currently enrolled students: Student Course Experience Questionnaire – whole year stratified random sample of all enrolled coursework students Student Research Experience Questionnaire – whole year population survey of all enrolled research students Unit of Study Evaluation - course compulsory triennial survey of all units of study

24. 24 Supplemented and supported by: Focus group discussions to further explore issues identified in surveys Academic Board Reviews of Faculties interview groups of postgraduate and undergraduate students to better understand their experiences Performance based funding model for teaching to reward Faculties enhancing the student experience Faculty Teaching and Learning Plans with a focus on Student Experience

25. 25 IMPLICATIONS FOR EVALUATING STUDENT LEARNING EXPERIENCES 1. Try to understand how and why student experience their courses and programs they way they do – quantitative survey, open-ended responses and follow up interview and focus groups 2. In responding, try to respond to why, not how, they experience their courses and programs the way they do – eg workload, assessment 3. Implement institutional policies, procedures and strategies that recognise and reward teachers, schools and departments and faculties efforts to respond to and enhance students learning experiences