1. 1 National Student Survey and the Student Learning Experience Michael Prosser Higher Education Academy

2. 2 Academy Mission The Academy’s mission is to help institutions, discipline groups and all staff to provide the best possible learning experience for their students.

3. 3 Presentation based upon: Over 20 years research into the student learning experience in higher education in the United Kingdom, Australia, Sweden and Hong Kong Experience in Australia with the Course Experience (Perceptions) Questionnaire The interpretation of the results of student surveys is not value or theory free: Interpretations in terms of student satisfaction and rankings Interpretations in terms of student perceptions relating to student learning

4. 4 Overview of the student learning perspective Figure 1: Model of Student Learning

5. 5 STUDENT APPROACHES TO LEARNING Surface Approach Intention to reproduce - rote memorise information needed for assessment - failure to distinguish principles from examples - treat tasks as external impositions - focus on discrete elements without integration Deep Approach Intention to understand - meaningfully memorise information for later use - relate new ideas to previous knowledge - relate concepts to everyday experiences - relate evidence to conclusions

6. 6 Approach to learning represents the relationship between the student and the task or subject Approaches vary between subjects Approaches vary between tasks within subjects Approaches not stable but variable Surface and deep approaches describe a fundamental variation in the way students approach learning – they do not describe the rich detail in the way students approach their learning

7. 7 STUDENTS’ PERCEPTIONS OF THE LEARNING CONTEXT Research by Entwistle and Ramsden (1983) using interviews and questionnaires identified a number of student perceptions relating to the way they approached their studies Student perceptions of: Quality of teaching – including quality of feedback (NSS: Teaching, Assessment and Feedback, Academic Support)) Clearness of goals of course and standards of assessment (NSS: Assessment and feedback) Workload so high that it was not possible to understand everything Assessment measuring reproduction and not understanding were found to relate to how they approach their studies and to learning outcomes (exam results and other indicators)

8. 8 RELATIONSHIP BETWEEN PERCEPTIONS AND APPROACHES Study of 8837 students in 51 first year subjects around Australia Amongst the data collected were students responses to: contextualised Ramsden's Course Experience Questionnaire and a contextualised Biggs Study Process Questionnaire 1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS

9. 9 Approaches to Study Surface Approach 32. Although I generally remember facts and details, I find it difficult to fit them together into an overall picture 35. The best way for me to understand what technical terms mean is to remember the textbook definitions Deep Approach 28. I try to relate ideas in this subject to those in other subjects, wherever possible 34. In trying to understand new ideas, I often try to relate them to real life situations to which they might apply. Biggs Study Process Questionnaire

10. 10 Student Perceptions Of Learning Context Good Teaching 15. The staff made a real effort to understand difficulties students might be having with their work. Clear Goals and Standards 1. It was always easy to know the standard of work expected 6. I usually had a clear idea of where I was going and what was expected of me in this subject. Appropriate Workload 25. The sheer volume of work in this subject meant that it couldn't all be thoroughly comprehended (-). Appropriate Assessment 8. To do well in this subject, all you really need is a good memory (-). Course Experience Questionnaire

11. 11 Analysis conducted using: 1.Individual student as unit of analysis across all first year subjects 2.Individual student as unit of analysis within first year subjects subjects 3.First year subjects as unit of analysis

12. 12 Factor Analysis of Perceptions of T & L Context and Approach to Study - individual _________________________________________________________________ ScaleFactors ____________________________ 12 _________________________________________________________________ Perceptions of Context Good teaching.80 Clear Goals and Standards.67 Appropriate Workload-.69 Appropriate Assessment-.65 Approach to Study Surface Approach.81 Deep Approach.73 _________________________________________________________________ Principal Components, Varimax Rotation, n=8837

13. 13 At individual student level: 1. A deep approach is associated with perceptions that the teaching is good and the goals and standards are clear (NSS: Teaching, Assessment and Feedback) 2. A surface approach is associated with perceptions that the workload is too high and assessment tests reproduction That is, variation in individual students’ perceptions of the learning context across all subjects is associated with the approaches to study (and learning outcome) 1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS

14. 14 Factor Analysis of Perceptions of T & L Context and Approach to Study – within first year medical subjects in different universities _________________________________________________________________ ScaleSubject ASubject BSubject C ________________________ 121212 _________________________________________________________________ Perceptions of Context Good teaching.84.77.79 Clear Goals and Standards.68.70.69 Appropriate Workload-.73-.68-.69 Appropriate Assessment-.70-.60-.60 Approach to Study Surface Approach.75.84.80 Deep Approach.62.64.72 __________________________________________________________________

15. 15 Factor Analysis of Perceptions of T & L Context and Approach to Study – individual – first year English _________________________________________________________________ ScaleFactors ____________________________ 12 _________________________________________________________________ Perceptions of Context Good teaching.85 Clear Goals and Standards.76 Appropriate Workload-.69 Appropriate Assessment-.69 Approach to Study Surface Approach.84 Deep Approach.78 _________________________________________________________________ Principal Components, Varimax Rotation, n=119

16. 16 Within each subject: 1. A deep approach is associated with perceptions that the teaching is good and the goals and standards are clear (NSS: Teaching, Assessment and Feedback) 2. A surface approach is associated with perceptions that the workload is too high and assessment tests reproduction That is, variation in students’ perceptions of the learning context within subjects is associated with the approaches to study within subjects – within subject variation in perception is not measurement error. 1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS

17. 17 Factor Analysis of Perceptions of T & L Context and Approach to Study - subject _________________________________________________________________ ScaleFactors __________________________ 12 _________________________________________________________________ Perceptions of Context Good teaching.81-.34 Clear Goals and Standards.78 Appropriate Workload-.75 Appropriate Assessment-.87 Approach to Study Surface Approach.91 Deep Approach.82 _________________________________________________________________ Principal Components, Variamax Rotation, n=51

18. 18 At subject level: 1. A deep approach is associated with perceptions that the teaching is good and the goals and standards are clear (NSS: Teaching, Assessment and Feedback) 2. A surface approach is associated with perceptions that the teaching is not so good, workload is too high and assessment tests reproduction That is, variation in students’ perceptions of the learning context between subjects is associated with the approaches to study 1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS

19. 19 Example for student learning in first year mathematics In a study of first year university mathematics students in a large research intensive university, analyses of short open ended written statements by 236 students identified: Conceptions of Mathematics: Fragmented Conceptions A.Maths as numbers, rules and formulae B.Maths as numbers etc with applications to problems Cohesive Conceptions C. Maths as a way of thinking D. Maths as a way of thinking for complex problem solving E. Maths provides insights for understanding the world

20. 20 Fragmented Conceptions A.Maths as numbers, rules and formulae Maths is the study of numbers, and the application of various methods to changing numbers B.Maths as numbers etc with applications to problems Maths is the study of numbers and their applications in other subjects and the physical world

21. 21 Cohesive Conceptions C.Maths as a way of thinking Maths is the study of logic. Numbers and symbols are used to study life in a systematic perspective and requires the mind to think in a logical and often precise manner D.Maths as a way of thinking for complex problem solving Maths is an abstract reasoning process which can be utilized to explore and solve problems E.Maths provides insights for understanding the world Techniques for thinking about observable, physical phenomena in a quantitative way and also for thinking more abstractly with little or no relation to the directly observable universe.

22. 22 Approaches to Studying Mathematics: Surface approach – focus on reproducing A.Learning by rote memorisation – to reproduce knowledge and procedures B.Learning by doing lots of examples – to reproduce knowledge and procedures Deep approach – focus on understanding C.Learning by doing lots of examples – to understand theory and concepts D.Learning by doing difficult problems – to understand and to relate to other knowledge E.Learning by doing difficult problems and studying theory to look for situations that theory may apply.

23. 23 Approaches to Studying Mathematics: Surface approach – focus on reproducing A.Learning by rote memorisation – to reproduce knowledge and procedures I liked calculus because I could remember formulas which is how I used to study. I would rote learn all the formulas and summarize all my theoretical notes B.Learning by doing lots of examples – to reproduce knowledge and procedures The way I go about studying for mathematics is by doing lots of examples and questions. Firstly I would study the notes and learn the formulas, then I put all of that to use by doing heaps of examples

24. 24 Deep approach – focus on understanding D.Learning by doing difficult problems – to understand and to relate to other knowledge After listening to explanation of how a particular maths works the most essential features a repetition to develop speed (this usually consists of boring menial tasks) and an equal component of very difficult problems which require a great deal of thought to explore that area and its various properties and their consequences E.Learning by doing difficult problems and studying theory to look for situations that theory may apply. Read the relevant theory and try to get on the same “wavelength” as the person who actually discovered it. Before I attempt any problems I try to think where you can use the concept: i.e. what the concept was invented for. Then I attempt problems (on my own).

25. 25 Relationship between Conception of Mathematics and Approach to Studying Mathematics ____________________________________________________________ ConceptionApproachTotal _________________ SurfaceDeep ____________________________________________________________ Fragmented (A & B)17917196 Cohesive (C, D, E)436 40 Total18353236 ____________________________________________________________ Chi-square=126, p<.001 Phi=.57 ES=2.1 Crawford, K., Gordon, S., Nicholas, J. and Prosser, M. (1998) Qualitatively different experiences of learning mathematics at university. Learning and Instruction, 8, 455-468.

26. 26 Defining Items from the Conceptions of Mathematics Questionnaire Scale ItemItem Fragmented Conception Cohesive Conception Mathematics is figuring out problems involving numbers Mathematics is a logical system which helps explain the things around us

27. 27 Cluster Analysis of Prior Experiences and Understandings, Perceptions and Approaches and Post Experiences and Understandings Subscales

28. 28 ___________________________________________________________________________ VariablesMean SurfaceMean Deep (n=147)(n=127) ___________________________________________________________________ Prior Experiences and Understandings Prior Fragmented Conception of Mathematics 0.31-0.46 Prior Cohesive Conception of Mathematics-0.39 0.45 Prior Surface Orientation 0.34-0.44 Prior Deep Orientation-0.56 0.59 Prior Academic Ranking (TER)-0.29 0.35 Perceptions and Approaches Good Teaching-0.37 0.37 Clear Goals and Standards-0.23 0.24 Appropriate Workload-0.30 0.32 Appropriate Assessment-0.12 0.12 Surface Approach 0.44-0.45 Deep Approach-0.54 0.57 Post Experiences and Understanding Post Fragmented Conception 0.34-0.41 Post Cohesive Conception-0.27 0.27 Achievement (Final Mark in Mathematics)-0.34 0.40 ___________________________________________________________________

29. 29 Conclusions The Table shows clear relations between prior experiences, perceptions of teaching and approaches and post experience and understanding. The analysis identified 2 groups of students In particular, those students: with more fragmented and less cohesive conceptions on entry, describe having adopted more of a surface approach to studying at high school and less of a deep approach, perceive the teaching to be poorer, to adopt more of a surface and less of a deep approach to studying at university, and have more fragmented and less cohesive conceptions after studying and poorer achievement The opposite was found for the other group

30. 30 Overview of the student learning perspective Figure 1: Model of Student Learning

31. 31 Substantial amount of research linking results on surveys such as the NSS with student approaches to study and learning outcomes Student perceptions are a function of both their prior experiences and understandings and the course design and teaching Do not improve student satisfaction by focusing on satisfaction - the individual items or scales – need to better understand why they responded the way they Spread of results – proportions responding in certain ways – better than mean of responses Substantial disciplinary variation – unreasonable to compare between disciplines Can only expect small effect sizes in changes in scores over time -.2 of a SD – if over 3 - 5 years changes of the order of.1 to.2 points (3.5 to 3.6 or 3.7).

32. 32 Focus on the student perception rather than satisfaction – better understand their perceptions – why they respond the way they have Comprehensive and aligned set of student evaluation instrument designed and interpreted in terms of student perceptions Institutions quality assurance and academic development and CPD activities aligned with the underlying model Treat data as indicators requiring further investigation – focus groups, open ended responses etc

33. 33 Academy Response 1.Conferences relating to student experiences 2.Workshops and seminars to interested HEI’S and groups of staff 3.Work with interested institutions on ways to improve their students’ experiences as indicated by the NSS 4.Suggesting ways of further developing and / or improving future surveys 1.Review of dissemination site 2.Development of a bank of optional items, a selection of which individual institutions may wish to add to their NSS survey 3.Further analysis of the 2005 / 2006 data

34. 34 Variable __________________________________________________________________ _ TASORDOS Teaching1.48 (.39).66 (.66).55 (.45)-.15 (-.01).60 (.50).82 (.80) Assessment1.57 (.55).42 (.34).03 (-.06).43 (.32).51 (.37) Support1.62 (.62).14 (.21).67 (.66).79 (.77) Organisation1.29 (.14).34 (.31).82 (.77) Resources1.06 (.04).21 (.17) Development1.65 (.55) OS1 Correlation matrix of NSS – institutional – 2006 (2005) n (2005) = 129; n (2006) = 129

35. 35 2006 Variable2005 Variable _______________________________________________________________ TASORDOS Teaching.77 Assessment.85 Support.81 Organisation.85 Resources.84 Development.69 OS.79 Correlation matrix of NSS – 2005 with 2006– institution

36. 36 Mean changeSD changeEffect size changeStat Sig Teaching-.005.095.05ns Assessment.017.110.16ns Support.031.103.30P<.005 Organisation.014.136.10ns Resources.051.141.36P<.001 Development.001.097.01ns OS-.026.123.21P<.05 Change scores of NSS – 2005 with 2006– institution

37. 37 Variable _______________________________________________________ TASORDOS Teaching1.49.57.45.33.48.65 Assessment1.54.49.33.39.51 Support1.54.36.50.60 Organisation1.38.41.57 Resources1.41.42 Development1.59 OS1 Correlation matrix of NSS – Aston – 2005 – individual student n = 987

38. 38 Processes of Undergraduate Essay writing Study of conceptions, approaches and outcomes of a particular essay in a first year sociology course Interviews and analyses of essays for 19 students Analysis of Interviews – phenomenographic Analyses of essays – systemic linguistics and examiners marks

39. 39 CONCEPTIONS OF THE ESSAY Multi structural conception of the essay - conceived of as a series of related points about the topic - each point related to topic - points related to one another, after being listed - facts and quotes used to support pre-determined opinions about the topic.... but I think generally just a basic idea of what you feel and a list of your points of what you are trying to explain and continue on what you are feeling and link all your ideas together in one essay.

40. 40 Relational conceptions of an essay - conceived of an argument - issues included because coherent part of argument, not just because they relate to question - reading for essay is way of forming view, not just collecting facts and quotes Well, I mean the standard thing is to argue about, about something which means um.. to try to prove something or disprove another thing and to show the pro and cons... actually create an argument for this particular question.. understand something a lot better, a lot deeper which then you can link up with other thing..... so I had to read a lot first, and through my reading I decided that, yes it did make that huge contribution to our understanding of society.. to systematically logically present all the evidence for what I was trying to show. (.... but I think generally just a basic idea of what you feel and a list of your points of what you are trying to explain and continue on what you are feeling and link all your ideas together in one essay. )

41. 41 APPROACHES TO WRITING THE ESSAY Surface approach to writing the essay Students who conceived of the essay as multistructural had an intention to reproduce a series of points in a coherent way and in reading focussed on listing key points related to the essay topic, and in writing focussed on producing a coherent set of points in written form I underline the main words like culture and society..... I have so many books I was just reading and underlining and taking quotes and that from here and there. And writing them on a piece of paper - the main points..... And then when I had all these points down I went and looked for points in my reading, in all the books and that and tried to relate them, and that's why I think I had problems because I wanted to relate everything and it wouldn't relate.

42. 42 Deep approach to writing essay Students who conceived of the essay as relational had an intention to present an argument in support of a particular point of view on the question and in reading for the essay focused on relating each author's meaning to the question and to their own previous viewpoint, and in writing the essay focussed on the coherence of the overall argument in favour of a particular viewpoint. Once I compiled, I think I used about five, six or seven references and written notes on each - probably five or six pages on each - I summarised the main arguments, I'd just write that at the top of the note page so I know when I can look at an author I can see basically what they're saying.... Yes I sit down and I take notes from the book, and then I read through my notes and I say well what's he basically saying, then I write down what he's saying - about 5 or 6 lines. I start with my introduction and I think I always find that helpful to give me some, to give me direction really.

43. 43 Assessment marks for essays: 13 surface approach with a multistructural conception (5,6,4,4,6,6,6,6,6,6,6,7,8) 6deep approach with a relational conception(9,8,8,9,5,8) Conclusion Those students who had a conception of an assay as parts needed to be related (multistructural), approached the writing in terms of accumulating bits of information (surface), and received poor marks. Those students who had a conception of an essay as a whole argument (relational), approached the writing in terms of understanding different perspectives (deep) and received high marks.