The Impact of Peer Learning on Assessment Literacy and Feedback Orientation Neil Ford Senior Teaching Fellow (Education Development) Southampton Business School 6th June 2019 International Peer Learning Forum, Brighton

Context Rationale Aims Approx. 250 students across 3 programmes Peer Learning since 2016 – based on PAL/PASS Evolved from previous buddy scheme Good attendance and student feedback Good engagement from Peer Leaders High first year failure BTEC vs A-Level performance Modular curriculum + Large cohorts = Poor cohort identity Learning behaviours/ Student engagement Student satisfaction Adapt quickly to Uni life Meet and work with students on the same programme Get a clear view of expectations Develop independent learning for HE Develop an understanding of assessment

OfS Catalyst B Project – Maximising Student Success in Assessment Through a Focus on Self-Regulation 2 year Office for students Catalyst B project Multi-institutional project researching interventions to develop student self- regulation in relation to assessment and feedback Standard set of validated research instruments to measure assessment literacy and feedback orientation Our sub project is looking at the impact of attending peer learning sessions on the development of assessment literacy and feedback orientation

Research methodology 2017-18 cohort 2018-19 cohort October: Post-its, Demographic data, Pre ALQ survey, November: Pre FOS Survey February: Focus group 1, Post-its, Post ALQ survey, Post FOS survey May: Focus group 2 Data analysis/ develop peer learning and induction October: Post-its, Demographic data, Pre ALQ survey, Pre FOS Survey February: Focus group 1, Post-its, Post ALQ survey, Post FOS survey May: Focus group 2, Peer Leader Focus Group Data analysis/ present findings

Lessons Learned Collecting data is hard work! Data collection (use paper and be careful about formatting!) Pick lectures that are going to be attended Data is inherently skewed to students who engage Post-it note surveys are easy and SO useful! Numbers required for quantitative analysis present real challenges

All this resulted in A LOT OF DATA!

Survey instruments Assessment Literacy Questionnaire (ALQ) alqAL Assessment for Learning e.g. I use assessment to work out how well I am doing alqMEO Minimum Effort Orientation e.g. My aim is to pass the course with as little work as possible alqAU Assessment Understanding e.g. I understand the criteria alqAJ Assessment Judgement e.g. I judge my own work using my knowledge of the criteria Feedback Orientation Scale (FOS) fosUT Feedback Utility e.g. Feedback is critical for improving performance fosACC Feedback Accountability e.g. I don’t feel a sense of closure until I respond to feedback fosSA Social Awareness e.g. Feedback lets me know how I am perceived by others fosSE Feedback Self-Efficacy e.g. I know that I can handle the feedback I receive Smith, C.D. et al., (2013) Assessment literacy and student learning: the case for explicitly developing students ‘assessment literacy’, Assessment & Evaluation in Higher Education, 38:1, 44-60 Linderbaum, B.A., and Levy, P.E. The Development and Validation of the Feedback Orientation Scale (FOS) Journal of Management, 36:6, 1372-1405

Findings: Assessment Literacy Students showed significant improvements in assessment understanding after semester 1 for both years (2017-18: M=3.53->M=3.77, p=0.002, 2018-19: M=3.40->M=3.86, p<0.001) In year 2, students also showed significant improvement in assessment judgement (M=3.00->M=3.20, p=0.028) Corresponding increase in Minimum Effort Orientation in year 2 (M=2.09->M=2.34, p=0.003)

Findings: Assessment Literacy and Peer Learning Negative correlation between pre_MEO and attendance in year 1 (i.e. students with a high minimum effort orientation attended fewer sessions). Positive correlation between attendance and assessment understanding in year 1 Aggregating both years data: High attenders significantly improved assessment understanding, judgement and minimum effort orientation – is this a good thing? Low attenders also significantly improved assessment understanding but not by as much as high attenders (gap of 0.45 vs 0.24) Differences in gap scores between high and low attenders for AU, AJ and MEO are approaching or at significance

Assessment Literacy and Peer Learning: Qualitative data Students in focus groups and peer leaders identified ways that assessment literacy had been improved: “Some of them weren’t aware of the criteria, the marking criteria being available on Blackboard and I showed them and they’re very confused because the lecturer didn’t show them” (Peer Leader) “They said they can advise on it, but, obviously, they can’t give us a grade, or whatever. I don’t think they’re able to do that (1st year student) “I can say they helped me with understanding how I am supposed to structure my essays, basically, which is good” (1st year student) “They tell you what to expect from assessments, and how to do them. It’s good overall” (1st year student)

Findings: Feedback orientation In general, feedback utility and feedback accountability start high and decline after the first semester!

Findings: Feedback orientation No significant findings between attendance and feedback orientation In 2017-18 white students had significantly higher feedback utility that BME students after semester 1 In 2017-18 students who’s parents attended uni had significantly higher feedback utility after semester 1 In 2018-19 these differences disappeared

Conclusions… so far! There is some evidence that structured peer learning can help to develop assessment literacy (understanding and judgement) Improving understanding and effort also results in a reduction of the amount of effort students perceive they need to put in! The relationship between peer learning and feedback orientation is less clear Attendance is significantly related to grades (but this is well known and could relate to motivation to study)

The benefits of Peer Learning go far beyond assessment and feedback! “They help you with actual student life, and not just your modules, and assessments, and things like that. People going to university, for most people it’s a really big change to their lifestyle, because they’re learning to live by themselves, and everything. I think, from someone who’s already had previous experience of it, like peer leaders, they have a lot of knowledge and things that they can actually teach us.” (1st year student)

Neil Ford Senior Teaching Fellow (Education Development) Southampton Business School N.J.Ford@soton.ac.uk