Student Peer-Assessment of critical appraisal essays Gill Price, 2012 Norwich Medical School University of East Anglia.

Slides:



Advertisements
Similar presentations
Refreshing institutional policies around academic integrity: a focus on student training Dr Neil Morris Faculty of Biological Sciences.
Advertisements

Experience of using formative assessment and students perception of formative assessment Paul Ong Greg Benfield Margaret Price.
Innovation in Assessment? Why? Poor student feedback regarding feedback timeliness and usefulness Staff workloads Student lack of awareness as to what.
Placement Workshop Y2, Sem 2 Professional Practice Module (PPM)
Audio-feedback Background – why I did it. My experiences – what I did. Student feedback – how did it go?
Professional Perspectives: Electronic Engineering Paul Spencer Dean of School, Electronic Engineering Kal Winston* Adviser, Study Skills Centre.
Designing Scoring Rubrics. What is a Rubric? Guidelines by which a product is judged Guidelines by which a product is judged Explain the standards for.
INCLUSIVE ASSESSMENT IN THE SCIENTIFIC CURRICULA Dr Kimberley Bennett Mr Sebastian Stevens.
By : Sarah A. Bajafer Choose a topic which interests and challenges you. Your attitude towards the topic may well determine the amount of effort and.
Doctoral Training Workshops Getting published and the reviewing process Steve Potter and Sue Oreszczyn January 2015.
Developing your Assessment Judy Cohen Curriculum Developer Unit for the Enhancement of Learning and Teaching.
Making Sense of Assessments in HE Modules (Demystifying Module Specification) Jan Anderson University Teaching Fellow L&T Coordinator SSSL
ACADEMIC INFRASTRUCTURE Framework for Higher Education Qualifications Subject Benchmark Statements Programme Specifications Code of Practice (for the assurance.
Oxford Centre for Staff and Learning Development Student assessment: lightening the load while increasing the learning Dr Chris Rust Head, Oxford Centre.
Peer Assessment Peer Assessment What’s it all about? Ian Hughes Faculty of Biological Sciences, University of Leeds, UK
Workshop on Peer Instruction: Setting Yourself up for Success Beth Simon Computer Science and Engineering University of California, San Diego Formerly,
Using the WebCT discussions function for seminars Mike Parkes School of Sport & Exercise Sciences.
INTERNAL ASSESSMENT Constitutes 20% of your IB history score
Project Workshops Results and Evaluation. General The Results section presents the results to demonstrate the performance of the proposed solution. It.
1 Health Management and Social Care Third Consultation.
Advanced Research Methodology
Course Basics Presented by Elisa P. Paramore Program Counselor.
May May Win MA (TESOL) Lancaster University.  Choosing the course  Choosing modules  Study tips  Support from the university  Coursework  Dissertation.
Doctor of Education (EdD). Programme Objectives March EdD Program 1  The Doctor of Education (Ed.D) is designed to produce high quality academics.
Feedback on exams Vicki Bruce On behalf of School of Psychology.
Writing a Literary Research Paper How to Read an Article of Literary Criticism.
Blackboard Based Learning and Assessment Tom Rippeth School of Ocean Sciences Bangor University.
ULTSEC Innovation Fund Ruth Valentine School of Dental Sciences Simon Cotterill Learning and Teaching Support Unit, Faculty of Medical Sciences.
Is PeerMark a useful tool for formative assessment of literature review? A trial in the School of Veterinary Science Duret, D & Durrani,
The Submission Process Jane Pritchard Learning and Teaching Advisor.
Developing your Assessment and Feedback Judy Cohen Curriculum Developer Unit for the Enhancement of Learning and Teaching.
The Learning and Teaching Conference nd April 2015.
CompSci 725 Handout 7: Oral Presentations, Projects and Term Reports Version July 2009 Clark Thomborson University of Auckland.
CM602: Effective Systems Development Exam Preparation Slide 1 Exam Preparation Preparing for the CM602 Examination D.Miller J. Horton.
Preparing papers for International Journals Sarah Aerni Special Projects Librarian University of Pittsburgh 20 April 2005.
Strategy BSNS7340 Studio 9 semester two >>FACULTY OF CREATIVE INDUSTRIES AND BUSINESS Industry Based Learning – attend the pre course session to.
Animals, Society and Culture Summer examination
Module 5 Week 11 Supplement 12. SPEAKING TRUTH EFFECTIVELY How to provide insightful and effective peer reviews.
Exam Taking Kinds of Tests and Test Taking Strategies.
From MPL to VPL Upping the visibility of Management Practice & Law Joëlle Darby 3 rd March 2015.
CLT Conference th July 2015 Nudge! A structured programme of support aimed at improving student achievement where average grades are just below.
Students’ and Faculty’s Perceptions of Assessment at Qassim College of Medicine Abdullah Alghasham - M. Nour-El-Din – Issam Barrimah Acknowledgment: This.
How to read a scientific paper
INTERNAL ASSESSMENT ADVICE Or…how to get a 7 on your Internal Assessment.
From description to analysis
Module, Course and Unit Evaluations Module, course or unit evaluations give you the opportunity to make your voice heard by giving feedback about your.
Critically reviewing a journal Paper Using the Rees Model
Intensive Course in Research Writing: Session 1 (27 June 2011)
Feedback in University Teaching Prof. Arif Khurshed Division of Accounting and Finance.
Project 1 (CGNB 413) Briefing
Why bother giving feedback?. How not to provide feedback?
SE 521 Dr. Lola Taylor Unit one. INTRODUCTION This week's seminar will focus on getting acquainted and understanding the expectations for the course.
INTERNAL ASSESSMENT ADVICE Or…how to get a 7 on your Internal Assessment.
Low Level Programming Introduction & Recap Duncan Smeed.
Assessment Feedback.
ASSESSMENT LITERACY AND SELF-ASSESSMENT STRATEGIES Dr Tina Kendall ALSS, Department of English & Media Course Leader for Film Studies/Film & Media Studies.
A MEMBER OF THE RUSSELL GROUP. Denis Duret School of Veterinary Science University of Liverpool Denis.
Assessment for Learning Practical ideas to use in your classroom every day.
Creating Assessments that Engage Students & Staff Professor Jon Green.
Assessment for Learning Centre for Academic Practice Enhancement, Middlesex University.
Assessment & Feedback Working Group Developing Departmental Assessment & Feedback Practices The ‘Quick Wins’ Paper.
Revising Your Paper Paul Lewis With thanks to Mark Weal.
MA English Language Teaching: Online
“Biology Microbial Diversity”
BIO1130 Lab 2 Scientific literature
Returning to Study (Study Skills)
Critical Analysis CHAPTER 7.
BIO1130 Lab 2 Scientific literature
Creative assessment and feedback
Presentation transcript:

Student Peer-Assessment of critical appraisal essays Gill Price, 2012 Norwich Medical School University of East Anglia

Outline Why critical appraisal essays –history Why peer-marking Why formative The assignment –Instructions, paper/s to be appraised –Marksheet Peer-marking arrangements Moderating

Outline (contd) Results –Marks –Student evaluations of the process –Student feedback to peers Conclusions

Critical appraisal – Why? Essential for practice of EBM (?) Required by ‘Tomorrow’s Doctors’ : “Outcomes for graduates: The Doctor as a Scholar and a Scientist #12 Applying scientific method to medical research Critically appraise the results of relevant diagnostic, prognostic and treatment trials and other qualitative and quantitative studies as reported in the medical and scientific literature.” (General Medical Council 2009)

Critical appraisal Local history (Norwich Medical School, UEA) 5 critical appraisal assignments (‘Analytical Reviews’, ARs) in Years 1-2 (~150 students each year) –Some short-answer, some essay format (all 2000 words) –General medical research article –5 weeks to brew –3 Study designs plus 1 qualitative methods –Formative assessment started in in Year 1, 1 in year 2

Critical appraisal examples instructions paper marksheet marking guidelines

Teaching support Extensive “Research Methods” teaching series in Year 1-2 (10-20 sessions/term) ‘Basic’ stats (plus regression) – all focussed on interpretation Study designs, critical appraisal seminars

Peer-assessment – why? To distil student learning and deepen understanding To give students experience & practice at giving feedback To save staff marking time

Bloom’s Taxonomy of educational objectives (Atherton, 2005)

Peer assessment - how Students submit one script to deadline Marking guidelines then made available – each student instructed to mark their own script in a week Next week have a ‘marking lecture’ –explains the procedure and main points Compulsory 2-hr marking session in 5 rooms – each has a tutor

Formative peer-assessment exercise Submitted scripts (ID number only) Room 4 29 stdts Mark room 5 scripts Room 2 30 stdts Mark room 1 scripts Room 5 29 stdts Mark room 4 scripts Room 1 30 stdts Mark room 3 scripts Room 3 30 stdts Mark room 2 scripts

Peer assessment – what each student does Each student receives the script of another plus marking guidelines Marking done within the session Marks and feedback to be given Marker is anonymous – script identified only by ID Staff have a record of who marked whose Hand in for ‘moderation’ and returned to author with feedback

Principles of Marking scope for flexibility in marking scheme not exact science, subjective, markers will differ ! give credit for sound and consistent arguments based on evidence credit points made in a different section must show understanding of concepts, not merely repeat what authors say must be consistent in argument, not contradictory ‘grammatical prose’

‘Moderation’ Module leader entered student marks (from second year of peer-marking only) and checked a sample of scripts from each stratum (around 20% of total) Added to student feedback where necessary Message sent to all students giving the distribution of marks, a summary of differences with moderator in the sample The mean difference between the moderator’s score and the student’s score was -0.2 (SD 1.4) in

Results (1): Student marks (% failed first time) * * Term 1 Formative 7% (summ- ative) No data36%28%13% Term 2 Summative 5%7%11%9% *From the marking range was expanded from 0-15 (with pass mark 8) to 0-24 (with pass mark 14)

Comparison of Term 1 formative and Term 2 summative marks for the same students (pass mark=8) Density Rot 1 mark (peer-assessed) Peer-assessed, formative marks , n= Density Rot 2 mark Staff-marked, summative marks , n=160

Comparison of Term 1 formative and Term 2 summative marks for the same students (pass mark=14) Density Rot 1 mark (peer-assessed) Peer-assessed, formative marks , n= Density corrected rot2 mark after moderating Staff-marked, summative marks , n=146

Results (2) Student feedback on the process (N=159) after first year of peer-assessment Most-common responses to open-ended questions What you learned What markers are looking for in AR*, what aspects to focus on (n=79) Marking ARs is difficult (24) How to organise/structure essay better (15) Subjectivity: Unclear what happens if you disagree with marker or include things not in guidelines (9) How others appraise a paper (6) What needs improving Was useful as it was (n=17) Want scripts marked also by examiners, more authoritative feedback (11) Streamline the process (12) Improve anonymity (11) *AR=analytical review (=critical appraisal)

Results (2) Student feedback on the process: The ‘formative’ effect In later years some students commented in evaluations about their peers not making an effort in writing the script – spoiled the marking experience! Scrutiny of the scripts with lowest peer-marks revealed that students wrote something vague but when they came to the parts where most students struggle (eg. the Results section!) they simply gave up. Some scripts could be regarded as ‘token submissions’ (very low word count and apparently little effort applied) Negation of the point of formative - ‘having a go’

Results (3) – some favourite examples of peer-to-peer feedback Marker: “ Avoid using confusing technical terms as sometimes you don’t seem to understand them ” Excerpt from script: “ Also the objectives the researchers were measuring were presented well which helped produce more readily interpretive and generalisable results. This also helped the aims of the researchers to be achieved. ”Comment by marker: “ Because the data was well presented? Really? ” Marker: “ It is evident that you understand what you’re talking about, just make sure and cover what is asked “

Some smilers..... [this and take ]

Some downers..... (an over-optimistic and not very assiduous marker commenting on a ‘token script’.....) Moderator marks on right edge of structured feedback-sheet

Some more attentive feedback

Some students do ‘get it’ !!...

[you know this as ] [ terminology correctly in ]

Conclusions Peer assessment can be a great learning experience if taken seriously Gives insight for writing the assignment, what is expected, how subjective is critical appraisal, and how hard it is to mark BUT Interpretations in essays which are not covered in marking guidelines are hard for students to assess Formative assignments mean many students do not try very hard: they skim over the ‘difficult’ parts Students not satisfied with peer-only assessment - still want staff to mark their work for ‘authoritative’ feedback

References ATHERTON JS. Learning and Teaching: Bloom's taxonomy [On-line] UK [updated 2005; cited January 2009 ]; Available from: General Medical Council (2009) Tomorrow's doctors: Outcomes and standards for undergraduate medical education. London: General Medical Council.