How can we lead change to ensure the academic integrity of examinations? A project to develop a TEQSA Good Practice Note for the sector Led by Maddy McMaster The University of Melbourne Jon Yorke, Lesley Sefcik Curtin University With colleagues from Melbourne, Curtin, and you!
Key themes we will discuss Academic integrity remains a significant risk (TEQSA) Examinations are no panacea for ensuring integrity Still no cost effective solution for online assessment In this presentation… 1- Embedding integrity in exams at University of Melbourne 2- Online assessment and remote proctoring at Curtin 3- The TEQSA Good Practice Note and a call for exemplars
Embedding integrity in examinations at the University of Melbourne
Melbourne Uni probes inside job exam scam Where it all began … THE AGE - OCTOBER 8, 2016 Melbourne Uni probes inside job exam scam Nick McKenzie and Richard Baker
The exams lifecycle Administrative support Departmental academic staff upload the exam to the printing company’s online portal. Pre-production If requested by the subject coordinator, the printing company provides hard and/or soft copies of the exam for proofing. Printing & transport The printing company produces the requisite number of hard copies of the finalised exam and delivers them to the University or exam venue. I Invigilation Students sit the exam at the two main exam venues or other venues on campus. Collation & transport The completed exam papers are collated by the invigilators and transported back to the faculty by an external courier company. Student viewing The exam papers are available for viewing by students once the final results have been published. Marking & result data entry The exam papers are marked, the results are collated and then entered in to the University’s central results database. Final storage The marked exam papers are stored at the faculty for at least six months. Exam paper creation Formulation of the exam questions and solutions by the subject coordinator and/or other faculty staff. At least one other faculty member will provide QA. 1 2 3 4 5 Examination lifecycle 7 11 10 8 9 Storage (pre- exam) The exam papers are stored at the University or exam venue prior to invigilation for up to two business days. 1 2 3 4 5 6 6 7 11 8 10 Storage (post-exam) The exam papers are stored at the faculty/ASO office until they are taken away for marking 9 Source: PricewaterhouseCoopers Consulting (Australia) 2017
The exams lifecycle Administrative support Departmental academic staff upload the exam to the printing company’s online portal. Pre-production If requested by the subject coordinator, the printing company provides hard and/or soft copies of the exam for proofing. Printing & transport The printing company produces the requisite number of hard copies of the finalised exam and delivers them to the University or exam venue. I Invigilation Students sit the exam at the two main exam venues or other venues on campus. Collation & transport The completed exam papers are collated by the invigilators and transported back to the faculty by an external courier company. Student viewing The exam papers are available for viewing by students once the final results have been published. Marking & result data entry The exam papers are marked, the results are collated and then entered in to the University’s central results database. Final storage The marked exam papers are stored at the faculty for at least six months. Exam paper creation Formulation of the exam questions and solutions by the subject coordinator and/or other faculty staff. At least one other faculty member will provide QA. 1 2 3 4 5 Examination lifecycle 7 11 10 8 9 Storage (pre- exam) The exam papers are stored at the University or exam venue prior to invigilation for up to two business days. 1 2 3 4 5 6 6 7 11 8 10 Storage (post-exam) The exam papers are stored at the faculty/ASO office until they are taken away for marking 9 Source: PricewaterhouseCoopers Consulting (Australia) 2017
EXAMPLE OF APPROACH TO MANAGING RISKS Stage 1: exam paper creation 2 3 4 5 Examination lifecycle 7 11 10 8 9 No university-wide procedure Too many eyes Hard copy versions in unsecured locations Printing in public areas No ‘chain of custody’ Use of personal ICT devices 6
ACTIONS TO MITIGATE AGAINST RISKS Stage 1: exam paper creation 2 3 4 5 Examination lifecycle 7 11 10 8 9 Discipline-appropriate electronic options for collaborative development ExamSoft trial New policy requirements Requirement to use enterprise exams portal Late submissions 6
Strategy for change Identify electronic solutions Amend policy and processes e.g. storage Increase awareness Develop local plans Longer-term end-to-end management
Work at Curtin - Online environments How can we assure integrity in online contexts, especially at scale? What practices are institutions adopting to address these issues? What opportunities are there for sector collaboration on common areas of concern? WA Forum on Contract Cheating, Perth – October 2016 National Forum: Assessment Integrity and the Role of Examinations, Melbourne - August 2017 Conference Stream: Assessment, Integrity Review, Melbourne – June 2018
Remote proctoring trials at Curtin Curtin Remote Invigilation System (CRIS) Functionality Records behaviour and identity Flags unusual activity for staff review Proof of concept testing Assessed feasibility within capstone units Examined ease of use, student and staff perceptions
Key findings CRIS works An academic integrity breach occurred in the proof of concept testing Staff workload seems manageable Instruction & setup ~1-3 hours Management ~1 min per student per eTest Minimal student queries Students became more comfortable over time
TEQSA Good Practice Note on exams integrity What clever things have you done to prevent cheating in exams at your university?
‘Chapters’ in the Good Practice Note Integrity in examination question development and drafting of exam papers. Ensuring secure processes for administration and organisation of exams. Managing allegations of cheating in examinations. Ensuring integrity in marking and results processes. Ensuring integrity in providing feedback to students. Secure storage and handling.