Evaluation of the ‘Electronic Feedback’ Marking Assistant and Analysis of a Novel Collusion Detection Facility Phil Denton Faculty Learning Development Manager Liverpool JMU

Typical Class at Liverpool JMU (Scouse Stereotype #34)

Computer Assisted Assessment (CAA) “CAA is essentially automated assessment through assessment through objective testing.” UK University Website

Types of CAA n Objective Testing n Data Retrieval Exercises n Electronic Submission n Free Text Analysis n Marking Assistants -May be used to assist in the marking of work that would normally be done exclusively by hand -Includes packages that can generate and feedback, after appropriate data have been inputted

Marking in the UK n Vast majority is still done by hand n CAA preferred for formative/diagnostic testing n Summative ‘high stakes’ assessments do not normally benefit from electronic assistance n There is an “... absence of any front-running software in this regard, compared to the high profile of Question Mark.” (D. P. Stephens 2001) n Most of the existing report-writing assistants are designed for the US High School market n Electronic Feedback: Designed for UK academics by a UK academic

Electronic Feedback n MS Excel/Word based Marking Assistant n Functionalised with VBA programming n Consists of 3 files: - Feedback.xls - Guide.xls - Fb.doc n Version 8 (2001/2): 22 Users n Version 9 (2002/3): 44 Users n Version 10 (Summer 2003): ??

Demonstration n Electronic Feedback 10 -Work in progress n Electronic Feedback 9 -Collusion detection -Feedback generation

Evaluation: Feedback Facility n ed questionnaire returned by 22 users of Version 8 n It was agreed that the software allows users to return: -more feedback (2.1) -of higher quality (1.6) -in a shorter space of time (2.1) (Averaged Likert Scale responses)

Evaluation: Collusion Detection n Tested over 11 JMU assessments n Identical feedback detected in 5 assessments, 19 instances in total. -17 cases were coincidental -1 case was suspicious -1 case was plagiarism n One ‘failed’ test repeated using customisable detection criteria  5 comments in common,  1 different -9 cases found, 1 was plagiarism

Conclusions n Electronic Feedback is... -returned more rapidly, -more extensive, -of higher quality. n V10 has these advantages and... -Facility for multiple markers, -Normal & Criterion styles available simultaneously, -Reduced Memory problems.

The Future? Development of a wholly web-based version of Electronic Feedback, integratable with existing Institutional databases

The End To arrange a training session at your institution, or receive a copy of V10, when ready, please the author