Simon.davis@york.ac.uk Wayne.britcliffe@york.ac.uk University of York Give them what they want: developing a flexible anonymous assignment workflow to meet diverse needs Simon.davis@york.ac.uk Wayne.britcliffe@york.ac.uk University of York

Outline EMA in the sector and York Requirement gathering &“Agile” development Evaluation and ways forward

Electronic Management of Assessment (EMA) in the UK Growing adoption and interest across the sector (UCISA 2014) Strategic priority for JISC / UCISA / HELF identified benefits for Students (convenience) Admin (efficiencies) Academics (pedagogic) Turnitin Grademark leading e-submission solution, 63% End to end solution remains elusive JISC, Southampton, Northumbria, BB/SITS integration?

Generic EMA lifecycle 1 - Specifying 8 Reflecting / evaluation 2 - Setting 3 - Supporting 4 - Submitting 5 - Harvesting the work 6 - Marking and feedback 7 - Returning marks and feedback 8 Reflecting / evaluation Lifecycle model adapted from Manchester University TRAFFIC Project

Context of EMA @ York ~14,000 students Anonymity mandatory “except where unfeasible” Assessment principles: Equity, Openness, Clarity, Consistency No current institutional EMA policy /mandate Widespread opt in to e-assessment workflows Iterative development delivering the big things first Flexibility for local adaptations Active user group and network driving local adoption 20,558 files submitted, 11,267 feedback files returned online last academic year

Anonymous assignment submission: iterative development 2007: Online submission Submitted files anonymised using exam numbers 2013: Feedback return Feedback forms / marksheets generated and returned to students through SITS 2014: Feedback flexibility Support for annotated work Marks and feedback for exams 2015: User experience Policy & guidance Markers Students 2016? Feedback hub Supervisor access Prototype in development Increasing departmental uptake

Lifecycle stage Common requirements Flex? Location Specifying Support Location Specifying Summative anonymity Depts specify assessment X Setting Deadlines, file types, late submissions, fb type Assessment details Y Supporting Student guidance MIT circs Central Submitting Anonymised with exam number Text files Multiple files in zips Harvesting Student work Feedback (forms and annotations) Marksheets Tracking tools Dept approach Marking and feedback File sharing Feedback forms / templates / annotation Platform Printing F’b approach Returning marks / fb 6 week anonymous fb turnaround Marks registered in SITS Reflecting / evaluation Student / supervisor access to anonymous feedback Dept / central FBH

York Anonymous Assignment: Evaluation Highlights Institution: Need to fill gaps in policy and guidance Admin (n=18): Easy to use and saves time Students (n=261): Generally +ive, FB quality, confidence issues Markers (n=48): Mixed reaction to all aspects of workflow Most negative response to reading work on screen Broadly positive response to feedback production Areas for development: Improved workflow and speed / More managed system Annotations – awareness, workflow, reusable, workload implications Resistance to greater use of technology

Polarised response from markers “I can see no merit in this system. I have other experience of on-line marking from my external examining and have found it very difficult, time consuming and problematic. There are some parts of the pedagogic process that are not amenable to on-line systems and marking essays is one of these. The student sget a worse experience, worse feedback and and I have found it tiresome, unnecessarily complicated and much much slower. I conclude that it is inefficient use of time, makes turning the essays round longer, and pedagogically worse outcome.” “This was my first year using this system and it was revolutionary. I could mark anywhere... Also there was no printing out of forms at the Dept Office. The whole thing was slick and easy.”

Next steps Continued incremental improvements Marker experience Annotation; explore use of Google drive / docs / forms Other forms of feedback; portfolios, media etc Supervisor access – Feedback hub prototype Central e-assignment strategy