1. Simon.davis@york.ac.uk Wayne.britcliffe@york.ac.uk University of York
Give them what they want: developing a flexible anonymous assignment workflow to meet diverse needs
University of York

2. Outline EMA in the sector and York
Requirement gathering &“Agile” development
Evaluation and ways forward

3. Electronic Management of Assessment (EMA) in the UK
Growing adoption and interest across the sector (UCISA 2014)
Strategic priority for JISC / UCISA / HELF identified benefits for
Students (convenience)
Admin (efficiencies)
Academics (pedagogic)
Turnitin Grademark leading e-submission solution, 63%
End to end solution remains elusive
JISC, Southampton, Northumbria, BB/SITS integration?

4. Generic EMA lifecycle 1 - Specifying 8 Reflecting / evaluation
2 - Setting
3 - Supporting
4 - Submitting
5 - Harvesting the work
6 - Marking and feedback
7 - Returning marks and feedback
8 Reflecting / evaluation
Lifecycle model adapted from Manchester University TRAFFIC Project

5. Context of EMA @ York ~14,000 students
Anonymity mandatory “except where unfeasible”
Assessment principles: Equity, Openness, Clarity, Consistency
No current institutional EMA policy /mandate
Widespread opt in to e-assessment workflows
Iterative development delivering the big things first
Flexibility for local adaptations
Active user group and network driving local adoption
20,558 files submitted, 11,267 feedback files returned online last academic year

6. Anonymous assignment submission: iterative development
2007: Online submission
Submitted files anonymised using exam numbers
2013: Feedback return
Feedback forms / marksheets generated and returned to students through SITS
2014: Feedback flexibility
Support for annotated work
Marks and feedback for exams
2015: User experience
Policy & guidance
Markers
Students
2016? Feedback hub
Supervisor access
Prototype in development
Increasing departmental uptake

7. Lifecycle stage Common requirements Flex? Location Specifying
Support
Location
Specifying
Summative anonymity
Depts specify assessment X
Setting
Deadlines, file types, late submissions, fb type
Assessment details Y
Supporting
Student guidance
MIT circs
Central
Submitting
Anonymised with exam number
Text files
Multiple files in zips
Harvesting
Student work
Feedback (forms and annotations)
Marksheets
Tracking tools
Dept approach
Marking and feedback
File sharing
Feedback forms / templates / annotation
Platform
Printing
F’b approach
Returning marks / fb
6 week anonymous fb turnaround
Marks registered in SITS
Reflecting / evaluation
Student / supervisor access to anonymous feedback
Dept / central
FBH

8. York Anonymous Assignment: Evaluation Highlights
Institution: Need to fill gaps in policy and guidance
Admin (n=18): Easy to use and saves time
Students (n=261): Generally +ive, FB quality, confidence issues
Markers (n=48): Mixed reaction to all aspects of workflow
Most negative response to reading work on screen
Broadly positive response to feedback production
Areas for development:
Improved workflow and speed / More managed system
Annotations – awareness, workflow, reusable, workload implications
Resistance to greater use of technology

9. Polarised response from markers
“I can see no merit in this system. I have other experience of on-line marking from my external examining and have found it very difficult, time consuming and problematic. There are some parts of the pedagogic process that are not amenable to on-line systems and marking essays is one of these. The student sget a worse experience, worse feedback and and I have found it tiresome, unnecessarily complicated and much much slower. I conclude that it is inefficient use of time, makes turning the essays round longer, and pedagogically worse outcome.” “This was my first year using this system and it was revolutionary. I could mark anywhere... Also there was no printing out of forms at the Dept Office. The whole thing was slick and easy.”

10. Next steps Continued incremental improvements Marker experience
Annotation; explore use of Google drive / docs / forms
Other forms of feedback; portfolios, media etc
Supervisor access – Feedback hub prototype
Central e-assignment strategy