1. Anonymous assignments: Core product or customisation? Simon.davis@york.ac.uk University of York

2. Session overview 1.EMA in the UK 2.EMA workflow requirements 3.Anonymous assignment submission @ York 4.Blackboard’s anonymous assignment submission 5.Ways forward: core product or bespoke development? 2

3. Session outcomes You – what we do at York – what BB does themselves – where do you need to be? Me – benchmark York within the sector – inform use of BB anon assignment – sanity check BlackBoard – anon assignment user feedback 3

4. Electronic Management of Assessment (EMA) in the UK Growing adoption and interest across the sector (UCISA 2014) Strategic priority for JISC / UCISA / HELF identified benefits for – Students (convenience) – Admin (efficiencies) – Academics (pedagogic) Turnitin Grademark leading e-submission solution, 63% End to end solution remains elusive – JISC, Southampton, Northumbria, BB/SITS integration? 4

5. Assessment and feedback lifecycle 5 1 - Specifying 2 - Setting 3 - Supporting 4 - Submitting 5 - Harvesting the work 6 - Marking and feedback 7 - Returning marks and feedback 8 Reflecting / evaluation Lifecycle model adapted from Manchester University TRAFFIC Project

6. EMA workflow requirements http://goo.gl/hMfx5q 6

7. EMA @ York: Contextual factors Anonymity mandatory “except where unfeasible” – Assessment principles: Equity, Openness, Clarity, Consistency No current institutional EMA policy /mandate, The “York Way” ~14,000 students Change is coming… – New L&T strategy and faculties – Growing DL provision 7

8. Anonymous assignment submission at York Bespoke building block deployed since 2007 for file based workflow Iterative development to support submission and feedback return (2014/15) Complete interdependence on SITS data; student exam number and assessment records Student work anonymised on submission and tracked throughout the rest of the lifecycle by exam number Flexible “barebones” toolset maps onto familiar admin processes 20,558 files submitted, 11,267 feedback files returned online this academic year (Oct 14 - March 2015) 8

9. York Anonymous Assignment: Workflow & features Deadlines and late submission alerts Restrict file types (access / annotation) Setting Submitted files (30MB max) labelled with exam no & timestamped E-mail receipt and alerts Submitting Associate submissions with SITS assessment record Download options; submissions, feedback (blank / template / annotated work) marksheet, non-submission and tracking info Harvesting the work Create feedback in standalone form / annotate work Marks recorded in marksheet Marking / feedback Completed feedback and marks batch uploaded and made available to students through SITS e:vision Returning marks / feedback 9 VLE Dept defined (shared drive / Google…) SITS e:vision

10. York Anonymous Assignment: Evaluation Highlights Institution: Need to fill gaps in policy and guidance Admin (n=18): Easy to use and saves time Students (n=261): Generally +ive, FB quality, confidence issues Markers (n=48): Mixed reaction to all aspects of workflow – Most negative response to reading work on screen – Broadly positive response to feedback production – Areas for development: Improved workflow and speed / More managed system Annotations – awareness, workflow, reusable, workload implications Resistance to greater use of technology 10

11. Polarised response “I can see no merit in this system. I have other experience of on-line marking from my external examining and have found it very difficult, time consuming and problematic. There are some parts of the pedagogic process that are not amenable to on-line systems and marking essays is one of these. The student sget a worse experience, worse feedback and and I have found it tiresome, unnecessarily complicated and much much slower. I conclude that it is inefficient use of time, makes turning the essays round longer, and pedagogically worse outcome.” “This was my first year using this system and it was revolutionary. I could mark anywhere (just using a USB stick and a backup) and there was no copying of disks. Also there was no printing out of forms at the Dept Office. The whole thing was slick and easy.” 11

12. York Anonymous Assignment: Pros / Cons Pros – Familiar and evolutionary In use since 2007 File based – closely aligned with paper based workflows Significant “organic” opt-in from Departments – Batch upload of feedback created offline – Low entry point – markers do not need to engage with VLE – Technology imposed where it counts – No possibility of anonymity being circumvented Cons – Feedback production cumbersome for some – Anonymity is permanent; sharing work with supervisors etc… – End to end workflow – Missed opportunity for greater use of TEL? 12

13. BlackBoard Anonymous assignment – step by step 1 – Activate / deactivate when editing an assignment 13

14. BlackBoard Anonymous assignment – step by step 2 – Students submit and advised of anonymity 14

15. BlackBoard Anonymous assignment – step by step 3 – Student submissions obfuscated in the GradeCentre 15

16. BlackBoard Anonymous assignment – step by step 4 – Read and provide feedback anonymously 16

17. BB Anon Assignment @ York? “I think that having the materials submitted online is good, and I prefer marking digitally. I found the VLE comments section less useful. I downloaded the essays and used the track changes function of Word, which worked great for me.” “I've also been trialling non-anonymous marking through the VLE for one module - where we annotated on screen. I really liked providing feedback this way.” Marker preference for working offline – Batch upload of feedback – Downloading doesn’t respect Smart Views / Delegation – No delegation without reconciliation Interface – Learning a new system – Limited display area – is it as good as reading / annotating in Word?display Integration with SITS e:vision – untested @ York Anonymity via randomly generated number – Tracking non submissions etc – All or nothing 17

18. Comparing York and BB Anonymous submission FeatureYorkBB SpecifyingLimited to 30MBNo limit SettingFile restrictions Requires SITS Formative / summative SupportingStaff for more advanced useGreater staff to get started SubmittingAnonymous trackingTracking difficulty Harvesting the work Admin download files File management issues Direct access by markers Download issues Marking and feedback Offline only Familiar feedback approaches Simple paper reading Disability stickers Paper logistically tricky Flexible Feedback tools Manual entry into VLE – no batch upload Returning marks / FB Pass to Admin Batch upload to SITS Direct through VLE Untested SITS integration Reflecting / evaluation Anonymous identifier cannot be shared Possibility within the VLE Longitudinal view problems 18

19. Does BB Anon Assignment meet our workflow requirements? http://goo.gl/hMfx5q 19

20. Links and resources JISC EMA guidance - http://www.jisc.ac.uk/guides/electronic- assessment-management JISC EMA guidance - http://www.jisc.ac.uk/guides/electronic- assessment-management 20

21. Detail on Evaluation outcomes follows 21

22. Evaluation: Administrators (n=18 from 11 depts) Easy to use, adoption simplified by fit with current practices Saves time – Student submission – 4.67 / 5 – Sharing files with markers – 4.25 / 5 – Adding marks to e:vision – 4.43 / 5 – Returning feedback to students 5 / 5 – Complete workflow 4.69 / 5 Minor enhancement requests Highlighted policy and guidance blindspots 22

23. Evaluation: Students (n=261 from 15 depts) Clear preference for “easy” online submission – Positive impact on student experience Preference for online feedback (delivery mode) – Feedback “utility” not linked to delivery mode – Correlation between feedback utility and feedback type (annotations preferred) General lack of confidence in IT systems – Potential to address with guidance / training 23

24. Markers: overall reaction Mixed reactions across all aspects of the workflow; – access to files – reading and marking – feedback production & quality – recording marks Broad acceptance from most markers, vitriolic resistance from some opponents 24

25. Markers: reading and marking work 83% read all work on screen despite options to print – 41% - “about the same as reading on paper” – 40% - “generally worse than paper” – 19% - “generally better than on paper” Polarised / negative response to marking on tablets – 33% - “Not at all – I hate the idea” – 29% - “Not bothered either way” – 23% - “Yes I really want to be able to do this” – 15% - “Other” (mainly qualified enthusiasm) 25

26. Markers: feedback provision and quality “Easier to give more tailored (and legible) feedback to students. Also was possible to use "boilerplate" text to multiple students who had made similar errors.” 26

27. Markers: Top suggested areas for development Improved workflow and speed – “The whole system is very cumbersome and clunky.” Annotations – Awareness – Workflow – Reusable – Workload implications / strategic design of feedback More managed system; VLE tools / Grademark – “Many other institutions use more stable systems where scripts and marks are linked and can be accessed and viewed more easily.” Resistance to greater use of technology – “I would much prefer to leave things as they are and continue with the paper-based system. I work on the computer all day and do not want to mark on the screen as well.” 27

28. 28

29. 29