1. SOLSTICE Conference 2015 4 th & 5 th June 2015 Technology enhanced formative assessment: Participant experiences, relative efficiencies and tutor learning curves Dr W. Rod Cullen, Manchester Metropolitan University

2. Dr W. Rod Cullen Susan Gregory Dr Neil Ringan Mark Roche

3. Formative assessment... “... is designed to provide learners with feedback on progress and inform development, but does not contribute to the overall assessment.” QAA 2000 “I can’t get my students to do work that doesn’t get a mark” “My students don’t use the feedback I give them – sometimes they don’t even pick it up” “That all looks very impressive, but how long is it going to take me to do that?”

4. Session outline Narrative account of evolution of “Enhancing Learning Teaching and Assessment with Technology” unit (ELTAT) Delivery model Feedback methods Feedback efficiencies Participant experiences Conclusions and recommendations

5. A brief history of ELTAT DateUnit TitleCreditWeeksTutor(s) 2007-09Designing Courses for VLEs (DCVLEs) 104WRC 2009-10Designing Courses for VLEs (DCVLEs) 104WRC, SG 2011-13Design Effective Blended and Online Learning (DEBOL) 104WRC, SG with guest appearances from NR and MR 2014Enhancing Learning Teaching and Assessment with Technology (ELTAT) 3012WRC, SG, NR, MR

6. Learner Profile Range of backgrounds – New and experienced academic staff – Academic support staff – Technical support staff Majority working full-time but some part-time Differing assessment requirements – PG Cert Academic Practice – MA Academic Practice – CPD for credits – CPD non-credits Just like real students!

7. Formative delivery model Online Preparation In Class Session 1 Online formative activity Online Preparation In Class Session 2 Online formative activity Week 1 Week 2 Feedback Mini (e)Portfolio Feedback

8. Providing feedback – Text based

9. Audio feedback MS Word file including annotations MP3 audio file

10. Screen capture feedback

11. Data Collection Tutor Activity – since 2007 1.Kept a detailed feedback activity log 2.Worked out time spent engaged in each feedback activity. End-to-end process and each stage: Download, Annotate, Write/ Record, Publish, Upload Participant experience – 2007-2014 1.Interviewed participants about experiences 2.End of unit survey 3.Submission data 4.Attendance 5.Informal anecdotal evidence Ongoing Opportunistic Action Research

12. Early Experiences DCVLEs

13. Feedback methods (DCVLEs) MethodsDetails WrittenAnnotated comments on Submission MS Word + Written Comments AudioAnnotated comments on Submission MS Word + Audio recording Video Screen Capture Annotated comments on Submission MS Word + Video Screen Capture

14. Producing formative feedback 1. Getting up to speed with new technology (Pebblepad) 2. Technical problems with codecs and files sizes (common with web-based video) 3. Initially uncomfortable with recording process Issues for Susan 4. Got up to speed with audio format more quickly Cullen W. R. (2011)

15. Percentage Attendance and engagement with formative tasks 2007-10 1. Reasonably high overall attendance 2. Generally high (70% +) submission rates for formative tasks 4. Consistently lower submission rates in Week 4 MA Summative Assessment 3. Use of the feedback is embedded in the online prep and F2F activities

16. Ongoing development 2011 Programme Review: Designing Effective Online and Blended Learning (DEBOL) Change in unit title Neil and Mark – contributions as specialist lecturers 2013 Major Programme Review: Enhancing Learning Teaching and Assessment with Technology (ELTAT) 30 Credits Much longer delivery – 12 weeks More time to complete tasks (and provide feedback) Team approach to formative feedback – Tutors allocated group of student to work with throughout the course

17. Recent experience (ELTAT)

18. Same delivery and assessment model

19. Feedback methods Formative Activities Learning outcomes review Complete LTA matrix review Revise LTA matrix Assessment strategy Revised LTA matrix 2 Monitoring and evaluation plan Timeline ELTAT - 2014 Feedback provision Moodle Assignment; MS Word Comments + Audio recording Moodle Assignment; MS Word Comments + Video Screen capture Moodle Assignment; MS Word Comments + Written Proforma Turnitin Assignment; Quickmarks, General Comment + Audio recording Moodle Assignment; MS Word Comments + Audio recording

20. Producing formative feedback 1. Rod + Susan very similar times 2. Neil + Mark learning the ropes 3. Pronounced for video and audio About the task as much as the technology 4. Similar times for written feedback – new task 5. Similar times for TII (Written + Audio)

21. Team overview 1. For team as a whole consistent ≈ 55 mins per submissions (end-to-end) 2. Variation in what the time is required for 3. Download times minimal 4. Upload of audio and video = minimal time but 2-3 mins more than written. 5. No down/upload for TII 6. Additional “publish” stage for audio 7. Reading through and annotating the submissions is most time consuming activity

22. Efficiency of production Written feedback: Word count tool in MS Word Divided by the total time (end-to-end) to produce the feedback Screen capture and audio feedback: Counted spoken words in 1 minute of recorded feedback = est. speaking rate. Length audio and video recording X speaking rate = estimate of the total number of words in each piece of feedback. Divided by the total time (end-to-end) to produce the feedback Turnitin: Pasted the general text comments into a word document – Word count tool. Length of audio feedback X tutors speaking rate = estimate of number of word spoken. Work count + Words spoken = estimate of the total number of words in each piece of feedback. Divided by the total time (end-to-end) to produce the feedback Annotations (comments in MS word and Quickmarks in Turnitin) not included

23. Estimated Feedback words/per minute 1. Audio and Video screen capture – learning curve? Task as well as tech 2. Written feedback - Personal preferences and expertise is an influence 3. TII least efficient – multi-media + suspect we made more comments in the text (Quickmarks) 5. Comparing efficiencies across technologies is more complex than anticipated 4. Mark has felt need to reflect on his feedback on all tasks

24. Student perspectives Prefer audio and video to written feedback Personal and Engaging Understandable Annotations important 50:50 split between audio and video Utilised differently Video more impact Audio more reflective

25. Learning curves associated with – “Technical skills” – Feelings of self-consciousness – Familiarity with requirements of the formative task Technical aspects not as time consuming as cognitive Audio feedback can be produced quickly and simply Model of delivery can influence engagement – need to add value to completing formative tasks Participants indicated a preference for audio and video screen captures – Linked to characteristics of the task – Possible novelty aspect Conclusions Take home messages To embed formative assessment into a curriculum Design it into delivery model – make the value of the feedback clear Align with summative assessment and signpost this to students Select feedback technology best matched to the type of tasks Be prepared for the learning curve

26. Just in case section

27. Feedback methods Online Formative Activities Learning outcomes review Complete LTA matrix Revise LTA matrix Activity Design Plan Monitoring and evaluation plan Timeline 2011-14 Feedback provision Moodle Assignment; Track Changes/ annotated script MS Word Moodle Assignment; Annotated MS Word doc + Video screen capture Moodle Assignment; Annotated MS Word doc + Audio recording 2008 Feedback provision Online discussion forum; Track changes/ annotated script MS Word Annotated MS Word doc + Written Comments Annotated MS Word doc + Video screen capture Annotated MS Word doc + Video screen 2009 and 2010 Feedback provision Written proformas via Pebblepad Annotated MS Word doc + Video screen capture Annotated MS Word doc + Audio recording

28. Data Collection All tutors kept details feedback activity logs relevant to the technologies being used Track Changes + Audio recording Track Changes + Screen Capture Track Changes + Written Proforma Turnitin (Quickmarks, Comments, Audio) Download Annotation (Track Changes) Annotation (Quickmarks) General Text Comment Recording Publishing Upload

29. Logistical considerations DCVLEs & DEBOL versions Core Option 10 credits – Nominal 100 hrs Very limited F2F contact (4 x 3 hours sessions - 12 hours in total) ELTAT 30 Credits – Nominal 300 hours More F2F Over a long period

30. Design considerations Relevant Purposeful