1. Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk Mobile Assessment of Work-Based Learning: Evaluation in Progress. Chris Dearnley – Learning and Teaching Fellow, University of Bradford Julie Laxton- ALPS Teaching Fellow, University of Leeds Shupikai Rinomhota – Lecturer, University of Leeds

2. Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk ALPS CETL Who we are? A centre for excellence in teaching and learning (CETL) focussing on assessment and learning in practice settings. 5 Universities (Leeds, Leeds Metropolitan, Bradford, Huddersfield and York St John) crossing 16 health and social care professions What is the aim of this programme? To achieve excellence in assessment and learning based on interprofessional Common Competency Maps, which can be delivered by mobile technology To ensure that students graduating from courses in health and social care are fully equipped to perform confidently and competently at the start of their professional careers

3. Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk ALPS CETL This programme is developing 3 approaches to improving confidence and competence in pre-registration health and social care students Promoting interprofessional assessment of core competences Testing the viability of the use of mobile devices to deliver assessment and learning in the practice setting Encouraging service user and carer involvement in student feedback

4. Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk ALPS Mobile Assessment processes 3 competency maps developed; 5 generic assessment scenarios; Working Interprofessionally Gaining Consent Providing Information When to Consult and Refer Demonstrating respect for Service User

5. Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk ALPS Mobile Assessment processes These facilitated formative feedback from a range of stakeholders Delivered onto the mobile devices, linked with e-portfolio system for tutor feedback - T-mobile Vario I with unlimited internet access

6. What was the purpose of this group? Initially - to research the learning and teaching aspects related to mobile assessment processes Eventually - incorporated evaluation of the ALPS assessment tools in all formats Our key motivation/question – DOES IT MAKE A DIFFERENCE? Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk E-Valuation working group

7. Representatives from all five sites 3 clearly defined work streams  Work Stream 1 – evaluation of devices and tools with students (two stages)  Work Stream 2 – evaluation of mobile assessment tool with staff (lecturers & mentors)  Work Stream 3 – reliability and usability questionnaire Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk E-Valuation working group

8. Please read the extract taken from the ALPS Evaluation Strategy and consider the practical implementation of this plan, particularly taking into account the following: The size and complexity of the cohorts involved The type of delivery of the tool Obtaining ethical clearance The data collection methods employed What issues might be encountered? Would you have used the same approach? Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk Evaluation Challenges

9. Diversity of HEIs Different ethical approval processes Uni-professional or multi-professional focus groups? How do we ensure consistency? Mobile vv paper based tools delivery The challenges of collaboration! Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk Evaluation Challenges

10. Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk Mobile device implementation processes Selected cohorts across the 5 sites All professions included Standard training delivered Placement areas prepared Confidentiality and security ensured

11. Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk Work Stream 1 – evaluation of mobile devices and assessment tools with students (stage 1) 1 Cohort specific focus groups; 7 professions & 79 students ALPS Research Officer attended all to ensure consistency across sites Thematic analysis of focus group sessions Students invited to complete on-line diary/blog Evaluation processes

12. Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk Findings Two extremes of students – those who use it a lot and those who don’t use it at all. When no alternative and it offers something of value they will use it –e.g. in community settings (OTs, SWs) or where they have no computer access. Perseverance by those who find it useful/have a reason e.g. dyslexic students

13. Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk Findings There appears to be a clear differences across the professions due to culture, client group and context in which they operate; e.g. audiologist as clear technicians as opposed to SW; medics Huge training needs so that students understand potential uses, e.g. audio

14. Assessment and Learning in Practice Settings (ALPS) © September 2008 http://www.alps-cetl.ac.uk Student Quotes – the good... Yes I think it’s good in that respect ……(student referring to reflection) …………I found it great on a weekend for me because I could go on to the library and access um a PDF and it’s actually really quite clear on the screen…….. ….You see it’s nice getting the comments from (Tutor) because I don’t get to see (Tutor) a lot in clinic …. and I can see if we’d been doing that from the start that would have been good because he’d have been able to see how I’d progressed…….

15. Assessment and Learning in Practice Settings (ALPS) © September 2008 http://www.alps-cetl.ac.uk Student Quotes – the bad... We need a device with better functionality so that the software just goes on it If I could use if for personal purposes as well like phone calls, I would use it more I think the idea of having something where you can look stuff up when you’re with somebody is, is great, but realistically with the, the size and quality of the internet pages you get on here and the speed of it, it doesn’t, doesn’t really make it very, very workable.

16. Assessment and Learning in Practice Settings (ALPS) © September 2008 http://www.alps-cetl.ac.uk Student quotes – the ugly... our deputy manager has asked me not to use it with clients because he thinks people associate them playing – little gizmos, with playing games, even though I explained what it was …. and it just takes up more time than it saves I’m afraid

17. Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk Work Stream 2 – evaluation of mobile assessment tool with staff (lecturers & mentors) There are barely enough computers in practice settings, so students need there own, but this also means there they are not technically minded enough to understand student use It skills are part of Band 5 requirements and we should expect an electronic portfolio The HPC ask for a “wide variety of evidence” I don't see why we shouldn't have shared assessment tools given the similarities (OT/physio)

18. Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk Work Stream 3 – reliability and usability questionnaire – 112 responses Whilst most participants used computers for a variety of purposes only 51.4% participants used their mobile devices and then it was only 1-2x per month Difficulties for their use – most commonly cited was; 42% no connection and 36% device lost their charge. 96% received training before using their device.

19. Assessment and Learning in Practice Settings (ALPS) © http://www.alps-cetl.ac.uk Overview of Outcomes Mobile learning & assessment offers huge potential for increasing flexibility and choice Many students and staff like the idea The technology has to be user friendly – students don't want an extra device –they want to use their own This presents security challenges There are cost and accessibility implications Mobile assessment of practice settings requires a shift in culture underpinned by training and awareness raising

20. Assessment and Learning in Practice Settings (ALPS) © September 2008 http://www.alps-cetl.ac.uk ALPS Contact details Chris Dearnley – c.a.dearnley1@bradford.ac.ukc.a.dearnley1@bradford.ac.uk Julie Laxton- j.c.laxton@leeds.ac.ukj.c.laxton@leeds.ac.uk Shupikai Rinomhota- a.s.rinomhota@leeds.ac.uka.s.rinomhota@leeds.ac.uk Email: alps@leeds.ac.uk Tel: 0113 343 6352 General Website: www.alps-cetl.ac.ukwww.alps-cetl.ac.uk