1. Participatory Design of a Learning Analytics enhancement E-portfolio: The Human Factor Eelco Schreurs 1, Atta Badii 2, Annemarie Camp 1, Inge Duimel-Peeters 1, Ovidiu S ̧ erban 2, Daniel Thiemert 2, Jaime Costa 2, Marieke van der Schaaf 3, Jeroen Donkers 1, Geraldine Clarebout 1 1 2 3

2. INTRODUCTION EARLI CONFERENCE - LIMASSOL (CYPRUS)– AUGUST 2015 1 Workplace learning Different learning opportunities for each learner Encountered cases Feedback Same competency level to be achieved Importance of “good” assessment instruments

3. INTRODUCTION EARLI CONFERENCE - LIMASSOL (CYPRUS)– AUGUST 2015 2 Importance of “good” assessment instruments Measure required competency level Considering variety of learning experiences (Electronic) Portfolio Work completed Feedback Progress made Reflections => Longitudinal and multi- faceted information (van Tartwijk, Driessen, van der Vleuten & Stokking, 2007; Sengler & Kanthan, 2012) www.blendspace.com

4. INTRODUCTION EARLI CONFERENCE - LIMASSOL (CYPRUS)– AUGUST 2015 3 Succes factors Mentor Part of institutional assessment procedure Flexible format Threads Availability of time User friendeliness (Driessen, van Tartwijk, van der Vleuten and Wass, 2007) Learning analytics? adaptive Persona -lized

5. (Greller & Drachsler, 2012, p. 44) LEARNING ANALYTICS WITHIN THE WATCHME PROJECT Learning analytics as (personal) probabilistic learning models Based on all information in electronic portfolio Computerized processing of large amount of data BUT: also human factor; initiated through interface (Dron & Anderson, 2009) EARLI CONFERENCE - LIMASSOL (CYPRUS)– AUGUST 2015 4

6. RESEARCH QUESTION Starting point: Human-centred design approach usability of e-portfolio Research Question What are the needs and objectives of the different stakeholders (‘the human factor’) in relation to a learning analytics enhanced electronic portfolio assessment? EARLI CONFERENCE - LIMASSOL (CYPRUS)– AUGUST 2015 5

7. METHOD Participants EARLI CONFERENCE - LIMASSOL (CYPRUS)– AUGUST 2015 6 Participatory meeting AnaesthesiologyVeterinary MedicineTeacher EducationTeam Members 121314 (10 technical, 4 educational) 24113 (technical) 37 (5 physicians, 2 trainees) --9 (technical)

8. METHOD Design Participatory design strategy (Berns, 2004; Könings, Brand-Gruwel & van Merriënboer, 2010) Three participatory meetings between May - September 2014 1 st participatory meeting visualise the landscape in which the WATCHME-project and its stakeholders will exist. 2 nd participatory meeting visualise the landscape for each domain (veterinary medicine, anaesthesiology and teacher education) 3 rd participatory meeting => ideas and requirements for just-in-time feedback => ideas and requirements for the visualization module => ideas for building a student model EARLI CONFERENCE - LIMASSOL (CYPRUS)– AUGUST 2015 7

9. RESULTS First participatory meeting: EARLI CONFERENCE - LIMASSOL (CYPRUS)– AUGUST 2015 8 Technical stake- holders Develop new FB modules Integration of different systems and models Creation of cross- domain system Manageri al stake- holders Compare different groups Hospitals: higher quality, safety of care, transparancy Curriculum improve- ment

10. RESULTS First participatory meeting: EARLI CONFERENCE - LIMASSOL (CYPRUS)– AUGUST 2015 9 Info from different sources / observers Trainers / assessors Easy and fast access to info 360° insight Trainees Trans- parancy Structured feedback conver- sations Safeguards confidentia- lity & anonymity History of learner => focussed FB Just-in- time feedback Longitu- dinal monitori ng Anywhere and anytime! Just-In-Time feedback

11. RESULTS Second participatory meeting: 1.Distinction needed between supervisors and assessors Supervisors: Analyze group of students of same cohort Development of individual student compared to group Accessible through personal computer Assessors: Quick overview on smartphone Elaborate overviews on personal computer Need for notification when assessment needed 2.Definition of different categories of JIT-feedback reminders/notifications content feedback longitudinal feedback EARLI CONFERENCE - LIMASSOL (CYPRUS)– AUGUST 2015 10

12. RESULTS Second participatory meeting: 3.feedback in a quantitative (scores) and qualitative way (narrative feedback) needed 4.Insight into the learning progress Over longer period about students’ performance on a general level Short term for performance for particular activities EARLI CONFERENCE - LIMASSOL (CYPRUS)– AUGUST 2015 11

13. RESULTS Third participatory meeting 1.Focus on JIT-feedback Preference for narrative feedback Possibility to tag narrative feedback For numeric feedback: comparison with peers or European standard Draw attention on difference score assessor and self-assessment Alert when they miss milestone or deadline 2.Rules for not well-performing students Number of mini-clinical evaluation exercises (mini-CEX) is dropping. Poor reflective skills: poor narratives, small sentences. Pattern in recent assessments. Poor performers seek supervisors who do not give feedback or only positive feedback (feedback avoidance) EARLI CONFERENCE - LIMASSOL (CYPRUS)– AUGUST 2015 12

14. DISCUSSION AND CONCLUSION Participatory meetings allowed to grasp the requirements and expectations of the stakeholders System will need to provide personalized feedback System will need to be adaptive with respect to: Device used Professional domain Role of the user Shortcomings: Not all target groups participated => convenience sample Data analysis => time pressure EARLI CONFERENCE - LIMASSOL (CYPRUS)– AUGUST 2015 13

15. ACKNOWLEDGMENTS European commission: grant agreement n°: 619349 : ‘WATCHME Workplace-based e-assessment technology for competency-based higher multi-professional learning Suzanne Schut (for acting as an interim coördinator) EARLI CONFERENCE - LIMASSOL (CYPRUS)– AUGUST 2015 14