1. SYMPOSIUM ENTRUSTMENT AND LEARNING ANALYTICS IN E-PORTFOLIOS FOR WORKPLACE LEARNING AND ASSESSMENT PRESENTERS: WATCHME-TEAM DISCUSSANT: PROF. DR. DAVID BOUD, UNIVERSITY OF TECHNOLOGY, SYDNEY, AUSTRALIA www.project-watchme.eu @Project_WatchMe Workplace-based e-Assessment Technology for Competency- based Higher Multi-professional Education This project has received funding from the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no 619349 EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS

2. 1.Utrecht University, NL 2.University Medical Centre Utrecht, NL 3.Szent Istvan University, Hungary 4.University of Tartu, Estonia 5.Universitätsmedizin Charité Berlin, Germany 6.University of California San Francisco, USA 7.Maastricht University, NL 8.Mateum, NL 9.University of Reading, UK 10.Jayway, Denmark 11.NetRom, Romania/NL EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 1

3. WATCHME’S AIM Improve efficiency and quality of workplace- based feedback and assessment by means of a mobile electronic portfolio system, that is enhanced: Conceptually with the concept of Entrustable Professional Activities Technically through Learning Analytics: Student models that monitor the learners’ competency development and inform learners and supervisors (based on data of students, supervisors and peers - probabilistic algorithms that learn from new incoming data) Personalized feedback and visualization of development. EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 2

4. OVERVIEW PRESENTATIONS 1. prof. Harm Peters, Charité - Universitätsmedizin Berlin, Germany Delphi study into Entrustable Professional Activities 2. dr. Bert Slof, Utrecht University, the Neth. and dr. Äli Leijen, University of Tartu, Estonia Competences and Assessment of Student Teachers 3. dr. Marieke van der Schaaf, Utrecht University, the Neth. and dr. Bert Slof, Utrecht University, the Neth. Electronic Portfolios and Learning Analytics 4. Ing. Eelco Scheurs, Maastricht University, The Netherlands Participatory Design of Learning Analytics EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 3

5. ENHANCING ELECTRONIC PORTFOLIOS FOR WORKPLACE-BASED ASSESSMENT BY LEARNING ANALYTICS UTRECHT UNIVERSITY: MARIEKE VAN DER SCHAAF AND BERT SLOF www.project-watchme.eu @Project_WatchMe EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS

6. Develop complex competences Integrated in context Demands long learning trajectories in workplace Deliberate practice: feedback and reflection EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 5

7. “Well done!” “Pleasure to supervise!” “Reliable candidate” “Poor fund of content knowledge” “Needs lots of supervision” UNFORTUNATELY, HOW MANY DAILY FEEDBACK PRACTICES LOOK LIKE EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 6

8. Personalized Feedback that gives (Sadler, 1989; 2010): insight into performance ability to evaluate and monitor own process suggestions to fill gap between expected norm and performance That feeds into learners’ major feedback questions (Hattie & Timperley, 2007): Where am I going? (goals, feedup) How am I going? (feedback) Where to next? (feedforward) WHAT IS NEEDED EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 7

9. PRINCIPLES OF GOOD FEEDBACK (NICOL AND MACFARLANE-DICK, 2006) 1. Helps clarify what good performance is 2. Facilitates the development of self-assessment in learning 3. Delivers high quality information to learners about their learning 4. Encourages teacher and peer dialogue around learning 5. Encourages positive motivational beliefs and self-esteem 6. Provides oppurtunities to close the gap between current and desired performance. 7. Provides information to supervisors EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 8

10. ENTRUSTABLE PROFESSIONAL ACTIVITY Task based instead of construct based approach Crucial question: would I entrust this learner unsupervised with this task? (with my sick mother, animal or teach my daughter/son…) An EPA is a task that an individual can be trusted to perform unsupervised, in a given professional context, once sufficient competence has been demonstrated. International Competency-Based Medical Education Collaborators, March 18, 2014 Ten Cate, Chen, Hoff, Peters, Bok & Van der Schaaf, 2015 EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 9

11. COMBINE EPA APPROACH WITH LEARNING ANALYTICS EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 10 Application of probabilistic student models that enable feedback based on multi sorted assessments Measurement, collection, analysis and reporting of data about trainees in their contexts, for the purpose of understanding, and optimising learning and the utilising of environments in which it occurs (Solar, 2013) Personalized feedback Visualizing learners’ development

12. ELECTRONIC PORTFOLIO EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 11

13. THE OVERALL WATCHME ARCHITECTURE EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 12

14. EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS ASSESSMENT ARGUMENTS (MISLEVY, 2006)  What EPAs should be assessed and how does the learner develop during the curriculum?  What performance indicators should be used to gain insight into a learner’s competence?  What instruments should be used to assess the EPAs? Student Model Evidence Model Task Model

15. ARCHITECTURE OF BAYESIAN STUDENT MODEL EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 14

16. BAYESIAN STUDENT MODEL represents the actual internal cognitive state of each learner as well as their actual learning context. contains enough pedagogical knowledge in order to be able to translate the internal state and context into meaningful messages and information for visualization. is a back-end component, meaning that no direct user interaction is made with this module. will make suggestions to help the learners, assessors or supervisors improve their performance. EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 15

17. Domain SM vs Individual SM Domain SM (DSM) Will encapsulate all the pedagogical knowledge required to model one domain Possible models are: Anaesthesiology Training, Veterinary Education, Teacher Education and General Undergraduate Medical Education Variations of these proposed models: Veterinary Education (NL vs HU) Teacher Education (NL vs EE) General Undergraduate Medical Education (NL vs DE) EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 16

18. Domain SM vs Individual SM Individual SM (ISM) Will encapsulate all the pedagogical knowledge required to model one learner. The Individual SM will be a personalized domain model One ISM cannot handle more than one domain at once For multiple roles of the same individual (e.g. a learner is both a supervisor and a trainee) the ISM will cover only the learner’s role EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 17

19. JUST IN TIME PERSONALIZED FEEDBACK Type of feedback Content of feedback Moment of feedback EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 18 The Bayesian student model must indicate appropriateness of: Some users are motivated by competition? Others find competition demotivating. Users want to know whether they are on track. Users want to be able to highlight and save useful feedback. Users would like an overview of weekly, monthly and yearly goals.

20. TASK AND EVIDENCE MODELS EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS

21. TYPES OF FEEDBACK EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 20

22. TYPES OF FEEDBACK EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 21

23. SURVEY REQUIREMENTS PERSONALIZED FEEDBACK Two rounds, n = 8 Experts perceived the preliminary design for personalized feedback to be indicative for high quality and useful feedback Most of the comments experts made were in congruence of the principles of good feedback Result: least attention was paid to support student’s self-esteem and motivation EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 22

24. Example program codes: JIT Feedback EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 23 { "EPA": "TT_UU_EPA5", "PerformanceIndicator": "TT_UU_PI1", "PerformanceIndicatorLevel": "TT_UU_LEVEL3", "Translations": { "en": “check more often when existing tests and digital testing systems are inadequate and design new tests (incl. correction sheets).", "nl": “vaker te controleren of bestaande toetsen en digitale toetssystemen ontoereikend zijn en ontwerp hiervoor een nieuwe toetsen (incl. correctiemodel)." }

25. Example Personalized feedback in EPASS portfolio EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 24

26. Example Personalized Feedback in EPASS portfolio EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 25

27. 1.Utrecht University, NL 2.University Medical Centre Utrecht, NL 3.Szent Istvan University, Hungary 4.University of Tartu, Estonia 5.Universitätsmedizin Charité Berlin, Germany 6.University of California San Francisco, USA 7.Maastricht University, NL 8.Mateum, NL 9.University of Reading, UK 10.Jayway, Denmark 11.NetRom, Rumania/NL THANK you for your attention! EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 26

28. ARCHITECTURE & API DESIGN EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 27

29. 2. 2. FACILITATES THE DEVELOPMENT OF SELF-ASSESSMENT (REFLECTION) IN LEARNING Timeline overview: Detailed visualization: EARLI CONFERENCE, 27 AUG 2015, LIMASSOL CYPRUS 28