1. IMPROVING WORKPLACE-BASED ASSESSMENT AND FEEDBACK BY LEARNING ANALYTICS AND E-PORTFOLIO MADRID 2014 SIG 1 CONFERENCE EARLI. AUGUST 27-29. MARIEKE VAN DER SCHAAF UTRECHT UNIVERSITY THE NETHERLANDS www.project-watchme.eu @Project_WatchMe Workplace-based e-Assessment Technology for Competency-based Higher Multi-professional Education EARLI SIG 1 MADRID AUG 2014

2. EU-PROJECT WATCHME EARLI SIG 1 MADRID AUG 2014

3. PARTNERS 1.Utrecht University, NL (www.uu.nl) GENERAL LEADwww.uu.nl 2.University Medical Centre Utrecht, NL (www.umcutrecht.nl)www.umcutrecht.nl 3.Szent Istvan University, Hungary (www.sziu.hu)www.sziu.hu 4.University of Tartu, Estonia (www.ut.ee)www.ut.ee 5.Universitätsmedizin Charité Berlin, Germany (www.charite.de)www.charite.de 6.University of California San Francisco, USA (www.ucsf.edu)www.ucsf.edu 7.Maastricht University, NL (www.maastrichtuniversity.nl) TECHNICAL LEADwww.maastrichtuniversity.nl 8.Mateum, NL (www.mateum.nl)www.mateum.nl 9.University of Reading, UK (www.reading.ac.uk)www.reading.ac.uk 10.Jayway, Denmark (www.jayway.com)www.jayway.com 11.NetRom, Rumania/NL (www.netrom.nl)www.netrom.nl Educational partners Technical partners

4. - Develop complex competences - Integrated in context - Demands long learning trajectories in workplace - Deliberate practice: feedback and reflection EARLI SIG 1 MADRID AUG 2014

5. Feedback that is (Wiggins, 1997): -Timely -Specific -Understandable to receiver -Allow receiver to act on (refine, revice, practice, retry) That feeds into students’major feedback questions (Hattie & Timperley, 2007): -Where am I going? (goals, feedup) -How am I going? (feedback) -Where to next? (feedforward) WHAT IS NEEDED EARLI SIG 1 MADRID AUG 2014

6. “Well done!” “Pleasure to supervise!” “Reliable candidate” “Poor fund of content knowledge” “Needs lots of supervision” EARLI SIG 1 MADRID AUG 2014 UNFORTUNATELY, HOW MANY DAILY FEEDBACK PRACTICES LOOK LIKE

7. CHALLENGE: COMPETENCY BASED EDUCATION AND ASSESSMENT World-wide wish to restructure professional’s training, focusing on the attainment of specific competencies and changing assessment practices. From 90s onwards. EARLI SIG 1 MADRID AUG 2014

8. FROM 90S ONWARDS: ELECTRONIC PORTFOLIOS Portfolios contain selected evidence of performance and products accompanied by reflections. Reflective ‘log’ of trainee over longer time span Repository of evidence regarding performance etc Different instruments and methods EARLI SIG 1 MADRID AUG 2014

9. ELECTRONIC PORTFOLIOS MIXED SUCCESS Underuse of potential data and not very well tailored to worksituations (not timely). Drawbacks: Authentic, personal unique character of evidence, difficults scoring. Full standardization impossible, asks for interpretation of assessor (no golden standard or external criterion). Impacts reliability and validity of the assessment. Limited impact on learning Low motivation of students EARLI SIG 1 MADRID AUG 2014

10. Example Digital portfolio, example veterinary education Aggregation of data points  Mini-CEX (clinical evaluation exerc.)  Multisource feedback  Case reports  Knowledge tests  Personal development plan EARLI SIG 1 MADRID AUG 2014

11. CRITICS TO USE OF CB-ASSESSMENT Too much emphasis on accountability, and outside control by politics and society Too behavioristic - leads to reductionist checkbox assessment approach Too theoretical; it remains unclear what competencies are Competencies cannot be measured validly EARLI SIG 1 MADRID AUG 2014

12. Illustration of critics

13. Medical expert Communicator Collaborator Manager Health advocate Scholar Professional MOST CRITICS COME DOWN TO THE ANALYTIC APPROACH EARLI SIG 1 MADRID AUG 2014

14. Medical expert Communicator Collaborator Manager Health advocate Scholar Professional THE ANALYTIC APPROACH With nursing staff With family With patients With colleagues With trainees … EARLI SIG 1 MADRID AUG 2014

15. Medical expert Communicator Collaborator Manager Health advocate Scholar Professional THE ANALYTIC APPROACH With nursing staff With family With patients With colleagues With trainees … Consultation Breaking bad news Explain medication With children With elderly … EARLI SIG 1 MADRID AUG 2014

16. WATCHME’S AIM Improve efficiency and quality problems of workplace- based feedback and assessment by means of a mobile electronic portfolio system, that is enhanced with: -student models that monitor the learners’ competency development. -visualisation tools that inform learners, teachers and institutes just in time as well as on an analytical level. EARLI SIG 1 MADRID AUG 2014

17. ELECTRONIC PORTFOLIO EARLI SIG 1 MADRID AUG 2014

18. STUDENT IS OWNER Trainee’s portfolio data remain personal property They control to whom they share information to Use for research purposes by means of informed consent EARLI SIG 1 MADRID AUG 2014

19. ENTRUSTABLE PROFESSIONAL ACTIVITY Task based instead of construct based approach Crucial question: would I entrust this candidate unsupervised with this task? (with my sick mother, animal or teach my daughter/son…) An EPA is a task that an individual can be trusted to perform unsupervised, in a given professional context, once sufficient competence has been demonstrated. International Competency-Based Medical Education Collaborators, March 18, 2014 EARLI SIG 1 MADRID AUG 2014

20. Medical expert Communicator Collaborator Manager Health advocate Scholar Professional THE SYNTHETIC FRAMEWORK APPROACH EPA1EPA2EPA3EPA4EPA5 EARLI SIG 1 MADRID AUG 2014

21. EXAMPLES ENTRUSTABLE PROFESSIONAL ACTIVITY (EPA) Conducting patient hand-overs Anesthetic management a patient Conducting a normal, low risk delivery Interviewing adolescents regarding high risk health behavior EARLI SIG 1 MADRID AUG 2014

22. SUPERVISION LEVELS OPERATIONALIZED 1.Acting is not allowed: more observation 2.Acting is allowed under direct supervision: supervision physically present; proactive supervision 3.Acting is allowed under indirect supervision: supervision not physically present, but quickly available if needed; reactive supervision 4.Acting is allowed “unsupervised”:under clinical oversight; distant/backstage supervision; post hoc report 5.Supervision may be provided to juniors (to junior residents, to medical students at level 2 or 3) A formal acknowledgment happens at level 4 EARLI SIG 1 MADRID AUG 2014

23. Developing Entrustable Professional Activities Title Keep it short; avoid words related to proficiency or skill. Ask yourself: Can a trainee be scheduled to do this? Can an entrustment decision for unsupervised practice for this EPA be made and documented? Description To enhance universal clarity, include everything necessary to specify: What is included? What limitations apply? Limit the description to the actual activity. Avoid justifications why the EPA is important, or references to knowledge and skills. Link with a competency framework Which competency domains apply? Which sub-competencies apply? Include only the most relevant ones. These links serve to justify the EPA and to build observation and assessment methods. Required know- ledge, skills and attitudes Which KSA are necessary to execute the EPA? Formulate this in a way to set expectations. Refer to resources that reflect necessary or helpful standards (books, a skills course, etc.) Information sources to assess progress Consider observations, products, monitoring of knowledge and skill, multi-source feedback. Planning level 4 (“unsupervised practice”) Estimate when during training full entrustment to allow for unsupervised practice is expected to be reached. Acknowledge the flexible nature of this timing. Expectations of entrustment moments can shape an individual workplace curriculum. Making a formal entrustment decision How often must the EPA be executed proficiently for Level 4 (unsupervised practice). Who will judge this? Should external experts be involved? What does a formal entrustment look like (signed, documented, publicly announced)? EARLI SIG 1 MADRID AUG 2014

24. EXAMPLE EPA: "Investigation Ordering and Interpretation" To enable an entrustment decision the candidate:.. (Communication) Communicates investigations and results clearly to the patient. (Advocacy) Assists patients who are unlikely able to navigate the health system in arranging the investigation (Scholar) Doesn’t order unnecessary screening tests (Scholar) Doesn’t order tests when the pre-test probability is minimal (Professional) Seeks assistance when s/he doesn’t know the answer. (Professional,Manager) Ensures timely receipt and management of results Thanks to Daniel Ince-Cushman (McGill) 10-3-14. EARLI SIG 1 MADRID AUG 2014

26. COMBINING THE COMPETENCY-FRAMEWORK WITH ENTRUSTABLE PROFESSIONAL ACTIVITIES Medical expert Communicator Collaborator Manager Health advocate Scholar Professional ++ + + + + + + + + + + + + EPA1EPA2EPA3EPA4EPA5 Assessment

27. EPA-approach used in Family Medicine, Kingston, Ontario Thanks to Jane Griffiths & Karen Schultz EARLI SIG 1 MADRID AUG 2014

28. GROWTH OF ENTRUSTABILITY OVER TIME trainingdeliberate professional practice proficient expert competent advanced novice EARLI SIG 1 MADRID AUG 2014

29. CURVES OF ONE TRAINEE trainingdeliberate professional practice EPA1 EPA4 EPA2 EPA3 EPA5 Compe- tence thres- hold Justified entrustment decisions EARLI SIG 1 MADRID AUG 2014

30. ANOTHER TRAINEE trainingdeliberate professional practice EPA1 EPA4 EPA2 EPA3 EPA5 Compe- tence thres- hold Justified entrustment decisions EARLI SIG 1 MADRID AUG 2014

31. Example of formalized entrustment decision From: EPA Handbook of Royal Australian & New Zealand College of Psychiatrists EARLI SIG 1 MADRID AUG 2014

32. The International Conference on Residency Education | La conférence internationale sur la formation des résidents Mehta et al 2013 Digital badge as EPA-STAR

33. EARLI SIG 1 MADRID AUG 2014

34. COMBINE EPA APPROACH WITH LEARNING ANALYTICS Measurement, collection, analysis and reporting of data about trainees in their contexts, for the purpose of understanding, and optimising learning and the utilising of environments in which it occurs (Solar, 2013). Application of probabilistic student models that enable tailored visualised feedback based on multi sorted assessments Bayesian models provide detailed model of trainees’ entrustability EARLI SIG 1 MADRID AUG 2014

35. BAYESIAN NETWORKS 1) CONSTRUCTION OF GRAPH REPRESENTING QUALITATIVE INFLUENCES OF SITUATION MODEL 2) ASSIGNMENT OF PROBABILITY TABLES TO EACH NODE IN GRAPH

36. MULTI-ENTITY BAYESIAN NETWORKS (LASKEY, 2008) Mfrag as basis concept Mfrag: parameterized Bayesian network fragment that represents uncertain relationships among a small collection of related hypotheses Can be instantiated and combined to form complex graphical probability models Learns from incoming data http://unbbayes.sourceforge.net/ University of Brasil & George Mason University (Costa & Laskey) Tool for constructing and querying MEBN and other extended Bayesian network formats

37. MEBN ARCHITECTURE ePASS databases Temporary BN Background Machine learning engine Background Machine learning engine REQUEST BN Constructor BN Constructor REPLY BN Reasoner BN Reasoner structure MEBN Fragment Store EARLI SIG 1 MADRID AUG 2014

38. ARCHITECTURE (ZOOMED OUT) EARLI SIG 1 MADRID AUG 2014

39. 1.Utrecht University, NL 2.University Medical Centre Utrecht, NL 3.Szent Istvan University, Hungary 4.University of Tartu, Estonia 5.Universitätsmedizin Charité Berlin, Germany 6.University of California San Francisco, USA 7.Maastricht University, NL 8.Mateum, NL 9.University of Reading, UK 10.Jayway, Denmark 11.NetRom, Rumania/NL THANK you for your attention! www.project-watchme.eu EARLI SIG 1 MADRID AUG 2014