1. www.project-watchme.eu @Project_WatchMe Workplace-based e-Assessment Technology for Competency-based Higher Multi-professional Education.

2. 1.Utrecht University, NL 2.University Medical Centre Utrecht, NL 3.Szent Istvan University, Hungary 4.University of Tartu, Estonia 5.Universitätsmedizin Charité Berlin, Germany 6.University of California San Francisco, USA 7.Maastricht University, NL 8.Mateum, NL 9.University of Reading, UK 10.Jayway, Denmark 11.NetRom, Rumania/NL

3. WATCHME’S AIM Solve efficiency and quality problems of workplace-based feedback and assessment by means of a mobile electronic portfolio system that is enhanced with a feedback module and a visualization module.

4. - Develop complex competences - Integrated in context - Demands long learning trajectories in workplace - Deliberate practice: feedback and reflection

5. FROM 90S ONWARDS: ELECTRONIC PORTFOLIOS Portfolios contain selected evidence of performance and products accompanied by reflections. Reflective ‘log’ of trainee over longer time span Repository of evidence regarding performance etc Different instruments and methods

6. Example Digital portfolio, example veterinary education Aggregation of data points  Mini-CEX (clinical evaluation exerc.)  Multisource feedback  Case reports  Knowledge tests  Personal development plan

7. ELECTRONIC PORTFOLIOS MIXED SUCCESS Underuse of potential data and not very well tailored to worksituations (not timely). Drawbacks: Authentic, personal unique character of evidence, difficults scoring. Full standardization impossible, asks for interpretation of assessor (no golden standard or external criterion). Impacts reliability and validity of the assessment. Limited impact on learning Low motivation of students

8. Medical expert Communicator Collaborator Manager Health advocate Scholar Professional CRITICS COMPETENCY-BASED ASSESSMENT MOST CRITICS COME DOWN TO THE ANALYTIC APPROACH (E.G. CANMEDS) Source: Ten Cate

9. Medical expert Communicator Collaborator Manager Health advocate Scholar Professional THE ANALYTIC APPROACH With nursing staff With family With patients With colleagues With trainees … Source: Ten Cate

10. Medical expert Communicator Collaborator Manager Health advocate Scholar Professional THE ANALYTIC APPROACH With nursing staff With family With patients With colleagues With trainees … Consultation Breaking bad news Explain medication With children With elderly … Source: Ten Cate

11. Medical expert Communicator Collaborator Manager Health advocate Scholar Professional THE SYNTHETIC FRAMEWORK APPROACH EPA1EPA2EPA3EPA4EPA5 Source: Ten Cate

12. ENTRUSTABLE PROFESSIONAL ACTIVITY Task based instead of construct based approach Crucial question: would I entrust this candidate unsupervised with this task? (with my sick mother, animal or teach my daughter/son…) An EPA is a task that an individual can be trusted to perform unsupervised, in a given professional context, once sufficient competence has been demonstrated. International Competency-Based Medical Education Collaborators, March 18, 2014 Source: Ten Cate

13. EXAMPLES ENTRUSTABLE PROFESSIONAL ACTIVITY (EPA) Conducting patient hand-overs Anesthetic management a patient Conducting a normal, low risk delivery Interviewing adolescents regarding high risk health behavior

14. SUPERVISION LEVELS OPERATIONALIZED 1.Acting is not allowed: more observation 2.Acting is allowed under direct supervision: supervision physically present; proactive supervision 3.Acting is allowed under indirect supervision: supervision not physically present, but quickly available if needed; reactive supervision 4.Acting is allowed “unsupervised”:under clinical oversight; distant/backstage supervision; post hoc report 5.Supervision may be provided to juniors (to junior residents, to medical students at level 2 or 3) A formal acknowledgment happens at level 4 Source: Ten Cate

15. COMBINING THE COMPETENCY-FRAMEWORK WITH ENTRUSTABLE PROFESSIONAL ACTIVITIES Medical expert Communicator Collaborator Manager Health advocate Scholar Professional ++ + + + + + + + + + + + + EPA1EPA2EPA3EPA4EPA5 Assessment Source: Ten Cate

16. COMBINE EPA APPROACH WITH LEARNING ANALYTICS Measurement, collection, analysis and reporting of data about trainees in their contexts, for the purpose of understanding, and optimising learning and the utilising of environments in which it occurs (Solar, 2013). Application of probabilistic student models that enable tailored visualised feedback based on multi sorted assessments Bayesian models provide detailed model of trainees’ entrustability

17. ELECTRONIC PORTFOLIO EARLI SIG 1 MADRID AUG 2014

27. EXAMPLE JUST-IN-TIME FEEDBACK Before workplace-based assessment is performed EPASS displays just-in-time feedback concerning learner. E.g.: “remind your communication skills towards patient. Explain treatment in simple words”.

28. EXAMPLE JUST-IN-TIME FEEDBACK

29. QUESTIONS OR WOULD YOU LIKE TO HAVE A LOOK AT EPASS? CONTACT ME DURING LUNCH BREAK! THANK YOU FOR YOUR ATTENTION!

30. ARCHITECTURE