Universiteit Maastricht Barcelona, 6 – 9 July th Ottawa conference on Medical Education

Universiteit Maastricht where do we go from here? Where do we go from here? Lambert Schuwirth new challenges in assessment of clinical competence

Universiteit Maastricht overview of the presentation -why this presentation Where do we go from here? - developments in competence assessment - implications for research - developments in performance assessment

Universiteit Maastricht overview of the presentation Where do we go from here? -why this presentation - developments in competence assessment - implications for research - developments in performance assessment

Universiteit Maastricht experts vs novices: context vs concept Where do we go from here? FzFz FfFf 1,25 kg

Universiteit Maastricht overview of the presentation Where do we go from here? -why this presentation - developments in competence assessment - implications for research - developments in performance assessment

Universiteit Maastricht Where do we go from here? simple isn’t it? test pass fail competent incompetent

Universiteit Maastricht Where do we go from here? simple, it isn’t competent incompetent pass fail

modern approaches to assessment Universiteit Maastricht 1re-evaluate the nature of medical competence 2shift towards more student-centred assessment 3include general professional competencies 4broaden the measurement time frame Where do we go from here?

1 trait orientated view on medical competence Universiteit Maastricht knowledge instrument skills instrument problem solving instrument attitudes instrument Where do we go from here?

one instrument for one trait Universiteit Maastricht Where do we go from here? instruments traits ABCD

one instrument for one trait Universiteit Maastricht Where do we go from here?

old model of expertise: problems Universiteit Maastricht Where do we go from here? more variation within than between methods

1 trait orientated view on medical competence Universiteit Maastricht knowledge instrument skills instrument problem solving instrument attitudes instrument Where do we go from here?

1 trait orientated view on medical competence Universiteit Maastricht knowledge instrument skills instrument problem solving instrument attitudes instrument Where do we go from here? instrument

old model of expertise: problems Universiteit Maastricht Where do we go from here? more variation within than between methods content determines the assessed competence

Universiteit Maastricht knows knows how shows how does Miller's pyramid Where do we go from here?

Universiteit Maastricht 2 shift towards student-centred assessment -self-assessment -peer-assessment -co-assessment self-evaluation skills assessing the unmeasurable peer pressure? peer-evaluation skills self, peer and co-evaluation skills peer pressure? joint responsibility Where do we go from here?

Universiteit Maastricht 3 include general professional competencies dealing with tasks dealing with others dealing with oneself planning and time management information gathering co-operation communicating giving feedback and criticising self-responsibility self-evaluation self-reflection Where do we go from here?

Universiteit Maastricht 4 broaden the measurement time frame specific test occasions - specific preparation of students - inflexibility longitudinal testing - unwanted influence on learning - certain aspects unmeasurable - no specific preparation - congruent with flexible curricula - wanted influence on learning Where do we go from here?

Universiteit Maastricht lessons learnt Where do we go from here? good assessment draws the whole picture no single method is a panacea the content is important, not the format too much structure does not work sampling strategies are essential

Universiteit Maastricht concept and context: subjectivity Where do we go from here? experts LambertMozart

Universiteit Maastricht Where do we go from here? sampling strategies are essential how do you feel now how did you feel 5’ ago how did you feel 10’ ago how will you feel in 5’ how will you feel in 10’ what does your partner about think how you feel LambertNeil Dale

Universiteit Maastricht lessons learnt Where do we go from here? sampling strategies are essential good assessment shows the whole picture no single method is a panacea the content is important, not the format too much structure does not work

Universiteit Maastricht overview of the presentation Where do we go from here? -why this presentation - developments in competence assessment - implications for research - developments in performance assessment

Universiteit Maastricht definitions Where do we go from here? competence what professionals can do in controlled representations of professional practice, performing at the maximum of their ability performance what professionals actually do in their professional practice

Universiteit Maastricht Where do we go from here? 1 clinical expertise 2 communication skills 3 collaboration skills 4 management skills 6 educational skills 7 professional attributes 8 personal health & self care 5 personal development current approaches: still trait orientated

Universiteit Maastricht Where do we go from here? instruments chart review data review interviews rating scales surveys video portfolios incognito simulated patiens critical incident analyses

Universiteit Maastricht Where do we go from here? but……. so……. no direct link between instrument and trait domains are not discrete entities trait-orientated approach did not work in competence assessment modern approaches define competence and performance in terms of professional roles

teacher Universiteit Maastricht Where do we go from here? patient care person professional role-orientated goals scientist health care management

Universiteit Maastricht Where do we go from here? 1 role – 1 instrument? instruments roles ABCD

Universiteit Maastricht Where do we go from here? multi-modal assessment roles ABCD sources of information

Universiteit Maastricht Where do we go from here? how: seeing the whole picture

Universiteit Maastricht Where do we go from here? how: approaches - variation in instruments needed - performance often involves subjective aspects -collation of information based on content-equal bits and not on method-equal bits no flight into structuring, but in sampling

Universiteit Maastricht Where do we go from here? how: organisation plan purposes elements of defensibility quality control measures pathways implementation administration consequences assessee

Universiteit Maastricht Where do we go from here? observations assessment is not mainly a measurement problem but more an (educational) design problem. instrument is not synonymous with a competence no single instrument is the panacea, a variety is needed: an assessment programme content is more important than format: it is the concept that counts, not the format

Universiteit Maastricht Where do we go from here? suggestions assessment is not mainly a measurement problem but more an educational design problem. - built-in quality control procedures and bodies - sampling in stead of (over) structuring - variation of methods, compensation, regulations - saturation of information approach - training of students, teachers, examiners, board members masterplan of the assessment system

Universiteit Maastricht overview of the presentation Where do we go from here? -why this presentation - developments in competence assessment - implications for research - developments in performance assessment

Universiteit Maastricht Where do we go from here? trait orientated assessment as a psychological testing problem - highly quantitative methodologies - reliability of methods - construct validity +++

Universiteit Maastricht Where do we go from here? trait orientated assessment as a instructional design problem - quantitative methodologies - reproducibility of methods - construct validity - qualititative methodologies - consequential validity - research into assessment systems - direct validity ±

Universiteit Maastricht Where do we go from here? observations research of assessment systems is more important now than research into single methods. methodology is not synonymous to research question no single one study/methodology will provide the definitive answer, a variety is needed the methodological concept is important not the specific analysis or method

Universiteit Maastricht Where do we go from here? research agenda what constitutes good assessment programmes? which building blocks should be included? how should information be collated: qualitatively / quantitatively? assessment as a programme: who should do what?

Universiteit Maastricht Where do we go from here? research agenda g-analyses reproducibility of (portfolio) judgments? other statistical approaches sequential testing? organisational approaches?

Universiteit Maastricht Where do we go from here? Research agenda how to “score” qualitative judgments? quality elements of multi-modal assessment identifiers for profile scores what influences the thinking of judges? fraud/plagiarism (internet) flexible adaptation of teaching to assessment outcomes

Universiteit Maastricht Where do we go from here? research agenda roles of format, content, scheduling and regulations consequential validity / educational impact what influences teacher behaviour

Universiteit Maastricht Where do we go from here? research agenda consequential validity / educational impact

Universiteit Maastricht Where do we go from here? research agenda how to change a programme? research into assessment programmes stakeholder involvement how are the instruments combined? how to use parts in stead of whole instruments? rules of design

Universiteit Maastricht Where do we go from here? SUMMARY overview of the presentation -why this presentation - developments in competence assessment - implications for research - developments in performance assessment

Universiteit Maastricht Where do we go from here? conclusions: in (education), assesment and research strength comes from collation of results the method is not synonymous with the concept no single method is a panacea translation of concepts into methods is useful

Universiteit Maastricht Where do we go from here? Thank you