The High Stakes Assessment Nick Cooling Graham Emblen

Learning Objectives Design an effective initial assessment of an underperforming GP registrar Be aware of what a defensible post intervention assessment should look like Reflected on the elements that are important for the developing a national approach to in training assessment

Workshop outline What is a high stakes assessment? Blueprinting initial assessment –small group Post intervention assessment- discussion National guidelines – lets collaborate!

Detecting the poorly performing GP registrar

The concerning registrar Trivial or transient problems Impaired Critical error Adverse feedback Unsure or need more evidence Need an assessment

Front and back end assesment Screening assessment Remediation Reassessment

Front and back end assessment- different purposes & stakes Screening assessment (A) –accurate, diagnosis Remediation Reassessment (B)- defensible, outcome based (3Ds) Exclude false negatives in screening assessment. Competence Learning needs Barriers to learning Safety Training

The initial assessment Is there actually a problem? What are the specific problem(s)? What levers would help the most

Possible diagnoses Pre-requisites for learning Knowledge or Skills Clinical reasoning Attitudes & motivation Roles –conflict , acceptance Cultural & Communication Systems Health, personality & family

Tools to detect different problems ECTV Video & medical record review Supervisor report Practice manager report DISQ Standardised patients Personality tests Many, many others

Exercise develop a blueprint What are the diagnoses? A scenario of an underperforming GP registrars ( 1 per group) What are the diagnoses? What tools would help? How many do you need? develop a blueprint

Exercise Groups of 5 At least one ME or supervisor with assessment experience 10 min discussion with one case Compile the diagnostic axis and assessment tools Present back to plenary

Group feedback

The initial assessment Multiple assessors- assemble your team Multiple instruments

Assessment Principles A feasible sample is required to achieve reliable inferences Bias is an inherent characteristic of expert judgment The validity lies in the users of the instruments, more than in the instruments Formative and summative functions are typically combined Qualitative, narrative information carries a lot of weight Competence is specific not generic Cees van derVlueten

Remediation

Now for the really high stakes assessment...

Post intervention (remediation) assessment Are they safe? What level of support do they need and where is the best environment to achieve this High stakes- defines type of assessment Same tools or different tools? Remediation process is part of the assessment Get external help

Post intervention assessment: 3 Ds Valid, Reliable, Efficient & Feasible...= Defensible Demonstrate progress/improvement and the registrar is capable of learning Clarify outcome = Direction

The big question How big is the risk of this (registrar) performing seriously below the standard in future cases, given his or her history and the newly collected information? Schuwirth & van der Vleuten 2006

Its decision time ! Resume usual training Resume training plus supervision/ education Further remediation Leave training

Post intervention assessment: Group discussion 1.Defensible assessment tools- Should they be the same or different? 2. Demonstration of improvement. How do you measure improvement? Standards –are they at the right level? 3. Direction: Outcome Who decides the outcome?

Screening of under performing GP registrar (Assessment A) Post-remediation (Assessment B) Type of assessment Formative (mostly) Summative (mostly) Aims Are they unsatisfactory? What are their learning needs? Are they safe? Do they continue training? Emphasis Discovery Feedback Learning Exclude false negatives Decision Standards Consequence Period of targeted remediation Suspend training time Limit access to training Stakes Low High Content of assessment Traditional tests eg MCQ and standardised patient interviews Predict future clinical ability and outcome Performance based (does) Adequate sampling (all subject areas and all contexts) Level of assessment Pre-vocational ? Vocational ? GPT1 Reliability of assessment 0.8 or above 0.9 or above Who is best involved in assessment RTP MEs & supervisors National panel of experienced MEs

Are high stakes assessments your problem? When the purposes are summative, the problems are primarily systematic leadership problems .... individual teachers should not be expected to tackle it themselves (Knight, 2001)

Take home messages More testing the better Experienced assessors best Use most appropriate test...and for what they were intended Dont exclude qualitative data

Assessment is continuous and supportive Assessment is continuous and supportive....it is neither formative or summative Lambert Schuwirth 2009

Developing national guidelines & resources

National guidelines ? Benefits What national resources would be helpful eg shared MEs, repository of tools, national assessment centre, intensives, expert hotline

? A National Assessment (and Remediation) Centre

Exercise Buzz session in pairs How would a national approach to high stakes assessment help your RTP?

National approach to High stakes ITA Standard setting Question banks Improved psychometrics Develop progress testing Pool of expert assessors Develop new assessment tools (greater educational impact and tools for difficult parts of the curriculum) ?Teaching /remediation

Where from here?

Summary Initial assessment helps diagnose the problem and determine interventions. It can be relatively easily blueprinted Post remediation assessments are more high stakes and require a defensible and possibly national approach

Appendix

Progress testing Acceptable Average FRACGP exam score Unacceptable Years in program

Cumulative assessment Assessment nodes throughout training Assessment truly drives learning Emphasis on demonstrating improvement & learning cycles

Blueprinting Levels of construction What is to be assessed Which assessment methods Balance of methods Stakeholder input Training Access to blueprint Monitoring

Assessment tools for difficult to assess areas Insight/ Common sense/Awareness of context Ability to learn Self care/ health Role acceptance Culture System Attitude or motivation

Insight/ common sense Keirsey Temperament Sorter http://www.keirsey.com/ Diagnostic Thinking Inventory http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2923.1990.tb02650.x/abstract

Paperback 464 pages July 2010, Wiley-Blackwell AUD $79.95

Assessment During Training Pre-commencement Assessment Screening assessment (A) –accurate, transparent diagnosis Remediation Reassessment (B)- defensible, outcome based (3Ds) FRACGP Exams High stakes High stakes Competence Learning needs Barriers to learning Safety Training

Report back Diagnoses Best tools Sample number A consensus blue print

Can Can these be assessed & measured?