Download presentation
Presentation is loading. Please wait.
1
The High Stakes Assessment
Nick Cooling Graham Emblen
2
Learning Objectives Design an effective initial assessment of an underperforming GP registrar Be aware of what a defensible post intervention assessment should look like Reflected on the elements that are important for the developing a national approach to in training assessment
3
Workshop outline What is a high stakes assessment?
Blueprinting initial assessment –small group Post intervention assessment- discussion National guidelines – lets collaborate!
4
Detecting the poorly performing GP registrar
5
The concerning registrar
Trivial or transient problems Impaired Critical error Adverse feedback Unsure or need more evidence Need an assessment
6
Front and back end assesment
Screening assessment Remediation Reassessment
7
Front and back end assessment- different purposes & stakes
Screening assessment (A) –accurate, diagnosis Remediation Reassessment (B)- defensible, outcome based (3Ds) Exclude false negatives in screening assessment. Competence Learning needs Barriers to learning Safety Training
8
The initial assessment
Is there actually a problem? What are the specific problem(s)? What levers would help the most
9
Possible diagnoses Pre-requisites for learning Knowledge or Skills
Clinical reasoning Attitudes & motivation Roles –conflict , acceptance Cultural & Communication Systems Health, personality & family
12
Tools to detect different problems
ECTV Video & medical record review Supervisor report Practice manager report DISQ Standardised patients Personality tests Many, many others
13
Exercise develop a blueprint What are the diagnoses?
A scenario of an underperforming GP registrars ( 1 per group) What are the diagnoses? What tools would help? How many do you need? develop a blueprint
14
Exercise Groups of 5 At least one ME or supervisor with assessment experience 10 min discussion with one case Compile the diagnostic axis and assessment tools Present back to plenary
15
Group feedback
16
The initial assessment
Multiple assessors- assemble your team Multiple instruments
17
Assessment Principles
A feasible sample is required to achieve reliable inferences Bias is an inherent characteristic of expert judgment The validity lies in the users of the instruments, more than in the instruments Formative and summative functions are typically combined Qualitative, narrative information carries a lot of weight Competence is specific not generic Cees van derVlueten
19
Remediation
21
Now for the really high stakes assessment...
22
Post intervention (remediation) assessment
Are they safe? What level of support do they need and where is the best environment to achieve this High stakes- defines type of assessment Same tools or different tools? Remediation process is part of the assessment Get external help
23
Post intervention assessment: 3 Ds
Valid, Reliable, Efficient & Feasible...= Defensible Demonstrate progress/improvement and the registrar is capable of learning Clarify outcome = Direction
24
The big question How big is the risk of this (registrar) performing seriously below the standard in future cases, given his or her history and the newly collected information? Schuwirth & van der Vleuten 2006
26
Its decision time ! Resume usual training
Resume training plus supervision/ education Further remediation Leave training
27
Post intervention assessment: Group discussion
1.Defensible assessment tools- Should they be the same or different? 2. Demonstration of improvement. How do you measure improvement? Standards –are they at the right level? 3. Direction: Outcome Who decides the outcome?
28
Screening of under performing GP registrar (Assessment A)
Post-remediation (Assessment B) Type of assessment Formative (mostly) Summative (mostly) Aims Are they unsatisfactory? What are their learning needs? Are they safe? Do they continue training? Emphasis Discovery Feedback Learning Exclude false negatives Decision Standards Consequence Period of targeted remediation Suspend training time Limit access to training Stakes Low High Content of assessment Traditional tests eg MCQ and standardised patient interviews Predict future clinical ability and outcome Performance based (does) Adequate sampling (all subject areas and all contexts) Level of assessment Pre-vocational ? Vocational ? GPT1 Reliability of assessment 0.8 or above 0.9 or above Who is best involved in assessment RTP MEs & supervisors National panel of experienced MEs
29
Are high stakes assessments your problem?
When the purposes are summative, the problems are primarily systematic leadership problems individual teachers should not be expected to tackle it themselves (Knight, 2001)
30
Take home messages More testing the better Experienced assessors best
Use most appropriate test...and for what they were intended Dont exclude qualitative data
31
Assessment is continuous and supportive
Assessment is continuous and supportive....it is neither formative or summative Lambert Schuwirth 2009
32
Developing national guidelines & resources
33
National guidelines ? Benefits
What national resources would be helpful eg shared MEs, repository of tools, national assessment centre, intensives, expert hotline
34
? A National Assessment (and Remediation) Centre
35
Exercise Buzz session in pairs
How would a national approach to high stakes assessment help your RTP?
36
National approach to High stakes ITA
Standard setting Question banks Improved psychometrics Develop progress testing Pool of expert assessors Develop new assessment tools (greater educational impact and tools for difficult parts of the curriculum) ?Teaching /remediation
37
Where from here?
38
Summary Initial assessment helps diagnose the problem and determine interventions. It can be relatively easily blueprinted Post remediation assessments are more high stakes and require a defensible and possibly national approach
40
Appendix
41
Progress testing Acceptable Average FRACGP exam score Unacceptable
Years in program
42
Cumulative assessment
Assessment nodes throughout training Assessment truly drives learning Emphasis on demonstrating improvement & learning cycles
43
Blueprinting Levels of construction What is to be assessed
Which assessment methods Balance of methods Stakeholder input Training Access to blueprint Monitoring
44
Assessment tools for difficult to assess areas
Insight/ Common sense/Awareness of context Ability to learn Self care/ health Role acceptance Culture System Attitude or motivation
45
Insight/ common sense Keirsey Temperament Sorter Diagnostic Thinking Inventory
46
Paperback 464 pages July 2010, Wiley-Blackwell AUD $79.95
47
Assessment During Training
Pre-commencement Assessment Screening assessment (A) –accurate, transparent diagnosis Remediation Reassessment (B)- defensible, outcome based (3Ds) FRACGP Exams High stakes High stakes Competence Learning needs Barriers to learning Safety Training
48
Report back Diagnoses Best tools Sample number A consensus blue print
49
Can Can these be assessed & measured?
Similar presentations
