Is the Script-Concordance Test a Valid Instrument for Assessment of Intra-operative Decision-making Skills? Brent Zabolotny 1, Robert Gagnon 2, Bernard Charlin 2 and Sarkis Meterissian 1 1 Division of General Surgery and Center for Medical Education, McGill University 2 Unit for Research and Development in Health Sciences Education, University of Montreal

The Basics of Medical Reasoning Medical reasoning is a hypothetico- deductive process (Ellstein 1978, Barrows et al 1982) involving: 1.Early generation of hypotheses 2.Oriented data collection 3.Decision-making judgements

Problem-solving in Clinical Medicine Well-defined problems are rare: Clear data, clear goals, clear solutions Technical knowledge Ill-defined problems are common: Uncertain data, unclear goals, different solutions Professional knowledge Donald A Schön, The Reflective Practitioner, 1983

What is an Expert? Experts can store, use and retrieve knowledge more efficiently than novices Experts possess elaborated networks of knowledge fitted to the tasks they regularly do These networks are called SCRIPTS (Feltovich 1983, Schmidt 1990, Charlin 2000)

SCRIPT Theory In clinical situations, clinicians fill their working memory with knowledge related to each relevant hypothesis Then a deductive process is used to seek information to either accept or refute each respective hypothesis A test has been developed that measures this deductive-reasoning process – The Script Concordance Test (SCT)

Script Concordance Test A challenging context – Authentic clinical situation – Not enough data or conflict - Relevant options A Likert scale – Capture opinion Scoring system – Takes into account variability of experts – Aggregate scoring method

The Qualities of a Good SCT There must be at least items to achieve a reliability coefficient (Cronbach alpha) of 80 (r =.80) Questions should not lead to a single answer (they must not be clearcut) otherwise the SCT would really be an MCQ Conversely, the answers should not be too widely spread thus indicating a vague question

Evidence that this works?

Purpose To develop and validate a script-concordance test for assessment of clinical reasoning (ie: intra- operative decision-making) in General Surgery Residents.

Methods: SCT Format

Methods: Example (1)

Methods:Example (2)

Methods: Scoring an SCT The scoring grid of an SCT should be derived by administering it to 10 experts (aggregate scoring method) Based on their responses some questions (if too vague or too clearcut) can be eliminated Scoring grid takes into account variability due to the uncertainty of the clinical situation

The Scoring Grid Number of experts choosing answer00541 Score005/104/101/10 Transformed score005/54/51/5 Credit per item

Methods Participants: 36 out of the 40 McGill surgical residents wrote the exam Panel of experts: 10 board-certified general surgeons

Methods Statistical Analysis: - Reliability: Cronbach alpha coefficient - Analysis of item-to-total correlation used to select the best items for the final analysis - Construct validity tested with a one-way ANOVA with post-hoc comparisons test and planned contrasts - All p values at alpha<5% were considered significant

Results Developed 196 questions Face and content validity assessed by 3 independent Board-certified General Surgeons Final exam consisted of the best 100 questions

Results Initial test: 100 items Cronbach alpha : 0.70 Optimisation of the items (elimination of items with an inter-item correlation less than 10%) : final test used for analysis = 62 items Cronbach alpha of the optimised test: 0.85

Results Examination Scores (62-item test) by Resident Level Resident Level nMean Score (%) Std Deviation (%) R R R R R

Results

Discussion The SCT developed in this study demonstrated: 1. excellent internal reliability 2. Good construct validity

Discussion: Questions: 1. Correlation with in-training evaluations? 2. Correlation with oral exam scores? 3. Ideal number of panel members? 4. Makeup of the panel members? subspecialists vs general surgeons 5. Can the SCT identify residents having difficulty with clinical reasoning?

More on Web site:

Results Evaluation of the exam: - 64% found the SCT tested their intra-operative decision-making skills - 50% found the exam initially difficult but 79% found that the exam became easier as they better understood the format - 71% found that the SCT exam was a valuable and educational examination tool