Cumulative Sum (CUSUM) charts for medical student peripheral venous cannulation; development of a difficulty-adjusted CUSUM Dr Harry Murgatroyd SpR Anaesthesia Leeds Teaching Hospitals Trust Sumaiyah Kola Medical Student Leeds University Medical School

I read with interest the articles on CUSUM scoring1,2 six months CUSUM log useful tool for assessment of initial competence and showing a learning curve also for continuing self-assessment once competency has been achieved. Runcie CJ. Assessing the performance of a consultant anaesthetist by control chart methodology. Anaesthesia. 2009; 64(3): 293-296

CUSUM Chart Developed initially to look at industrial processes Learning Curves Monitors performance What Is a Cusum Chart!? Developed to look at industrial processes to monitor the production line Graphical presentation of successes and failures within a practical procedure performed over time Used to determine competency Can be used to monitor individual performance Presentation of a learning curve Audit of quality clinical practice Graphical presentation over time Used to determine competency

Theory Set: definition of success / failure acceptable failure rate unacceptable failure rate error Collect: binary data Algorithm: Score falls with success Score increases with failure Graph Boundary Lines

Sequential cannulation attempts Failure seen as a rise in the graph Success is seen as a fall in the graph Each point represents a single cannulation attempt Overall downward trend shows that the actual failure rate here is less than the failure rate set in the CUSUM Score derived using the CUSUM formula

Problems Patient variability Failure rates Standard CUSUM Constant failure and success rates Risk adjusted CUSUM Complicated Not intuitive Failure rates Set by user Can affect results considerably This is the slide I started for the DA-CUSUM. I’m not sure what other information you wanted to place here, but feel free to reorganise the slides

This graph uses the same data for each line but each set of data is analyised using different failure rates. The top line uses a failure of 0.1, a very strict failure rate, the result is the line increases as the failure rate is more than 0.1 The middle line has a more lenient failure rate of 0.2, which results in the line remaining nearly level showing that the actual failure rate is close to the 0.2 The lowest line has a leanent failure rate of 0.4, the line drops steeply showing the actual failure rate was much less than this. Changing the failure rate changes the CUSUM alot

Medical student project Aims Proficiency at intravenous cannulation Plot individual CUSUM charts Develop a ‘difficulty-adjusted’ CUSUM technique Time Scale: 5 weeks Aim: to gain experience in anaesthetics, airway management, central lines Plotting a CUSUM of the attempts and other variables to determine this competency

How to insert an intravenous cannula 101

Methodology Setting Procedures Teaching hospital Elective surgical lists Procedures Verbal consent Peripheral venous cannulation Standard technique Size of cannula appropriate to surgical procedure Data collection Success or failure Appearance of vein Size of cannula Patient awake or anaesthetised Success - free flow of the infusion fluid into vein after a single skin puncture Failure - attempting another vein, or an anaesthetist taking over lastly an acceptable and unacceptable failure rate was agreed upon by numerous anaesthetists Set variables- mentioning scoring of vein, awake or general

Conventional CUSUM Definition of ‘success and failure’ Acceptable and unacceptable failure rates Consultant consensus Literature 0.2 and 0.4 respectively Calculation Published formulae Error rates = 0.1 MS Excel de Oliveira. Anesth Analg 2002;95:411-6. Williams et al. BMJ 1992;304:1359-61.

Conventional CUSUM Failure rates Upper and lower boundries 0= failure,1= success Data is plotted sequentially Running total, CUSUM Example if “IF” formula in Excel

Difficulty Adjusted CUSUM Difficulty score Appearance of vein Cannula size Awake or anaesthetised Different failure rates Two stages Vein adjusted All three variables Intervention line Average of all prior lines

Difficulty Adjusted CUSUM Vein adjusted Vein appearance Acceptable failure rate Unacceptable failure rate Visible, palpable 0.15 0.3 Just visible 0.6

Three variable methodology Table shows the scoring of each of the recorded variables. These are then added up to give the total score for the cannulation attempt Shows the standard CUSUM formula, whilst incorporating different failure rates and scores dependent on the difficulty of the variables recorded. Spreadsheet showing the final added up scores of the different variables. Hence including the vein, consciousness and cannula size. Using “IF” formulas the correct value of S is selected from the table above and the CUSUM then plotted in the same way as before.

Difficulty Adjusted CUSUM Successful difficult cannulation Large fall in score Failed difficult cannulation Small rise in score Successful easy cannulation Small fall in score Failed easy cannulation Large rise in score

Conventional CUSUM vs Vein DA-CUSUM

Conventional CUSUM vs Three Variable DA-CUSUM So the bottom blue line is a stanard CUSUM plot of my cannulation attempts The top line in red is one where the difficulty of the cannulation has been taken into account. The jumped made in the standard CUSUM are all the same but if you look at the red line is jumps are lots of different sizes representing how easy or difficult the cannulation was The main difference you can see if the red line is much higher as the cannulations I failed on where easyer meaning the graph rose more, and the cannulation I was successful in where also easy to the graph does not go down as much.

Three variable DA-CUSUM for two students Monday Morning

POSITIVES of CUSUM LIMITATIONS of CUSUM Objective Simple Calculations Shows improvement in learners Early detection of poor performance Allows comparisons between students LIMITATIONS of CUSUM Only technical skills Must have binary outcome Relies on logbooks and honesty of user Time consuming Open to manipulation Does not show improvements that do not change binary outcome POSITIVES Gives a good picture of experience from novice to a skilled performer Relatively simple calculations that can easily be performed on EXCEL LIMITATIONS Only works where performance has a quantitative outcome, use may be limited in cognitive disciplines Only assesses technical skills, other skills such as communication and interpersonal skills cannot be assessed Relies on logbooks hence truthfulness of the log may be criticised self reporting logbooks Time consuming: a minimum number of procedures need to be performed and this may be difficult within a hospital setting, and may take quite a long time overall Bolson S, Colon M. Int J Health Care Qual Assur 2000;12:433-438. Kestin IG. BJA 1995;75:805-809.

Difficulty adjusted CUSUM POSITIVES Potentially corrects for patient variability Easier and more intuitive than other methods of adjustment LIMITATIONS Failure rates set by the user The more variables ‘corrected’ for, the more layers of estimation and inaccuracy Loss of statistical element of conventional CUSUM

Summary Easy technique Objective Handheld devices Electronic logbooks Objective Can be adjusted for patient variability Allows Charting of ‘learning curve’ Comparison between practitioners Identification of poor performance

?