1. Cumulative Sum (CUSUM) charts for medical student peripheral venous cannulation; development of a difficulty-adjusted CUSUM
Dr Harry Murgatroyd
SpR Anaesthesia
Leeds Teaching Hospitals Trust
Sumaiyah Kola
Medical Student
Leeds University Medical School

2. I read with interest the articles on CUSUM scoring1,2
six months CUSUM log
useful tool for assessment of initial competence and showing a learning curve
also for continuing self-assessment once competency has been achieved.
Runcie CJ. Assessing the performance of a consultant anaesthetist by control chart methodology. Anaesthesia. 2009; 64(3):

3. CUSUM Chart Developed initially to look at industrial processes
Learning Curves
Monitors performance
What Is a Cusum Chart!?
Developed to look at industrial processes to monitor the production line
Graphical presentation of successes and failures within a practical procedure performed over time
Used to determine competency
Can be used to monitor individual performance
Presentation of a learning curve
Audit of quality clinical practice
Graphical presentation over time
Used to determine competency

4. Theory Set: definition of success / failure acceptable failure rate
unacceptable failure rate
error
Collect:
binary data
Algorithm:
Score falls with success
Score increases with failure
Graph
Boundary Lines

5. Sequential cannulation attempts
Failure seen as a rise in the graph
Success is seen as a fall in the graph
Each point represents a single cannulation attempt
Overall downward trend shows that the actual failure rate here is less than the failure rate set in the CUSUM
Score derived using the CUSUM formula

6. Problems Patient variability Failure rates Standard CUSUM
Constant failure and success rates
Risk adjusted CUSUM
Complicated
Not intuitive
Failure rates
Set by user
Can affect results considerably
This is the slide I started for the DA-CUSUM. I’m not sure what other information you wanted to place here, but feel free to reorganise the slides

7. This graph uses the same data for each line but each set of data is analyised using different failure rates.
The top line uses a failure of 0.1, a very strict failure rate, the result is the line increases as the failure rate is more than 0.1
The middle line has a more lenient failure rate of 0.2, which results in the line remaining nearly level showing that the actual failure rate is close to the 0.2
The lowest line has a leanent failure rate of 0.4, the line drops steeply showing the actual failure rate was much less than this.
Changing the failure rate changes the CUSUM alot

8. Medical student project
Aims
Proficiency at intravenous cannulation
Plot individual CUSUM charts
Develop a ‘difficulty-adjusted’ CUSUM technique
Time Scale:
5 weeks
Aim: to gain experience in anaesthetics, airway management, central lines
Plotting a CUSUM of the attempts and other variables to determine this competency

9. How to insert an intravenous cannula 101

10. Methodology Setting Procedures Teaching hospital
Elective surgical lists
Procedures
Verbal consent
Peripheral venous cannulation
Standard technique
Size of cannula appropriate to surgical procedure
Data collection
Success or failure
Appearance of vein
Size of cannula
Patient awake or anaesthetised
Success - free flow of the infusion fluid into vein after a single skin puncture
Failure - attempting another vein, or an anaesthetist taking over
lastly an acceptable and unacceptable failure rate was agreed upon by numerous anaesthetists
Set variables- mentioning scoring of vein, awake or general

11. Conventional CUSUM Definition of ‘success and failure’
Acceptable and unacceptable failure rates
Consultant consensus
Literature
0.2 and 0.4 respectively
Calculation
Published formulae
Error rates = 0.1
MS Excel
de Oliveira. Anesth Analg 2002;95:411-6.
Williams et al. BMJ 1992;304:

12. Conventional CUSUM Failure rates Upper and lower boundries
0= failure,1= success
Data is plotted sequentially
Running total, CUSUM
Example if “IF” formula in Excel

13. Difficulty Adjusted CUSUM
Difficulty score
Appearance of vein
Cannula size
Awake or anaesthetised
Different failure rates
Two stages
Vein adjusted
All three variables
Intervention line
Average of all prior lines

14. Difficulty Adjusted CUSUM
Vein adjusted
Vein appearance
Acceptable failure rate
Unacceptable failure rate
Visible, palpable
0.15
0.3
Just visible
0.6

15. Three variable methodology
Table shows the scoring of each of the recorded variables. These are then added up to give the total score for the cannulation attempt
Shows the standard CUSUM formula, whilst incorporating different failure rates and scores dependent on the difficulty of the variables recorded.
Spreadsheet showing the final added up scores of the different variables. Hence including the vein, consciousness and cannula size. Using “IF” formulas the correct value of S is selected from the table above and the CUSUM then plotted in the same way as before.

16. Difficulty Adjusted CUSUM
Successful difficult cannulation
Large fall in score
Failed difficult cannulation
Small rise in score
Successful easy cannulation
Small fall in score
Failed easy cannulation
Large rise in score

17. Conventional CUSUM vs Vein DA-CUSUM

18. Conventional CUSUM vs Three Variable DA-CUSUM
So the bottom blue line is a stanard CUSUM plot of my cannulation attempts
The top line in red is one where the difficulty of the cannulation has been taken into account.
The jumped made in the standard CUSUM are all the same but if you look at the red line is jumps are lots of different sizes representing how easy or difficult the cannulation was
The main difference you can see if the red line is much higher as the cannulations I failed on where easyer meaning the graph rose more, and the cannulation I was successful in where also easy to the graph does not go down as much.

19. Three variable DA-CUSUM for two students
Monday Morning

20. POSITIVES of CUSUM LIMITATIONS of CUSUM
Objective
Simple Calculations
Shows improvement in learners
Early detection of poor performance
Allows comparisons between students
LIMITATIONS of CUSUM
Only technical skills
Must have binary outcome
Relies on logbooks and honesty of user
Time consuming
Open to manipulation
Does not show improvements that do not change binary outcome
POSITIVES
Gives a good picture of experience from novice to a skilled performer
Relatively simple calculations that can easily be performed on EXCEL
LIMITATIONS
Only works where performance has a quantitative outcome, use may be limited in cognitive disciplines
Only assesses technical skills, other skills such as communication and interpersonal skills cannot be assessed
Relies on logbooks hence truthfulness of the log may be criticised self reporting logbooks
Time consuming: a minimum number of procedures need to be performed and this may be difficult within a hospital setting, and may take quite a long time overall
Bolson S, Colon M. Int J Health Care Qual Assur 2000;12:
Kestin IG. BJA 1995;75:

21. Difficulty adjusted CUSUM
POSITIVES
Potentially corrects for patient variability
Easier and more intuitive than other methods of adjustment
LIMITATIONS
Failure rates set by the user
The more variables ‘corrected’ for, the more layers of estimation and inaccuracy
Loss of statistical element of conventional CUSUM

22. Summary Easy technique Objective
Handheld devices
Electronic logbooks
Objective
Can be adjusted for patient variability
Allows
Charting of ‘learning curve’
Comparison between practitioners
Identification of poor performance

23. ?