1. Reconceptualising balancing measures in
healthcare quality improvement and patient safety
Qualitative analysis of individual and focus group interviews with improvement experts
Scottish Improvement Science
Collaborating Centre
Key Message:
Balancing measures or a ‘balanced accounting’ of both intended and unintended consequences of improvement interventions
Participants supported the implementation of a balanced measurement strategy that focused
on more than the initial improvement goals. Their accounts reflected a desire to have a more ‘balanced accounting’ of the impact of improvement than the more narrowly framed idea
of balancing measures as trade-offs and highlighted the need to consider all consequences of improvement in terms of desirability/undesirability and the extent to which consequences were expected/unexpected when planning improvement.
Madalina Toma, Research Fellow1
Tobias Dreischulte, Lead Pharmacist for Research and Development1,2,4
Nicola M Gray, Associate Director for Programmes and Evaluation1
Diane Campbell, Associate Director for Patient Safety1,3,
Bruce Guthrie, Professor of Primary Care Medicine and SISCC Lead for Primary Care1,4
1 Scottish Improvement Science Collaborating Centre (SISCC), School of Nursing
and Health Sciences, University of Dundee, 11 Airlie Place, Dundee, DD1 4HJ, UK
2 Tayside Medicines Unit, NHS Tayside, Dundee, UK
3 Nurses Residence, Room C3006, Ninewells Hospital, NHS Tayside, Dundee, UK
4 Quality, Safety and Informatics Research Group, Population Health Sciences Division, University of Dundee, Kirsty Semple Way, Dundee DD2 4BF, UK
A refined framework of types of consequences of quality improvement projects
Background
Although the need to be alert to the unintended consequences is being increasingly recognised, there is relatively little research examining how and when health services can and should actively seek to identify the whole range of unintended consequences and how potential balancing measures should be defined, operationalised and implemented.
Goals are typically, measured from the beginning as part of improvement projects and
defined by the initial measurement plan.
Unsure if pleasant surprises could be counted with a balancing measure since they were on the same side of the ‘benefit-harm balance’ as goals
Example: Improving writing and reading skills in secondary schools led to an unexpected reduction in absence rates as a result of better students’ engagement with different activities across the school. Teachers also reported improvements in externalised behaviour and overall attitudes towards learning.
(Interview participant)
Active reflection and/or systematic measurement of potential unpleasant surprises during the ‘improvement pause’
Example: Routine induction of pregnant women at term aimed to decrease the risk of stillbirth or newborn death but led to the use of extra interventions such as continuous fetal monitoring, the need for additional medication and potential emergency Caesareans which in turn increased associated costs and decreased overall patient satisfaction.
(Focus group participant)
Trade-offs are implicitly believed to be smaller in magnitude than the goals and so an ‘acceptable compromise’
Example: Care bundles to prevent ventilator-associated pneumonia aimed at reducing the time patients spend on a ventilator after surgery. However, an increased frequency of unplanned reintubations was expected and confirmed, but on balance care bundle implementation was still judged worthwhile.
Aims
To explore the purpose, design and use of balancing measures in order to develop a framework to inform their use in healthcare quality improvement.
Definitely expected from outset
Define goals and trade-offs; Speculate on surprises; Develop initial measurement plan;
Consider costs;
Definitely unexpected from outset
Improvement pause to define and measure any surprises;
Develop new measurement strategy;
Consider costs;
Data collection 
Two phased research, with 15 semi-structured interviews used in the initial phase to formulate a draft conceptual framework for considering all consequences of improvement activity, and two focus group to refine and elaborate the framework, and to consider its relevance to healthcare quality improvement.
What is the expected direction of change?
Improvement goals
Pleasant surprises
Improvement trade-offs
Unpleasant surprises
Participants
39 professionals with a wide range of roles in improvement science across the UK including clinicians and managers delivering healthcare improvement locally or leading it nationally, improvement advisors, academics supporting improvement activity and public representatives.
Desirable
Undesirable
All four consequences can be measured using either
process or outcome measures
All four consequences can arise in the same area of care targeted by the improvement intervention  or elsewhere in the health and social care system
Data analysis
Both iterative and interpretative, using the framework approach to identify themes and develop conceptual schemes and drawing on the Diffusion of Innovations Literature to develop a coding template for analysis.
Expectations may not be well-articulated before or during implementation or may vary across involved individuals,
so expectation is a spectrum.
Barriers, Facilitators and Measurement Prioritisation
Several participants found the distinction between ‘process’, ‘outcome’ and ‘balancing’ measures in some of the improvement literature confusing, since balancing measures could relate to processes and outcomes depending on the context.
The process of considering unintended consequences and implementing balancing measures was perceived to have value in its own right in terms of improving staff engagement with improvement and overcoming resistance to change.
A balanced accounting of all consequences of improvement was seen as necessary but it had to be set against the cost of additional data collection in already resource-constrained systems, implying a need to prioritise what to measure.
References
Ash JS, Sittig DF, Dykstra RH, et al. Categorizing the unintended sociotechnical consequences of computerized provider order entry. Int J Med Inform 2007;76(suppl 1):21-27.
Bloomrosen M, Starren J, Lorenzi NM, et al. Anticipating and addressing the unintended consequences of health IT and policy: a report from the AMIA 2009 Health Policy Meeting. J Am Med Inform Assoc. 2011;18:82-90.
Campbell EM, Sittig DF, Ash JS, Guappone KP, et al. Types of unintended consequences related to computerized provider order entry J Am Med Inform Assoc 2006;13:  
Ritchie J, Lewis J, McNaughton Nicholls C, et al. Qualitative Research Practice. A guide for social science students and researchers. 2nd ed. London: Sage Publications; 2003.
Rogers, EM. (1995), Diffusion of Innovations, 4th ed., Free Press, New York.
The Scottish Improvement Science Collaborating Centre (SISCC), led by the University of Dundee with NHS Tayside, aims to improve health and care by developing the evidence base for sustainable, large-scale improvement in health and social care. The SISCC brings together researchers, NHS staff, policy makers, educators, and the third sector from across Scotland and internationally in a coordinated and coherent way, to add value to existing investment and deliver a ‘step change’ in improvement knowledge and practice, and maximise benefit for Scotland and beyond. This study is part of ‘Improvement Science Methods’ Research theme which examines the use of measurement in health and social care quality improvement. You can visit us at or follow us on
For more information please contact:
Dr Madalina Toma,
@toma_madalina84