1 Choices and Outcomes: The Effects of Improvement Project Portfolio Choices on Clinical Outcomes Anita L. Tucker Assistant Professor, U. of Pennsylvania Senior Fellow, Leonard Davis Institute of Health Economics Ingrid M. Nembhard Doctoral Candidate, Harvard Business School Harvard Graduate School of Arts and Sciences KLIC 4: INFORMS Conference November 6, 2006 Financial Support from HBS DOR, Wharton’s Fishman Davidson Center In Collaboration with Jeffrey Horbar, Richard Bohmer and Amy Edmondson

2 Background: Health Care in America 1999: Institute of Medicine (IOM) reports 100,000 Americans die annually from preventable medical errors 1998: National Roundtable reports “serious and widespread problems” exist in American medicine”.. Problems of underuse, overuse, misuse.. “Quality of care is the problem.” 2003: RAND Study reports only 55% of patients receive the recommended care for their condition 2000: IOM finds quality problems are a systemic property, requiring process improvement 1996: Dartmouth Atlas Project shows inappropriately geographic variations in care Need for improvement projects in health care organizations The Data: = + Publicity:

3 Improvement Projects Improvement Projects “solve complex organizational problems through the work of formal teams that use a structured improvement method” (Christianson et al., 2005: p. 610)  Healthcare Project Examples Target Area Increase handwashing (Reduce Infections) Reduce heelsticks in Neonates (Pain management) Increase collaboration among MDs and RNs (Staff Retention) Portfolio of Improvement Projects: An organization’s set of improvement projects that are in progress at the same time and draw on the same limited set of human, managerial and financial resources (Cooper, Edgett & Kleinschmidt, 1999; Wheelwright & Clark, 1992)

4 Research Question How should health care organizations structure their portfolios of improvement projects to achieve better outcomes? Hypotheses about Portfolio Choices 1)Number of projects (+) 2)Concentration within a target area (+) 3)Level of evidence for portfolio (+) 4)Novices should start with clinically- oriented portfolio (+) 5)Extent of physician involvement (+)

5  Collaborative of Neonatal Intensive Care Units (NICU)  44 NICUs working together for 2 ½ years (Apr Oct 2004)  Identified 7 target areas for improvement and 93 improvement projects across those areas  Met twice a year to learn QI methods (PDSA cycles), work on developing best practice guidelines and share experiences  In between meetings, implemented practices, conducted site visits to other NICUs and phone conferences Research Setting: Vermont Oxford Network Horbar, J. D. et al, Collaborative quality improvement for neonatal intensive care. Pediatrics 107 (1)

6 Portfolio Project Options Target AreaAim # of practices Sample practice project(s) Pain & Sedation Decrease mean pain score by 50% during NICU stay 10 Pain management during heelsticks Intubation of medications Infection Control Decrease hospital-acquired infections by 25-50% over 2 years 7 Promote hand washing education and practices to prevent nosocomial infection Respiratory Care Decrease chronic lung disease by 10%, and decrease oxygen days, ventilator days, steroid use 15 Vitamin A supplementation Ventilation by Tidal Volume Monitoring OB/NICU Relations Improve maternal & newborn caregiver collaboration: periviability, delivery response, comfort care 6 Design process to increase collaboration and communication during high-risk delivery Staff RetentionDecrease staff turnover by 50%5 Improve nurse-physician collaboration Family-Centered Care Enhance ability to co-ordinate and deliver care so the infant and family needs are met 27 Provide resource materials that depict newborn premature infants’ maturational and postnatal environment. Discharge Planning Embed discharge planning into all aspect of patient care & communication 23 Develop "trigger point" checklist for discharge teaching 93Potential Projects in the Portfolio Clinically-oriented Operationally-oriented **Each NICU indicated which projects they included in their project portfolio to the collaborative sponsor.

7 Unique Portfolios of Practices Excerpt from the practices from Pain Management Implemented during collaborative Working on Hospital 100 Implemented/Working on: Reducing frequency of tracheal suctioning, standarized sucrose analgesia, peripheral vascular procedures, circumcision, post op pain, weaning from opiods VERSUS Hospital 102: Reducing frequency of heelsticks

8 Evidence-base for the portfolio  Level of evidence for all projects within each target area assessed by target area team using Muir-Gray (MG) score to rate articles 1 = strong evidence from at least one systematic review of multiple, well-designed, randomized, controlled trials 2 = properly designed randomized control trial of appropriate size 3 = well-designed trials without randomization 4 = well-designed non-experimental trials 5 = opinions of respected authorities, based on clinical evidence, descriptive studies or reports of expert committees  Evidence base for NICU portfolio = the average MG score of the portfolio

9 Outcome: Standardized Mortality Ratio SMR level of analysis: Babies nested in NICU 1.LOGIT model (clustered by NICU) Outcome Death (0,1) Independent Variables: Established risk factors (Zupanic et al. 2006) 2.Predict probability of death for each baby 3.By NiCU, sum up probability of death, actual deaths 4.Compute ratio: SMR < 1 Outcomes BETTER than expected SMR = 1 Outcomes equal to expected SMR > 1 Outcomes WORSE than expected

10 Means, SD, and correlations (N=27) ^=p<.1; *=p<.05; **=p<.01

11 OLS Regression results (H1, H3, H4) Outcome measureImprovement SMR mortality ( ) Control Variables Staff to beds ratio-0.336* (0.145) Cardiac surgery provider (0.240) History of quality improvement0.569* (0.208) Team size (0.023) Teaching Status0.189 (0.279) Independent Variables Number of projects-0.105* (0.044) Number of projects squared0.004** (0.001) Evidence supporting portfolio-0.526^ (0.293) % of MDs on QI team-3.429* (1.507) Constant4.139** (1.291) Adj. R-squared0.37 F2.62 Sig0.04 df9, 16 N = 26, (std error) ^ significant at 10%; * significant at 5%; ** significant at 1% H1: Supported u-shape H3: Less evidence->Imp H4: Supported

12  H2: Concentrating the within a target area helps Outcome measureImprovement SMR length of stay ( ) Control Variables Staff to beds ratio (0.036) Cardiac surgery provider0.118 (0.071) History of quality improvement0.082 (0.048) Team size0.008 (0.006) Teaching Status (0.077) Independent Variables Number of LOS projects-0.013^ (0.006) Constant (0.161) Adj. R-squared0.23 F2.22 Sig0.09 df6, 19 N = 26, (std error) ^ significant at 10%; * significant at 5%; ** significant at 1% H2: Supported

13  H5: Initial portfolio orientation matters Outcome measureImprovement SMR mortality ( ) Control Variables Staff to beds ratio (0.144) Cardiac surgery provider History of quality improvement Team size Teaching status Independent Variables Number of projects-0.213* (0.082) Number of projects squared0.005* (0.002) Evidence supporting portfolio % of MDs on QI team Clinically oriented portfolio-1.751* (0.697) Constant2.951* (1.008) Adj. R-squared0.16 F (df)7.50 (4,6) Sig0.02 N= 11 (Robust std error) ^ significant at 10%; * significant at 5%; ** significant at 1% H5: Supported

14 Summary and Implications An effective improvement project portfolio: Includes neither too few or too many projects to manage the tradeoff between synergy and distraction Concentrates its efforts within a target area to maximize inter-project learning Focuses on operationally-oriented projects which build performance-improvement capability For novices is clinically-oriented where clearer benchmarks are available Is led by a team with significant physician membership.