Multilevel and Multi-method Designs Capturing the Effects of Organizational Context and Process in Implementation Research AHRQ 2007 Annual Conference: Improving Healthcare, Improving Lives September 26, 2007 Jeff Alexander, Ph.D. Department of Health Management and Policy University of Michigan School of Public Health

Background Health care provided in organizational context Health care provided in organizational context behavior of clinicians influenced by the organizations in which they work behavior of clinicians influenced by the organizations in which they work policies (e.g. P4P) vary greatly in how they are implemented and what effects they have at the local level. policies (e.g. P4P) vary greatly in how they are implemented and what effects they have at the local level. recognition of the interconnections among components of organizations (clinical teams function within hospitals, interact with other clinical teams, - embeddedness recognition of the interconnections among components of organizations (clinical teams function within hospitals, interact with other clinical teams, - embeddedness

Implementation: the influence of content, context, and process Implementation Content Process Opinion leaders, change champion Systemic processes (e.g., supervisory practices, quality improvement) Organizational learning Triability Innovation type Evidence interpretation and packaging Internal: Organizational culture Organizational structure Practice setting characteristics Local stakeholders (e.g., attitudes and behaviors) Resources External: Networks regulation Economic (e.g., reimbursement) Competition Context

Problems with Implementation Context Measurement and Analyses assigning the same group value to all members of a group assigning the same group value to all members of a group aggregating individual outcomes to the group level aggregating individual outcomes to the group level Separate analyses of organizational and individual phenomena Separate analyses of organizational and individual phenomena

Advantages of Multilevel Designs statistically efficient estimates of regression coefficients statistically efficient estimates of regression coefficients Use of clustering information provides correct standard errors, confidence intervals and significance tests Use of clustering information provides correct standard errors, confidence intervals and significance tests Allows for uneven assessments and different program tenures (for longitudinal studies) Allows for uneven assessments and different program tenures (for longitudinal studies)

Advantages of MLD Measurement at any of the levels of a hierarchy enables examination of whether differences in average outcomes between organizations are explained by factors such as organizational practices/structures, or other characteristics of individual patients or providers Measurement at any of the levels of a hierarchy enables examination of whether differences in average outcomes between organizations are explained by factors such as organizational practices/structures, or other characteristics of individual patients or providers

In a multilevel analysis, variance in the dependent variable is decomposed into within and between group components. Two equations result; In a multilevel analysis, variance in the dependent variable is decomposed into within and between group components. Two equations result; a within-unit model: a within-unit model: Yij=Boj +rij  and a between-unit model: Boj=Goo +Uoj

Potentially Avoidable Adverse Outcomes Risk adjusted mortality QI/TQM Implementation Scope – – Number of participating units – –% hospital staff on QI Teams – – % hospital managers on QI Teams – –% MD/FTEs in QI teams   Intensity _# of statistical process tools _#guidelines in use _ Use of Quality of care data Organizational Context -Resources / Infrastructure for QI – –financial resources/support – – IT – – Leadership support Environmental Context Market Conditions – – Managed care penetration – –Number of competitors – –regulation

Potential applications of MLD Effects of organizational infrastructure on implementation in micro teams Effects of organizational infrastructure on implementation in micro teams Effects of org. culture on individual provider attitudes and behavior (e.g. physician use of clinical guidelines) Effects of org. culture on individual provider attitudes and behavior (e.g. physician use of clinical guidelines) Translational research Translational research Effects of micro-team structure and process on patient outcomes Effects of micro-team structure and process on patient outcomes

Issues with Multilevel Analysis Data requirements Data requirements Statistical power Statistical power Analysis and interpretation issues Analysis and interpretation issues

Multi-method Designs Quantitative-Qualitative Quantitative-Qualitative RCTs-case study RCTs-case study Process studies-outcome studies Process studies-outcome studies Sustainability- long term studies Sustainability- long term studies