Applying Multiple Frameworks and Theories in Implementation Research Jeffrey Smith Implementation Research Coordinator Mental Health QUERI

“In theory there is no difference between theory and practice… in practice there is.” Yogi Berra

Definitions Formative evaluation (FE) Formative evaluation (FE)  Rigorous assessment process designed to identify potential and actual influences on the progress and effectiveness of implementation efforts (Stetler et al, JGIM 2006; 21:(Suppl 2):S1-8.) Summative evaluation Summative evaluation  Systematic process of collecting data on impacts, outputs, products, outcomes for a study  Used to assess success, effectiveness, or goal achievement

MH QUERI Approach to Implementation Design interventions based on theory and/or results from formative evaluation Design interventions based on theory and/or results from formative evaluation Conduct formative evaluation Conduct formative evaluation –identify determinants of care –tailor intervention design and implementation to local context –assess barriers to implementation Use external facilitation techniques Use external facilitation techniques –Identify and test new approaches and methods for overcoming barriers Conduct summative or impact evaluation Conduct summative or impact evaluation

An Approach to Using Theory for Implementation Planning Select theory of planned behavior change Assess fit with initial theory Identify potential strategies for achieving change Select interventions that fit with planned strategies (based on theory) Identify intervention tools that fit both strategy and theory Launch intervention using identified tools and strategies Evaluate effectiveness of intervention, strategies, tools From: Sales A, Smith JL, Curran G, Kochevar L. Models, strategies and tools: The role of theory in implementing evidence-based findings into health care practice. Journal of General Internal Medicine 2006; 21:S43-49.

Antipsychotic Treatment Improvement Program (ATIP) Goals Goals –Improve adherence to CPGs for psychosis –Reduce use of very high doses of antipsychotic medications –Encourage use of newer “atypical” antipsychotic medications for non-responders to conventional antipsychotic medications

ATIP Application of Frameworks, Theory and Planning Models (cont) Social Influence Theory, Diffusion of Innovation, Social Cognitive Theory Social Influence Theory, Diffusion of Innovation, Social Cognitive Theory – –Utilize influential local clinician leaders (opinion leaders) to inform other clinical staff about evidence-based antipsychotic medication management, model-targeted prescribing behaviors, and motivate practice change Complexity Theory – –Initial conditions in HCOs are important in intervention planning, but HCOs are adaptive and change over time – –Need mechanism for assisting clinical partners in modifying or adapting initial intervention strategy based on changing circumstances, initial success

ATIP Application of Frameworks, Theory and Planning Models (cont) Promoting Action Research Implementation in Health Services (PARiHS) Promoting Action Research Implementation in Health Services (PARiHS) – –Use external facilitation techniques to maintain contact with clinical opinion leader to assist with problem-solving and addressing challenges to intervention implementation PRECEDE Planning Model (Predisposing, Reinforcing, and Enabling Constructs in Education Diagnosis and Evaluation) – –Address predisposing factors (eg, knowledge, attitudes) that influence EBP adoption – –Enable providers to follow guideline recommendations at the point of care – –Apply social incentives through performance audit/feedback to reinforce providers’ implementation of EBPs

Facilitation is a process of interactive problem-solving and support to meet specific implementation goals, which occurs in the context of a recognized need for improvement and a supportive interpersonal relationship. “Refined” Description of Facilitation Based on VA QUERI Experience * * Stetler CB, et al. Implementation Science 2006; 1:23.

Operationalization of External Facilitation in MH QUERI Facilitator Facilitator –Maintains regular contact w/ local QI leader / team communication communication Telephone Telephone Participate in QI Team meetings (as feasible) Participate in QI Team meetings (as feasible) –Assist in goal-setting; monitor implementation of project tools/strategies; performance on clinical QI goals –Identify and problem-solve barriers to implementation –Assist in adapting tools/strategies as needed or suggested by site partners to meet project goals

ATIP Intervention Components Intervention Component Supporting Theory or Planning Model Clinical Opinion Leader Diffusion of Innovation, Social Cognitive Theory, Social Influence Theory External Facilitation PARiHS, Complexity Theory Psychosis Guidelines Help File PRECEDE (predisposing) Pocket Card on Antipsychotic Treatment PRECEDE (enabling) Pharmacy Order-Entry Reminder on Antipsychotic Dose Recommendations PRECEDE (enabling) Clinical Reminder on Metabolic Side Effects Associated with Olanzapine PRECEDE (enabling) Audit/Feedback Performance Reports on Antipsychotic Dosing / Side Effect Monitoring PRECEDE (reinforcing)

Formative Evaluation: Barriers and Lessons Learned Physicians do not always agree on who is an opinion leader; some sites may have no opinion leader Physicians do not always agree on who is an opinion leader; some sites may have no opinion leader Some sites had poorly developed formal and informal social networks among physicians Some sites had poorly developed formal and informal social networks among physicians A focus on physicians only as agents of change A focus on physicians only as agents of change How much directive should be given to the opinion leaders concerning how to influence attitudes and behaviors? How much directive should be given to the opinion leaders concerning how to influence attitudes and behaviors? From: Curran GM, Thrush CR, Smith JL, et al. Jt Commission J Qual Safety 2005; 31(12):

Summative Evaluation ATIP intervention improved antipsychotic medication management in concordance with CPGs Reduced pharmacy costs for antipsychotics Participating clinicians reported positive experiences with ATIP educational materials and clinical support tools

Summary Application of multiple frameworks/theories in guiding intervention design and implementation can be successful in implementing EBPs Application of multiple frameworks/theories in guiding intervention design and implementation can be successful in implementing EBPs – accommodates tailoring to setting when combined with formative evaluation acknowledges there are generally multi-level determinants to complex, clinical QI issues acknowledges there are generally multi-level determinants to complex, clinical QI issues – organizational-level – team-level – interpersonal-level – individual-level

Multiple theory approach Strengths – useful in designing multifaceted interventions to influence multi-level determinants of care (flexible) – allows integration of theory, knowledge, methods from multiple disciplines (multidisciplinary) Limitation – Can be unwieldy… need to provide rationale for applying multiple theory approach, and rationale for selecting the specific frameworks / theories applied Key Guidance on Evaluation – combine with rigorous formative evaluation – conduct summative (impact) evaluation to assess intervention effectiveness on key study outcomes – confirm, refute or propose refinements to selected theory(ies) based on study findings

Developmental Identify determinants of current practice Identify potential barriers / facilitators Assess feasibility of proposed intervention Integrate findings into intervention design and refinement prior to implementation Implementation-Focused Assess discrepancies between implementation plan and execution, exploring issues of fidelity, intensity, exposure Understand and document nature and implications of local adaptation Progress-Focused Monitor impacts and indicators of progress toward project goals Use data to inform need for modifying or refining original strategy Provide positive reinforcement to high performers; negative reinforcement to low performers Interpretive Assess intervention usefulness/value from stakeholder perspective Elicit stakeholder recommendations for further intervention refinements Assess satisfaction with intervention and implementation process Identify additional barriers / facilitators Stages of FE Post- Implementation Pre- Implementation Implementation