Evolution of the Revolution: How Can Evidence-Based Practice Work in the Real World? Bruce F. Chorpita Wing Institute Annual Summit on Evidence Based Special Education April 24, 2008

Acknowledgments

The Story of John A Tale of Two Cultures

This is John.

Evidence Based Practice: Not As Easy As It Looks

Attributes of Innovations 1.Relative Advantage 2.Compatibility 3.Complexity 4.Observability Diffusion of Innovation Communication Channels 1.Mass Media 2.Interpersonal Change Agent Properties 1.Effort 2.Homophily Rate of Adoption Adapted from Rogers (1995)

Diffusion Curve Time 

ATTRIBUTES OF THE INNOVATION

Practitioner Concerns n Fixed content n Fixed intensity n Fixed length n Single target approach n Replacement n Empty cell problem n Crowded cell problem n Expiration problem Aarons (2004); Addis & Krasnow (2000); Addis, Wade, & Hatgis (2004); Chorpita, Daleiden, & Weisz (2005); Kimhan & Chorpita (2006); Persons (1995)

Some EBS Effect Sizes (Chorpita et al. 2002) n CBT for anxiety: ES =1.05 n Exposure for anxiety: ES=2.02 n Modeling for anxiety: ES=0.55 n CBT with Parent/Child for anxiety: ES=1.68 n Behavior Therapy for ADHD: ES=1.24 n CBT for depression: ES=1.74 n IPT for depression: ES=1.51 n Relaxation for depression: ES=1.48 n Parent training for ODD: ES=0.89 n MST for delinquency: ES=0.50 Diffusion strategies: Relative Advantage

Comparison with Usual Care (Weisz et al. 1995) Diffusion strategies: Relative Advantage

Throw out the Bath Water: Keep the Baby n Move away from the idea that “Problem A gets Treatment B” n We decided to map the decisions made at the system level – not just the IF- THEN of treatment selection n Goal: To build an Evidence Based System

Where should we treat the client? Service Setting

How should we treat the client? Therapeutic Practices Service Setting

Are we providing quality service to the client? Therapeutic Practices Service Setting Treatment Integrity

Is the client getting better? Therapeutic Practices Service Setting Client Progress Treatment Integrity

Who should treat the client? Therapeutic Practices Service Setting Treatment Team Client Progress Treatment Integrity

How should we manage the treatment? Therapeutic Practices Service Setting Treatment TeamSupervision Client Progress Treatment Integrity

Treatment programs formalize these elements Therapeutic Practices Service Setting Treatment TeamSupervision Client Progress Treatment Integrity

Selecting a program structures other decisions Tx Program Selection Therapeutic Practices Service Setting Treatment TeamSupervision Client Progress Treatment Integrity

Selecting a program structures other decisions Tx Program Selection Therapeutic Practices Service Setting Treatment TeamSupervision Client Progress Treatment Integrity Multisystemic Therapy (MST) Example Home & Community Based TAMS & SAMS Instrumental & Ultimate Outcomes MST 4 – 5 Members Team Supervisor Cross-Team Supervisor Family Therapy, etc.

Tx Program Selection Valid alternatives to deciding by program? Therapeutic Practices Service Setting Treatment TeamSupervision Client Progress Treatment Integrity

How should we make decisions? Tx Program Selection Therapeutic Practices Service Setting Treatment TeamSupervision Client Progress Treatment Integrity

How should we make decisions? General Services Research Tx Program Selection Therapeutic Practices Service Setting Treatment TeamSupervision Client Progress Treatment Integrity Evidence-Based Services Model

How should we make decisions? Case-Specific Historical Information Causal Mechanism Research Tx Program Selection Therapeutic Practices Service Setting Treatment TeamSupervision Client Progress Treatment Integrity Individualized Case Conceptualization Model

How should we make decisions? Local Aggregate Evidence Case-Specific Historical Information Tx Program Selection Therapeutic Practices Service Setting Treatment TeamSupervision Client Progress Treatment Integrity Practice-Based Evidence Model

The Full System Model General Services Research Local Aggregate Evidence Case-Specific Historical Information Causal Mechanism Research Tx Program Selection Therapeutic Practices Service Setting Treatment TeamSupervision Client Progress Treatment Integrity

The Phases of Evidence 1. Data: Discretely identifiable units 2. Information: Data in a context that provides it meaning 3. Knowledge: Information helpful to decision- making 4. Wisdom: Knowing when to apply our knowledge c.f., Speigler, I. (2000). Knowledge management: A new idea or a recycled concept? Communications of the Association for Information Systems, 3, 1 – 23.

The Phases of Evidence: Example 1. Data: Information: 70º F 3. Knowledge: It is warm enough to wear shorts. 4. Wisdom: I am giving a professional talk today, so my knowledge that it is warm enough to wear shorts is irrelevant to my attire.

General Services Research: Turning Data into Knowledge I n Meta Analysis of Children’s Treatment Research –Over 350 studies, 40 years –Largest meta-analysis to date –Made dynamic for providers Diffusion strategies: Compatibility, Observability, Complexity

How do we get this knowledge to the treatment team? General Services Research Local Aggregate Evidence Case-Specific Historical Information Causal Mechanism Research Tx Program Selection Therapeutic Practices Service Setting Treatment TeamSupervision Client Progress Treatment Integrity

Other Options n SAMHSA NREP n Published meta-analyses n Evidence based practice guidelines n Paper/web reports –Hawaii CAMHD Biennial Report –Hawaii’s “Blue Menu”

The Blue Menu is being substantially updated based on hundreds of additional studies. Some problem areas that are now omitted will reappear in updated form on future editions. Problem AreaLevel 1- BEST SUPPORT Level 2- GOOD SUPPORT Level 3- MODERATE SUPPORT Level 4- MINIMAL SUPPORT Level 5- NO SUPPORT Anxious or Avoidant Behaviors Cognitive Behavior Therapy, Education, Exposure, Modeling Assertiveness Training, Cognitive Behavior Therapy and Medication, Cognitive Behavior Therapy with Parents, Hypnosis, Relaxation NoneBiofeedback, Play Therapy, Psychodynamic Therapy, Rational Emotive Therapy Client Centered Therapy, EMDR, Relationship Counseling, Teacher Psychoeducation [poorly tested: CBT with Parents Only; CBT with Child and CBT with Parent separately] Attention and Hyperactivity Behaviors Behavior Therapy and Medication, Self Verbalization Biofeedback, Contingency Management, Education, Parent Management Training, Parent Management Training and Problem Solving, Physical Exercise, Relaxation and Physical Exercise, Relaxation and Physical Exercise, Social Skills and Medication, Working Memory Training NoneParent Management Training and Social Skills, Relaxation, Social Skills Client Centered Therapy [poorly tested: Parent Management Training and Self-Verbalization, Self Control Training, Self Verbalization and Medication, Skill Development] Autistic Spectrum Disorders Intensive Behavior Therapy, Intensive Communication Training None [poorly tested: Auditory Integration Training] Delinquency and Disruptive Behavior Assertiveness Training, Cognitive Behavior Therapy, Contingency Management, Multisystemic Therapy, Parent Management Training, Social Skills Anger Control, Client Centered Therapy, Communication Skills, Functional Family Therapy, Multidimensional Treatment Foster Care, Parent Management Training and Problem Solving, Problem Solving, Rational Emotive Therapy, Relaxation, Transactional Analysis Outreach Counseling, Peer Pairing, Self Control Training Physical Exercise, Stress Inoculation Education, Exposure, Family Systems Therapy, Group Therapy (!!), Life Skills, Project CARE (!!),Skill Development [poorly tested: Catharsis, Collaborative Problem Solving, Family Empowerment, Physical Exercise, Psychodynamic Therapy, Self Verbalization] Depressive or Withdrawn Behaviors Cognitive Behavior Therapy, Cognitive Behavior Therapy and Medication Client Centered Therapy, Cognitive Behavior Therapy with Parents, Interpersonal Therapy, Family Therapy, Relaxation NoneSelf Control Training, Self Modeling Attention, Counselors Care, Counselors Care and Anger Management, Life Skills [poorly tested: Problem Solving, Social Skills] Eating Disorders NoneFamily Therapy (anorexia only)None Substance Use NoneCognitive Behavior Therapy, Contingency Management, Purdue Brief Family Therapy, Family Therapy, Family Systems Therapy None Client Centered Therapy, Education, Group Therapy (!!), Project CARE (!!), Twelve Step Program Traumatic Stress Cognitive Behavior Therapy with ParentsCognitive Behavior TherapyNonePlay Therapy, PsychodramaClient Centered Therapy, Cognitive Behavior Therapy with Parents Only [poorly tested: EMDR] “ Blue Menu ” - Evidence-Based Child and Adolescent Psychosocial Interventions This tool has been developed to guide teams (inclusive of youth, family, educators and mental health practitioners) in developing appropriate plans using psychosocial interventions. Teams should use this information to prioritize promising options. For specific details about these interventions and their applications (e.g., age setting, gender) see the most recent Evidence Based Services Committee Biennial Report ( > Note: Level 5 refers to treatments that were tested and found ineffective. Risk of harm is noted by the symbol (!!), which indicates that at least one study found negative effects on the main outcome measure. The risk of using such treatments should be weighed against potential benefits.

General Services Research: Turning Data into Knowledge II n Common Elements approach –Identified components of evidence based practices –Complements integrated program approach Diffusion strategies: Compatibility, Complexity

Is there a different level of analysis? Protocol Family

Is there a different level of analysis? Incredible Years PCIT Defiant Children Parent Training

Is there a different level of analysis? Protocol Family Practice Element Practice Element Practice Element Practice Element Practice Element Practice Element

Is there a different level of analysis? Incredible Years PCIT Defiant Children Parent Training Commands Attending Time Out Rewards Time Out These are “practice elements.”

Example Attending

Anxiety

ADHD

Autism

Depression

Disruptive Behavior

Traumatic Stress

The Full System Model General Services Research Local Aggregate Evidence Case-Specific Historical Information Causal Mechanism Research Tx Program Selection Therapeutic Practices Service Setting Treatment TeamSupervision Client Progress Treatment Integrity

Case Specific History: Turning Data into Knowledge III n Clinical Dashboard –Progress –Practices Diffusion strategies: Compatibility, Observability

CLINICAL PROCESS MODELING Now how do we put the evidence bases together…?

A Foray into Wisdom Clinical Progress? Clinical Dashboard: Progress Pane Continue plan until goals met Significant concerns? Critical Incidents & Complaints Reports, etc. Consult with specialists as needed yes no yes no yes no yes Clinical Dashboard: Practice Pane Prob. w/ Tx Selection? Treatment Integrity? Therapy Protocols, Dashboards, EBS DB, Consultation Consider adding consultation or training supports Identify barriers and revise plan Options 1. Increase supports 2. Change Intervention 3. Further Consultation 4. Add intervention EBS DB, Local Best Practices, Tx Team New Case? no yes Select Evidence- Based Service (EBS) EBS Database Start

A Foray into Wisdom Clinical Progress? Clinical Dashboard: Progress Pane Continue plan until goals met Significant concerns? Critical Incidents & Complaints Reports, etc. Consult with specialists as needed yes no yes no yes no yes Clinical Dashboard: Practice Pane, EBS DB Prob. w/ Tx Selection? Treatment Integrity? Therapy Protocols, Dashboards, EBS DB, Consultation Consider adding consultation or training supports Identify barriers and revise plan Options 1. Increase supports 2. Change Intervention 3. Further Consultation 4. Add intervention EBS DB, Local Best Practices, Tx Team New Case? no yes Select Evidence- Based Service (EBS) EBS Database Start

A Foray into Wisdom Clinical Progress? Clinical Dashboard: Progress Pane Continue plan until goals met Significant concerns? Critical Incidents & Complaints Reports, etc. Consult with specialists as needed yes no yes no yes no yes Clinical Dashboard: Practice Pane, EBS DB Prob. w/ Tx Selection? Treatment Integrity? Therapy Protocols, Dashboards, EBS DB, Consultation Consider adding consultation or training supports Identify barriers and revise plan Options 1. Increase supports 2. Change Intervention 3. Further Consultation 4. Add intervention EBS DB, Local Best Practices, Tx Team New Case? no yes Select Evidence- Based Service (EBS) EBS Database Start

A Foray into Wisdom Clinical Progress? Clinical Dashboard: Progress Pane Continue plan until goals met Significant concerns? Critical Incidents & Complaints Reports, etc. Consult with specialists as needed yes no yes no yes no yes Clinical Dashboard: Practice Pane, EBS DB Prob. w/ Tx Selection? Treatment Integrity? Therapy Protocols, Dashboards, EBS DB, Consultation Consider adding consultation or training supports Identify barriers and revise plan Options 1. Increase supports 2. Change Intervention 3. Further Consultation 4. Add intervention EBS DB, Local Best Practices, Tx Team New Case? no yes Select Evidence- Based Service (EBS) EBS Database Start

This tells you the treatment types that work for this problem.

This tells you the practice elements associated with those treatment types.

Do the practices fit the problem?

A Bit of Practice Wisdom Clinical Progress? Clinical Dashboard: Progress Pane Continue plan until goals met Significant concerns? Critical Incidents & Complaints Reports, etc. Consult with specialists as needed yes no yes no yes no yes Clinical Dashboard: Practice Pane, PW DB Prob. w/ Tx Selection? Treatment Integrity? Therapy Protocols, Dashboards, PW DB, Consultation Consider adding consultation or training supports Identify barriers and revise plan Options 1. Increase supports 2. Change Intervention 3. Further Consultation 4. Add intervention PW DB, Local Best Practices, Tx Team New Case? no yes Select Evidence- Based Service (EBS) PW Database Start

Select new practices?

Seek Consultation?

MODULARITY IN PRACTICE

MATCH-ADC: Modular Approach to Therapy for Children Diffusion strategies: Compatibility

Three Steps Social Skills Cognitive: BLUE Skill Building Relaxation Activity Selection Problem Solving Psychoed Child Psychoed - Parent Covert Relaxation Wrap up Cognitive: FUN

Three Steps Complete next in sequence Interference Evaluate & Triage Yes No Able to proceed Yes No Social Skills Cognitive: BLUE Skill Building Relaxation Activity Selection Problem Solving Psychoed Child Psychoed - Parent Covert Relaxation Wrap up Cognitive: FUN Gains Complete

Three Steps Complete next in sequence Exposure Fear Related Behavior Related Interference Other Evaluate & Triage Yes No Able to proceed Yes No Commands Active Ignoring Time Out Rewards Anticipating Problems Non compliance Specific Triggers Attention Seeking Low Motivation Serious Behavior Social Skills Cognitive: BLUE Skill Building Relaxation Activity Selection Problem Solving Psychoed Child Psychoed - Parent Covert Relaxation Wrap up Cognitive: FUN Gains Complete

Child STEPs Treatment Project n Youth Network on Child Mental Health n 5-Year, multisite randomized trial –Boston, Honolulu n Anxiety, Depression, Conduct Problems n Community therapists n SMT, MMT, Usual Care n 185 children enrolled so far

Training Results p <.01

Attributes of Innovations 1.Relative Advantage 2.Compatibility 3.Complexity 4.Observability Diffusion of Innovation Communication Channels 1.Mass Media 2.Interpersonal Change Agent Properties 1.Effort 2.Homophily Rate of Adoption Adapted from Rogers (1995)

Attributes of Innovations 1.Relative Advantage 2.Compatibility 3.Complexity 4.Observability Diffusion of Innovation Communication Channels 1.Mass Media 2.Interpersonal Change Agent Properties 1.Effort 2.Homophily Rate of Adoption Adapted from Rogers (1995)

Diffusion Curve Time  From Rogers (1995)

Why have a review committee n Generates “buy-in” (homophily) n Establishes an organizational culture n Promotes relevance n Keeps evidence current n Activates interpersonal channels Diffusion strategies: Interpersonal, Homophily

Invite all stakeholders, (especially your “enemies”) n Maintains multiple perspectives n Coordinates opinion leaders from various social circles n We can always agree on one thing as a starting point… Diffusion strategies: Interpersonal, Effort

Where The Rubber Meets the Road n One-by-one supervision and support n Respect for system members –Pink spreadsheet story n Interpersonal critical mass –Practice guidelines story Diffusion strategies: Effort, Interpersonal, Compatibility

Results in Hawaii From Daleiden, Chorpita, Arensdorf, Donkervoet, & Brogan (2006)

Rate of Improvement? Final Effect Size for Change =.07/mo,.84/yr

Cost per Outcome

Innovation is as much a social process as a technological one. Rogers (1995)

This is John now.

Thank You!

Relevance Mapping n How relevant is the evidence base to my service population? n Under what assumptions? n Where do we focus our efforts?

Procedure n Compare every member of a population to every study in the evidence base n Examine different inclusion assumptions –Problem –Age –Gender –Ethnicity n Aggregate the results

Initial Sample (N = 712) n 34% Disruptive; 22% ADHD; 22% Anxiety; 6% Depression n Age 4 to 18 n 68% boys, 32% girls n 51% multiethnic, 16% white, 14% Asian, 8% Pacific Islander

Who is covered? TreatmentsPPAGPAGE 011.8%18.7%72.1% 1 or more88.2%81.3%27.9%

How are they covered? Treatment Family PPAGPAGE Contingency Management 59.3%18.1%2.4% Parent Management Training 58.4%28.8%2.8% Parent Management Training and Problem Solving 56.7%25.6%2.4% Cognitive Behavior Therapy 51.8%45.5%17.3% Relaxation 50.1%5.8%1.1% Education 48.9%15.6%2.5% Assertiveness Training 46.2%2.0%0.0% Client Centered Therapy 44.9%22.6%3.5% Problem Solving 38.1%35.3%4.9%