1. Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and Drug Programs

2. Goals for Presentation  Review consumer perception of care as a concept  Review success criteria for its measurement  Review the development & testing of CSAT’s Modular Survey

3. The Concept  Measuring consumer response = core business function  In healthcare tied to growth of consumerism & CQI initiatives  A NOMS domain  Consumer perception of care ≠ satisfaction

4. Measuring Consumer Perception of Care  Approaches to measurement differ depending on scope & purpose  For purposes of comparability, improvement over time, & benchmarking measures must be :  Meaningful  Scientifically-sound  Actionable

5. The Problems with Satisfaction  No evidence linking the measurement of satisfaction to client outcomes  Few satisfaction surveys scientifically validated  Data is not actionable (ceiling effect)

6. The Modular Survey  SAMHSA –supported initiative  Conducted under the auspices of the Forum on Performance Measurement & the Washington Circle  Conducted in 2 phases  Phase 1 in conjunction with mental health Rx  Phase 2 substance abuse – specific

7. Design Requirements  Phase 1 focus on commonality, not comprehensiveness  Short  Scientifically sound  Actionable  Use of existing, widely-used, non-proprietary surveys  Consensus-driven

8. “Field-level” Common Measures Adult Common Measures Child/ Adolescent Common Measures Adult Mental Health Core Measures Adult Substance Abuse Core Measures Adol Substance Abuse Core Measures C/Adol Mental Health Core Measures Common Design Template Modular Survey Flow of Common Questions for Individual Respondent

9. Approach to Phase 1  4 workgroups to develop consensus  Selection of instruments  Identification of concerns  Identification of potential items  Ranking of items  Final item selection (modified Delphi)  Pilot testing

10. Phase 1 Pilot Testing  Conducted during summer/fall 2004  Primary data collection in Cincinnati United Way agencies (N = 1157)  Secondary analysis using MHSIP data (16 state & LA County data sets)  Final N > 22,000 respondents  Pool of items reduced from 28 to 11  All items common to both fields, both populations

11. Approach to Phase 2  Stand-alone SUD Rx initiative  Under Washington Circle with Forum as “subcontractor”  New item development (no existing SUD survey)  Content work group co-chaired by Tom McLellan (TRI) & Doreen Cavanaugh (Georgetown)  Support from Forum Methods Work Group & Ann Doucette (George Washington  Public Provider & Consumer Advisory Groups

12. Phase 2 SUD Initiative  Closely coordinated with NOMS  Identification of concerns  Relationship to treatment program  Self awareness of problem/commitment to change  Perceived outcomes  Social connectedness  Generation of items (35 in testing pool)

13. Phase 2 Pilot Testing  OMB & IRB approval spring 2006  Conducted in 3 rounds  Round 1 – Adult & Adolescent (summer 2006)  14 programs, N = 1207  Round 2 – Adult & Adolescent (winter 2006 – 07)  6 programs, N = 585  Round 3 – Adolescent (spring-summer 2007)  8 programs, N = 268  Final adult N = 1549 (2 samples)  Final adolescent N = 492 (1 sample)  All demographic groups covered except Native American

14. Phase 2 Completion  Analysis & recommendations by Ann Doucette PH.D.  Use of IRT  For a copy of the technical report, e-mail jbartlett@avisagroup.com  Review by Forum Methods Work Group (November 2007)  Review & Approval by SUD Content Committee (November 2007)

15. Final SUD Modular Survey  21 items (11 from Phase 1, 10 from Phase 2)  Quality – 6 items  Perceived Outcomes – 6 items  Social Connectedness – 7 items  Commitment to Change – 2 items  10 demographic & background items  Spanish translation available

16. Modular Survey Flow of Common Questions for Individual Respondent Final Version Phase 1 Common Items * Phase 2 Mental Health Items Phase 2 SUD Items Common Design Template * All populations, all fields

17. Convergence with NOMS  In Spring 2006 NOMS Technical Consulting Group convened  Recommended 17 items from 8 different instruments  9 of the 17 from the Modular Survey  5 are in the Final Modular Survey

18. In summary……  Consumer perception of care key measurement domain  Its measurement must meet certain criteria in order to be worth the effort  The Modular Survey is the only current instrument measuring consumer perception of care that is:  SUD Rx – specific  Product of both consensus and empirical analysis  Short and actionable