Using Focus Groups & Cognitive Interviews to Re-Evaluate   Tune-up Time Using Focus Groups & Cognitive Interviews to Re-Evaluate an Established Survey Carol Cosenza Center for Survey Research University of Massachusetts Boston

Acknowledgments Thanks to Jack Fowler (CSR) and Paul Cleary (Yale) Work on this project was supported by a cooperative agreement from the Agency for Healthcare Research and Quality (#U18HS016978).

Focus Groups Cognitive Interviews Usually done before writing the survey Learn about ideas/concepts/terminology Formulate hypotheses Cognitive Interviews After survey questions written How do Rs handle cognitive tasks (Comprehension, Retrieval of Information, Judgment, and Formation of Answers)

The Survey E xperience of C are and H ealth O utcomes

Background For those who have received mental health or substance abuse services through a health plan Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Family of surveys (CAHPS Dental, CAHPS Hospice, Clinician & Group) Original goal to develop standardized patient surveys used to compare results across health plans Now focus on providers and used nationally to report on quality of care Consumers/patients report on and evaluate their experiences Access to care Communication with providers

Creating the ECHO® Survey Work started in 2000 and continued for 6+ years Included stakeholder meetings, focus groups, cognitive interviews, and field testing Combination of 2 surveys Mental Health Statistics Program's Consumer Survey (MHSIP) Consumer Assessment of Behavioral Health Services (CABHS) Endorsed by the National Quality Forum July 2007 Survey content Not based on a specific provider encounter Focuses on “counseling and treatment” & overall ratings Includes Qs about administrative services (of health plan)

What you’re thinking now Did she say the survey was written 15 years ago? Why is a mental health survey called “ECHO”? If you were paying VERY close attention: Do respondents know what “counseling and treatment” means?

Things have changed in the last 15 years

Insurance Changes Structure Affordable Care Act Accountable Care Organizations (ACO) Patient Centered Medical Homes (PCMH) Affordable Care Act Not allowed to exclude pre-existing conditions Kids can stay on parent’s account until 26 Overall fewer uninsured Change in how clients interact with insurance companies Methods of communication Choices and availability

Changes in Mental Health Care Mental Health Parity Coverage for mental illness must be comparable to that of physical ailments ACA Mental health and substance abuse treatment as one of the 10 “essential health benefits” Antidepressants and other medications More common More prescribed by non-mental health professionals Advertised direct-to-consumer Societal changes

COGNITIVE INTERVIEWS Our Task Add some questions about mental health care (from ECHO) into a general provider survey (CG-CAHPS) We have questions already written We know CG-CAHPS works We want to know if adding these Qs is OK COGNITIVE INTERVIEWS

Some questions included A LOT of definitions (that were ignored) In the last 6 months (not counting emergency rooms or crisis centers), how many times did you go to an office, clinic, or other treatment program to get counseling, treatment or medicine for yourself? None 1 to 10 11 to 20 21 or more  

Some questions included A LOT of separate phrases 1 2 In the last 6 months (not counting emergency rooms or crisis centers), how many times did you go to an office, clinic, or other treatment program to get counseling, treatment or medicine for yourself? None 1 to 10 11 to 20 21 or more   3 4 5 6

The next questions ask about your counseling or treatment The next questions ask about your counseling or treatment. Do not include counseling or treatment during an overnight hospital stay or from a self-help group. Professional counseling can include care from psychologists, psychiatrists, social workers, nurse practitioners, or other professional counselors. 18. In the last 6 months, did you call someone to get professional counseling on the phone for yourself?

The next questions ask about your counseling or treatment The next questions ask about your counseling or treatment. Do not include counseling or treatment during an overnight hospital stay or from a self-help group. Professional counseling can include care from psychologists, psychiatrists, social workers, nurse practitioners, or other professional counselors. 18. In the last 6 months, did you call someone to get professional counseling on the phone for yourself?

Multiple Providers We found a lot of Rs saw more than one provider for mental health care Talked to a counselor/therapist Got medicine from someone else Involvement with Primary Care doctors Different experiences – hard to summarize Making appointments Wait time Overall rating

Some phrases not understood “treatment” Some questions used “counseling or treatment” – others used “counseling, treatment, or medicine” Rs didn’t know what to include Especially problematic in in context of provider survey “crisis center” “professional counseling on the phone” getting an appointment for counseling or treatment “as soon as you needed” Standing appointments Some thought this was double-barreled

Overall Findings It didn’t work

Time to Pull out the Toolbox – what can we do now? Expert Review Interest in access to care How easy to get services – counseling and medicines ECHO is PLAN-based (not provider-based) Harder for Rs to summarize How mental health services are provided – and how patients use those services – has changed

Cognitive Interviews – Round 2 Split out getting counseling from getting medicines Most Rs thought getting care “as soon as needed” was about emergency care Difference between how hard it is to get something (appointments & medicines) and how often you were able to get it as soon as you needed

Overall Findings – Round 2 It still needs work

Need to look at the entire instrument EvaluationOptions Are the questions (still) consistently understood? Do the Qs accurately describe patients’ experiences? Are concepts asked about still relevant? Any new concepts that should be asked about? FOCUS GROUPS

What we talked about - Access Getting first appointment Getting other appointments Provider continuity How this differs when talking about getting counseling and getting medicines

What we talked about – New ways of interacting Web-based Face-to-Face (Skype-like) Live chat Messaging Texting Phone

What we talked about - Insurance Most had no idea about topics that were asked about Limits on number of visits/ “using up” benefits Referrals Paperwork and calling customer service line Most paperwork seems to be done through providers office When survey was written, health plans were trying to manage/limit mental health services

Other things talked about – What to call it “Mental health” not as stigmatizing as it once was No one calls it (or likes) “behavioral health” “Treatment” needs a modifier – especially if asking about it in context of other health care “Counseling or mental health treatment” might work

Other things talked about Emergency care Getting care right away works in other CAHPS instruments – still not sure why it’s not working here Involvement in care Other treatment options Mandatory treatment This came up when talking about the right to refuse treatment

What did we learn? Cognitive interviews showed we had problems Focus groups gave us a big picture about how mental health care has changed We live in a different world in 2016 than we did in 2000 New technology New ways of interacting with ALL health care

Where are we? Reframe Survey Need more/different Qs about access Move towards a provider-based instrument asking Qs about experience with one provider Need more/different Qs about access Re-evaluate terms that may no longer be relevant Crisis Center Call someone for care on phone Perhaps add Qs about new modes of mental health care

What’s next? Meet with stakeholders Modify survey What we learned from cognitive interviews and focus groups Make more consistent with other CAHPS surveys Cognitive interviews with the revised instrument

Thank you ECHO®