1. Using Focus Groups & Cognitive Interviews to Re-Evaluate
  Tune-up Time
Using Focus Groups & Cognitive Interviews to Re-Evaluate
an Established Survey
Carol Cosenza
Center for Survey Research
University of Massachusetts Boston

2. Acknowledgments Thanks to Jack Fowler (CSR) and Paul Cleary (Yale)
Work on this project was supported by a cooperative agreement from the Agency for Healthcare Research and Quality (#U18HS016978).

4. Focus Groups Cognitive Interviews
Usually done before writing the survey
Learn about ideas/concepts/terminology
Formulate hypotheses
Cognitive Interviews
After survey questions written
How do Rs handle cognitive tasks (Comprehension, Retrieval of Information, Judgment, and Formation of Answers)

5. The Survey E
xperience of C
are and H
ealth O
utcomes

6. Background
For those who have received mental health or substance abuse services through a health plan
Consumer Assessment of Healthcare Providers and Systems (CAHPS®)
Family of surveys (CAHPS Dental, CAHPS Hospice, Clinician & Group)
Original goal to develop standardized patient surveys used to compare results across health plans
Now focus on providers and used nationally to report on quality of care
Consumers/patients report on and evaluate their experiences
Access to care
Communication with providers

7. Creating the ECHO® Survey
Work started in 2000 and continued for 6+ years
Included stakeholder meetings, focus groups, cognitive interviews, and field testing
Combination of 2 surveys
Mental Health Statistics Program's Consumer Survey (MHSIP)
Consumer Assessment of Behavioral Health Services (CABHS)
Endorsed by the National Quality Forum July 2007
Survey content
Not based on a specific provider encounter
Focuses on “counseling and treatment” & overall ratings
Includes Qs about administrative services (of health plan)

8. What you’re thinking now
Did she say the survey was written
15 years ago?
Why is a mental health survey called “ECHO”?
If you were paying VERY close attention:
Do respondents know what “counseling
and treatment” means?

9. Things have changed in the last 15 years

10. Insurance Changes Structure Affordable Care Act
Accountable Care Organizations (ACO)
Patient Centered Medical Homes (PCMH)
Affordable Care Act
Not allowed to exclude pre-existing conditions
Kids can stay on parent’s account until 26
Overall fewer uninsured
Change in how clients interact with insurance companies
Methods of communication
Choices and availability

11. Changes in Mental Health Care
Mental Health Parity
Coverage for mental illness must be comparable to that of physical ailments
ACA
Mental health and substance abuse treatment as one of the 10 “essential health benefits”
Antidepressants and other medications
More common
More prescribed by non-mental health professionals
Advertised direct-to-consumer
Societal changes

12. COGNITIVE INTERVIEWS Our Task
Add some questions about mental health care (from ECHO) into a general provider survey (CG-CAHPS)
We have questions already written
We know CG-CAHPS works
We want to know if adding these Qs is OK
COGNITIVE INTERVIEWS

13. Some questions included A LOT of definitions (that were ignored)
In the last 6 months (not counting emergency rooms or crisis centers), how many times did you go to an office, clinic, or other treatment program to get counseling, treatment or medicine for yourself?
None
1 to 10
11 to 20
21 or more

14. Some questions included A LOT of separate phrases1 2
In the last 6 months (not counting emergency rooms or crisis centers), how many times did you go to an office, clinic, or other treatment program to get counseling, treatment or medicine for yourself?
None
1 to 10
11 to 20
21 or more 3 4 5 6

15. The next questions ask about your counseling or treatment
The next questions ask about your counseling or treatment. Do not include counseling or treatment during an overnight hospital stay or from a self-help group. Professional counseling can include care from psychologists, psychiatrists, social workers, nurse practitioners, or other professional counselors. 18. In the last 6 months, did you call someone to get professional counseling on the phone for yourself?

16. The next questions ask about your counseling or treatment
The next questions ask about your counseling or treatment. Do not include counseling or treatment during an overnight hospital stay or from a self-help group. Professional counseling can include care from psychologists, psychiatrists, social workers, nurse practitioners, or other professional counselors. 18. In the last 6 months, did you call someone to get professional counseling on the phone for yourself?

17. Multiple Providers
We found a lot of Rs saw more than one provider for mental health care
Talked to a counselor/therapist
Got medicine from someone else
Involvement with Primary Care doctors
Different experiences – hard to summarize
Making appointments
Wait time
Overall rating

18. Some phrases not understood
“treatment”
Some questions used “counseling or treatment” – others used “counseling, treatment, or medicine”
Rs didn’t know what to include
Especially problematic in in context of provider survey
“crisis center”
“professional counseling on the phone”
getting an appointment for counseling or treatment “as soon as you needed”
Standing appointments
Some thought this was double-barreled

19. Overall Findings
It didn’t work

20. Time to Pull out the Toolbox – what can we do now?
Expert Review
Interest in access to care
How easy to get services – counseling and medicines
ECHO is PLAN-based (not provider-based)
Harder for Rs to summarize
How mental health services are provided – and how patients use those services – has changed

21. Cognitive Interviews – Round 2
Split out getting counseling from getting medicines
Most Rs thought getting care “as soon as needed” was about emergency care
Difference between how hard it is to get something (appointments & medicines) and how often you were able to get it as soon as you needed

22. Overall Findings – Round 2
It still needs work

23. Need to look at the entire instrument
EvaluationOptions
Are the questions (still) consistently understood?
Do the Qs accurately describe patients’ experiences?
Are concepts asked about still relevant?
Any new concepts that should be asked about?
FOCUS GROUPS

24. What we talked about - Access
Getting first appointment
Getting other appointments
Provider continuity
How this differs when talking about getting counseling and getting medicines

25. What we talked about – New ways of interacting
Web-based
Face-to-Face (Skype-like)
Live chat
Messaging
Texting
Phone

26. What we talked about - Insurance
Most had no idea about topics that were asked about
Limits on number of visits/ “using up” benefits
Referrals
Paperwork and calling customer service line
Most paperwork seems to be done through providers office
When survey was written, health plans were trying to manage/limit mental health services

27. Other things talked about – What to call it
“Mental health” not as stigmatizing as it once was
No one calls it (or likes) “behavioral health”
“Treatment” needs a modifier – especially if asking about it in context of other health care
“Counseling or mental health treatment” might work

28. Other things talked about
Emergency care
Getting care right away works in other CAHPS instruments – still not sure why it’s not working here
Involvement in care
Other treatment options
Mandatory treatment
This came up when talking about the right to refuse treatment

29. What did we learn? Cognitive interviews showed we had problems
Focus groups gave us a big picture about how mental health care has changed
We live in a different world in 2016 than we did in 2000
New technology
New ways of interacting with ALL health care

30. Where are we? Reframe Survey Need more/different Qs about access
Move towards a provider-based instrument asking Qs about experience with one provider
Need more/different Qs about access
Re-evaluate terms that may no longer be relevant
Crisis Center
Call someone for care on phone
Perhaps add Qs about new modes of mental health care

31. What’s next? Meet with stakeholders Modify survey
What we learned from cognitive interviews and focus groups
Make more consistent with other CAHPS surveys
Cognitive interviews with the revised instrument

32. Thank you
ECHO®