1. Consumer Involvement in Evaluating a Recovery-based Systems Change Initiative Vicki Cousins, Director Office of Consumer Affairs South Carolina Department of Mental Health Katherine M. Roberts, MPH Coordinator, Consumer-to-Consumer Evaluation Team South Carolina Department of Mental Health

2. Words to Live By in South Carolina Nothing About Us Without Us Consumer Involvement in Four Key Areas Planning Planning Policy-making Policy-making Service Provision Service Provision Program Evaluation Program Evaluation

3. Consumer Involvement in Planning and Policy-making Central Administration Offices Central Administration Offices Regional Service Sites Regional Service Sites (Community Mental Health Centers in 17 locations serving 46 counties)

4. Consumer Involvement in Planning and Policy-making New SCDMH consumer-centered policies/directives being developed as a result of our systems emphasis on recovery.

5. Consumer Involvement in Service Provision Following Larry Fricks Georgia model, South Carolina has developed a Medicaid-billable service description for peer support 40-hr. certification training for consumers to become credentialed as CPSS 40-hr. certification training for consumers to become credentialed as CPSS Includes oral and written examination Includes oral and written examination Final approval from DHSS expected July 1, 2003 Final approval from DHSS expected July 1, 2003 10 graduates to date; trainings to be year twice a year 10 graduates to date; trainings to be year twice a year

6. Consumer Involvement in Program Evaluation SCDMH Consumer-to-Consumer Evaluation Team Several Years in the Making 1996 … … 2002 Found our place in the systems … 2002 Found our place in the systems recovery initiative

7. The Purpose of Consumer Involvement Surgeon Generals Report 1999 Consumer staff, clients, and the mental health system all benefit from consumer involvement.

8. The Purpose of Consumer Involvement a Mental Health System Surgeon Generals Report 1999 Consumer Staff meaningful activity or employment meaningful activity or employment role models of recovery role models of recovery enhance sensitivity of the service system enhance sensitivity of the service system

9. The Purpose of Consumer Involvement a Mental Health System Service system benefits: Increases in service satisfaction as related to peoples recovery needs Increases in service satisfaction as related to peoples recovery needs Reduced hospitalizations Reduced hospitalizations Cost-savings in community program operating costs Cost-savings in community program operating costs (Chamberlin & Rogers, 1990; Carpinello & Knight, 1993)

10. SCDMH Consumer-to-Consumer Evaluation Team Project Mission To involve individuals who receive services, family members, independent advocates and citizens-at-large in program evaluation and continuous quality improvement of the South Carolina Department of Mental Health.

11. SCDMH Consumer-to-Consumer Evaluation Team Office of Consumer Affairs, Division of Health Care Reform Office of Consumer Affairs, Division of Health Care Reform Consumer-driven evaluation survey process Consumer-driven evaluation survey process Primary consumer team members Primary consumer team members Paid employees of SCDMH Paid employees of SCDMH Participates in instrument development Participates in instrument development Conducts interviews, compiles data, and produces reports Conducts interviews, compiles data, and produces reports

12. History of Consumer Involvement in Evaluation For several years in the late 1990s, the Team conducted open-ended surveys with over 900 clients, striving to evaluate individual CMHC programs. For several years in the late 1990s, the Team conducted open-ended surveys with over 900 clients, striving to evaluate individual CMHC programs. Starting in 2002, a fast evolution began as the emphasis shifted from program evaluation to system evaluation. Survey questions became closed ended. Starting in 2002, a fast evolution began as the emphasis shifted from program evaluation to system evaluation. Survey questions became closed ended.

13. History of Recovery Measurement in South Carolina March 2002: Formal planning process for recovery system evaluation begins. Expert researchers partner with the CCET. March 2002: Formal planning process for recovery system evaluation begins. Expert researchers partner with the CCET. June 2002: Consumer focus groups, staff surveys June 2002: Consumer focus groups, staff surveys August–October 2002: Surveying to test for validity and reliability August–October 2002: Surveying to test for validity and reliability

14. Instruments MHSIP MHSIP Consumer-to-Consumer Evaluation Scale (CCET) Consumer-to-Consumer Evaluation Scale (CCET) Recovery Questionnaire Recovery Questionnaire

15. South Carolina Consumers Define Recovery Focus Groups What does it mean to be in recovery or recovered from mental illness? What does it mean to be in recovery or recovered from mental illness? What helps a person recover from mental illness? What helps a person recover from mental illness? What prevents a person from recovering from mental illness? What prevents a person from recovering from mental illness?

16. Recovery Questionnaire Domains (10) from the Focus Groups Hope - Self-esteem - Housing Hope - Self-esteem - Housing Employment - Stigma - Self-management Skills Employment - Stigma - Self-management Skills Empowerment/Independence/Self Empowerment/Independence/SelfResponsibility/Control Relationships - Social Relationships - Social Relationships - Family Relationships - Family Spirituality/Higher Power/Religion Spirituality/Higher Power/Religion

17. Sample Size Plan A: Sample consisted of 50 adult clients with open cases in all 17 community mental health centers. Plan A: Sample consisted of 50 adult clients with open cases in all 17 community mental health centers. Expectation: 25 (425) would complete the initial survey and half of that number would agree to be resurveyed 2 to 4 weeks later. Plan A Results: <10% available to participate. Plan B: Asked for available adult clients with open cases to volunteer. Plan B: Asked for available adult clients with open cases to volunteer.

18. Method Informed consent attained. Informed consent attained. Initial face-to-face interviews conducted by CCET team members. Initial face-to-face interviews conducted by CCET team members. Follow-up surveys were conducted either face-to- face or by phone. The process was the choice of the client. Follow-up surveys were conducted either face-to- face or by phone. The process was the choice of the client.

19. Incentives for Client Participation Each client received a button, a $5 McDonalds gift certificate, and a certificate of appreciation signed by the SC State Mental Health Director, George P. Gintoli, for the initial interview. Each client received a button, a $5 McDonalds gift certificate, and a certificate of appreciation signed by the SC State Mental Health Director, George P. Gintoli, for the initial interview. On the second interview, clients received 2 more buttons and an additional gift certificate. On the second interview, clients received 2 more buttons and an additional gift certificate.

20. Process 400+ clients were surveyed/resurveyed from mid-August through mid-October 2002. 400+ clients were surveyed/resurveyed from mid-August through mid-October 2002. Scannable forms were reviewed and prepared for data entry. Scannable forms were reviewed and prepared for data entry. All analysis was conducted in a de-identified manner. All analysis was conducted in a de-identified manner.

21. Analysis The psychometric properties of the MHSIP, the Recovery Questionnaire, and the Consumer-to-Consumer Evaluation Scale included: Item-to-Scale Correlations Item-to-Scale Correlations Test-Retest Reliability Test-Retest Reliability

22. Results Based on the research, changes were made to the Based on the research, changes were made to the Recovery Questionnaire and to the Consumer-to- Consumer Evaluation Scale. The Recovery Questionnaire is presently being used by The Recovery Questionnaire is presently being used by SC Share in South Carolina to evaluate their Recovery for Life program. Plans are underway by South Carolinas Columbia Area Plans are underway by South Carolinas Columbia Area MH Center, the largest community mental health center, to utilize the Recovery Questionnaire in June 2003.

23. Lessons Learned Challenging to go to a full-system survey approach in a limited time period – 2 times in 8 weeks - from individual program site assignments. Challenging to go to a full-system survey approach in a limited time period – 2 times in 8 weeks - from individual program site assignments. Dramatic increase in consumer hours, conflicted with consumers Medicaid benefits, stressed Team Members. Dramatic increase in consumer hours, conflicted with consumers Medicaid benefits, stressed Team Members. But still … state car break-downs, a crash, But still … state car break-downs, a crash, getting lost, getting sick, losing surveyors … It was a terrific learning experience with … It was a terrific learning experience with positive outcomes. We survived it all.

24. Consumer Involvement in Evaluating a Recovery-based Systems Change Initiative Vicki Cousins, Director Office of Consumer Affairs South Carolina Department of Mental Health Katherine M. Roberts, MPH Coordinator, Consumer-to-Consumer Evaluation Team South Carolina Department of Mental Health