The Role of Client Outcomes in Evaluating a System of Care Model of Mental Health Delivery Richard Phillips, Ph.D. Director of Research and Evaluation, Idaho Child Welfare Research & Training Center Eastern Washington University Patty Gregory, LCSW Director Outline for presentation Introduction to outcomes: different types – Rick Introduction to system of care for SED children: the child/family centered model (slide #4) – Patty Type of data required by federal/state guidelines (slide #5) – Rick Type of data useful to partners to make decisions about a case (slide #6) – Patty Data/themes collected from the sites: (slide #7 – notes pages Family centered practice and conflict with standardized data: Rick The reality of collaboration: Patty The role of advocacy: Rick Recommendations: (slide #11 – notes pages) Ensure that system of care values and principles extends into the bureaucracy – Patty Work with local sites to establish data collection routines that celebrate their actions, rather than their reporting functions: Rick Provide evaluation support for local collaboratives so that volunteer hours are spent where the members are most effective – in collaborative functions, not recording or documenting functions: Patty Provide support for moving to a participatory evaluation model that recognizes that client ownership of outcomes is the single most important function of mental health intervention, and that acknowledges that validity and reliability may not be as important as changes in client behavior - Rick

Design On site meeting at each demonstration site for: information gathering case staffing observation training on evaluation issues conducting focus groups for process data collection Goal setting process Council member identification of barriers encountered in reaching the goals. Critical case review Goal setting process with the three demonstration sites to identify goals they were trying to accomplish as a community council Council members were surveyed for an initial goal list, goals were ranked and goals that were most frequently rated were top priorities. As a parallel process members were asked to identify barriers they were encountering in reaching these goals. The critical case review methodology was adopted to collect information around the issues faced by community councils. The critical case review process analyzes how individual councils function around one case they identify as having a successful outcome and one case in which they could not identify a successful outcome. A group interview process captured personal memory and case files to describe the sequence of one case in each category and used the differences to understand how various issues influenced practice.

Role of Outcomes in a System of Care What are outcomes? Changes in adaptive behaviors Basic assumption: what people do as a result of professional interaction is the best measure of professional effectiveness What’s the difference between outcomes and something else? Outputs: services/activities provided to clients to help (them) improve their condition Basic assumption: the amount of services is the best measure of professional effectiveness.

Kinds of Outcomes Representative Outcomes Refers to the health or capacity of a system around specific issues or mandates Represents how an entity or person is doing on a global scale. Refers to how individual outcomes relate to a larger system. Serves to help an entity or person assess how they are doing on a global scale. Examples include: State average scores on standardized academic achievement tests Decreases in CPS referrals or re-referrals Decreases in CAFAS scores

Kinds of Outcomes Functional Outcomes Refers to adaptive behaviors of a person or a program or system as they strive to reach their goals; what people actually do to reach their goals. Serves to enhance the self-reliance of persons in relation to the goals in question Example: Client C visited 3 agencies for the first time and received food and clothing for a week Social Worker B collaborated with the school counselor to create an early referral form Student K stayed in school and completed his past assignments

Participatory Evaluation: Using Client Progress to Help Clients Progress Functional Outcomes What people do as a result of participating in a relationship Take actions Change attitudes Increase performance Become more able Representative Outcomes What happens as a result of aggregated functional outcomes Increased classroom performance Increased grades or attendance Positive changes in CAFAS scores

System of Care for Children Set of shared values and principles Focus on strengths of family; not on diagnosis of the child Maintain child in their own home, in their own school, and in their own community Service array planned with family to meet family identified needs Non-traditional Traditional

Post Treatment Data School attendance/conduct for first full quarter after completion of service # of failing school academic core subject grades out of total possible CAFAS score Post Juvenile Probation Assessment score Disposition of case Approximate # of hours spent by council in chambers Approximate # of hours spent by council members in case management Number of council members involved in direct service on case Family satisfaction survey results School attendance/conduct office referrals full days of in-school suspension full days of out-of-school suspension Disposition of case out of community placement out of state placement in-patient placement

Nothing About Us Without Us Parent perspective on functional outcomes First and foremost, we seek safety That our children are insured a quality of life as happy productive members of community as measured by: sharing a ceremony or celebration, being “in this together” making it through a school suspension while adjusting to a new medication We need strengths acknowledged, resources valued and our children treated with love and respect CrossBear, S. (Spring/Summer 2002) Nothing about us without us, Data Matters, National Technical Assistance Center for Children’s Mental Health, Georgetown University Child Development Center Parents need to know that those who provide care for their children, first and foremost do no harm. The first outcome parents seek is that their children live. Quality of life insures that children have the ability to be happy and productive members of the community What is the measure of that? Can be measured by success in school and progress in educational plans. Measured by children’s presence in the juvenile justice system, out of home placements and community-based services. But it can also be measured by nights at the supper table without conflict. Measured by family indicators: Community councils, in order to assist family to make decisions about a case, first need: Parent participation Identified strengths of the family and child from multiple arenas in multiple situations Clearly understand the problem situation from the family’s situation Support from the agency bureaucracy to support “whatever it takes” in assisting families Understanding of the limits/constraints and possibilities of community partners Respect for families and each other in the decision-making process Full partnership participation

Generalized Service Model Needs Assessment & Diagnosis Client Presenting Issues Tx plan and Clinical goals Services Provided Periodic Assessment & Service Plan Review Demonstrates reliance on representational outcomes and on CAFAS and CBCL Case Closure when Clinical goals met

Developing Community Council Model Council Goals Family Centered Problem Solving Client Referral Tx Goals & Service Plan Interventions & Assessments Collaborative Reviews Case Closure when Council and Family Goals met

Conclusion When the probable results of inaction, and the lack of results of current actions, form the most important reasons for referral to a program, then a conclusion about standardized data can be made. Any data that are used in predictive sense to direct what the parent should do are data that are seen as harmful to the process of getting the family to engage as partners with the council in setting out a treatment plan for their child.

The Reality of Collaboration Council members defensive because of agency history Lack of experience with engaging families in a process where the family leads interventions Lack of understanding of the participation agencies regulations and guidelines Lack of trust – in the family and in the professional process Inability to get key partners to the table – not full participation Lack of available concrete services/programs Funding constraints

The Role of Advocacy A key measure of council effectiveness is the willingness and ability of council members to integrate council client concerns into their daily agency routines Advocacy looks like: making connections with other professionals as council members go about their daily routines Intervening in one’s own agency to help clients gain access to services Being supportive by other council members in challenging status quo treatment options Advocacy does not look like being held accountable for collecting data to show client progress

Greatest Challenge: Supporting care providers as they move from an expert, service provider orientation to a facilitating, partnering orientation

Recommendations Ensure that system of care extends into the bureaucracy Work with local sites to establish data collection routines that celebrate client actions Provide evaluation support for local councils so volunteer hours are spent in collaborative functions, not recording or documentation functions Train to a participatory evaluation model that recognizes that client ownership of outcomes may be the single most important function of a mental health intervention Learn to acknowledge that validity and reliability may be obstructionist constructs for support changes in client behavior Ensure that system of care extends into the bureaucracy – Patty Work with local sites to establish data collection routines that celebrate their actions, rather than their reporting functions – Rick Provide evaluation support for local collaboratives so that volunteer hours are spent where members are most effective – in collaborative functions, not recording or documenting functions (use of data collection and management services in the form of a full-time person) Patty Provide support for moving to a participatory evaluation model that recognizes that client ownership of outcomes is the single most important function of mental health intervention, and that acknowledges validity and reliability may not be as important as changes in client behavior - Rick

Functional Outcome Evaluation Client Progress Professional Client set Goals Goals

Bibliography Burns, B., & Goldman, S., (1998) Promising practices in wraparound for children with serious emotional disturbance and their families. Washington DC: Georgetown University Child Development Center, CASSP technical Assistance Center. Substance Abuse and Mental Health Services Administration. U.S. Department of Health and Human Services CrossBear, S. (Spring/Summer 2002) Nothing about us without us, Data Matters, National Technical Assistance Center for Children’s Mental Health, Georgetown University Child Development Center Friesen, B., & Huff, B., (1996) Family perspectives on systems of care. In B. Stroul (Ed.) Children’s Mental Health. Creating Systems of Care in A Changing Society. Baltimore, MD: Brookes Publishing. Lourie, I.S., & Davis, C. (1999) A Needs Assessment of Idaho’s Children With Serious Emotional Disturbances and Their Families. Pines, S. (2002) Building Systems of Care: A Primer. National Technology Assistance Center for Children’s Mental Health State of Idaho (2002). Building on Each Other’s Strengths. Child Mental Health Initiative Grant, GFA: SM-02-002 CFDA 93.104.