for quality and accountability Measuring Outcomes for quality and accountability

The changing landscape in mental health and rehabilitation Increasing demand for accountability i wide range of service settings Funding agencies demand that providers examine service effectiveness and measure consumer outcomes Indications of quality improvement and capacity of management Meet core values and get input from service consumers

Planning a program evaluation Individual accounts and anecdotes Concrete and standardized ways to measure service effectiveness What is “best practice”?

Planning a program evaluation Program administrators are often not well informed about outcome measurement … and how best to link outcome measures to questions that they want to address related to services

The best starting point is to determine: What outcomes need to be addressed? Who is the evaluation information being developed for? Who will receive information about the results? Example: Results can be disseminated to stakeholders such as funding and accreditation bodies, a board of directors, primary consumers, or family members.

A program evaluation effort can be started at any time – when a service starts or added later to an existing service

Clarifying outcomes to be measured Conducting an evaluation, study, or quality assurance effort begins with understanding which outcomes need to be addressed

Steps towards clarification: Get an understanding of your agency’s intended areas of effectiveness Example: The agency’s stated goal is “Increase in employment” Ask whether the services your agency offers are likely to go beyond the intended outcomes to other unintended outcomes Example: A bonus goal is “Increase in self-esteem among service recipients”

Examine your agency’s mission statement, strategic plan, or results of former program evaluation efforts to get closer to your agency’s intended effects and thus the outcomes you should measure

Determine what domains need to be measured from these efforts (housing, employment, social inclusion, satisfaction, etc.) Develop clear definitions or criteria for these domains so that the appropriate measures or instruments can be located

Selecting an outcome measure or instrument Providers must have confidence in the measures Choose high quality and appropriate measures Look for existing measures that match the outcomes needing to be addressed, rather than using home-grown surveys or instruments developed just for the purpose at hand

Ensure that the measures have been tested and are known to be reliable and valid, so … that you can trust what they measure, and … that they measure what they say they should

Two important paradigm shifts The need for reliable and valid measures of the subjective aspects of recovery The need for measures that were developed from a consumer perspective

“The times they are a-changing” Providers, practitioners, managers, administrators, and policy makers can ease pressures they face in the present and in the future by … making use of established, high quality measures and instruments, and clarifying which effects and outcomes ought to be measured

The results from evaluations … may be used not only to satisfy funding bodies and other stakeholders - they will also provide useful information that can guide the direction of changes needed to create more effective ways of helping the people being served

Evaluation is an integrated component in program development Procedures Record keeping Activities Supervision structure Policy statements Job descriptions Mission Evaluation mechanisms