+ Using Client-Focused Research Methods to Improve Outcomes Saara T. Grizzell, Ph.D., CRC, LVRC & Julie F. Smart, Ph.D., CRC (ret), LPC, LVRC, ABDA, AAPC, CCFC, NCC Utah State University

+ Objectives Specify research methods/procedures List possible uses of client-focused research ID three considerations Conceptualize approaches

+ Overview Introduce client-focused research approach and how it can be used in program evaluation Research methods/procedures for tracking outcomes with the use of feedback (pilot study example) approaches for analyzing the data of client- focused interventions (pilot study example) what to consider when implementing successful client-focused interventions 3

+ Introduction

+ Background First introduced in 1996 by Howard, Moras, Brill, Martinovich, & Lutz Howard et al, 1996, argued that three questions were at the root of all intervention assessments. 5

+ Assessing an Intervention: Three Questions Does the intervention work under special, experimental conditions? Does the intervention work in practice? Does the intervention work for this particular consumer? (Howard, Moras, Brill, Martinovich, & Lutz, 1996) 6

+ Setting the Context: Program Evaluation Answers specific questions using a systematic method Types of Questions Formative and Summative May be conducted at several stages during a program’s lifecycle 7

+ Applying the Three Questions to Program Evaluation Does this program work in a special, experimental context (e.g. to meet the needs of individuals with certain types of disabilities, levels of income, certain ethnic groups, etc.)? Does this program work in general practice? Does the program work for this particular consumer? 8

+ Client-Focused Approach

+ Involves assessing the client’s progress on a weekly basis during the treatment course and providing weekly feedback to the counselor and the client about the client’s progress 10

+ Treatment-focused vs. Client- focused Research Concerned with establishing comparative and intervention efficacy Provides aggregated results over groups of participants Focuses on the group’s response to treatment Monitors individual progress over the course of treatment Provides feedback to the client and the clinician during the course of treatment Focuses on client- specific response to treatment Treatment-FocusedPatient or Client-Focused 11

+ Client-Focused Assessment Systems Although several systems have been developed, only two systems have demonstrated gains in randomized controlled trials (Duncan, 2013): The Outcome Questionnaire-45 (OQ-45; Lambert, Kahler, Harmon, Burlingame, & Shimokawa, 2013) Partners for Change Outcome Management System (PCOMS; Miller, Duncan, Sorrell, & Brown, 2005) 12

+ Client-Focused Research: Enhancing Outcomes Particularly helpful in decreasing deterioration rates Used in inpatient and outpatient settings Positively related to client outcomes Used as a quality assurance system in agency settings More recently used in a state VR setting (Hawkins et al, 2004; Lambert, Hansen, & Finch, 2001a; Lambert et al, 2002a; Shimokawa, Lambert, & Smart, 2010; Whipple, Lambert, Vermeersch, Smart, Nielsen, & Hawkins, 2003; ; Hawkins, Lambert, Vermeersch, Slade, & Tuttle, 2004) 13

+ Using Client-Focused Approach in a State VR Setting: Pilot Study Example

+ Study Purpose Examine the impact of providing treatment progress feedback to 30 individuals receiving services at a vocational rehabilitation agency 15

+ Study Design and Conditions Study Design: Repeated measures randomized wait- list control design with matching prior to randomization Feedback Condition (Fb): Group counseling with feedback Treatment as Usual (TAU): Group counseling but no feedback 16

+ OQ-45: Progress and Outcome Outcome Questionnaire-45 (OQ-45) designed to track progress during treatment and outcomes at termination Responses used to generate progress report Progress report acts as the feedback (Lambert, Kahler, Harmon, Burlingame, & Shimokawa, 2013) 17

+ Recruitment/Enrollment Identified clients from own client base who met criteria Informed potential participants of study Administered the GRQ, a screening tool that assesses readiness for group Contacted interested participants Enrolled participants Conducted initial interview Administered baseline QO CounselorsResearcher

+ Counseling Groups 5 counseling groups, 4 to 6 members each Conducted by a facilitator and co-facilitator Group facilitator or co-facilitator served as participant’s individual VR counselor Duration and frequency: 1.5 hours per session, once weekly for 10 weeks 19

+ Feedback Provision Feedback provided to counselors regarding progress of clients in Fb condition One page report about client treatment response One page report provided to participants in the Fb condition A graph charting progress and a narrative about the rate of progress Feedback to CounselorsFeedback to Participants 20

+ End of Study Researcher asked all participants to rate progress toward employment and group counseling goals 21

+ Data Analyses The independent variables of time, condition, and office were included in a Linear Mixed Effects Model analysis for each of the dependent variables (symptom distress, interpersonal relationships, and social role performance) 22

+ Data Analyses Chi square analysis was used to analyze potential differences in categorical demographic variables between conditions McNemar test for correlated proportions was used to determine employment outcomes Independent samples t-tests were used to analyze the employment progress ratings 23

+ Results The social role performance and mental health functioning scores in both conditions showed significant improvement. Ratings for employment progress were statistically significant, f(251)=2.77, p=.006, two tailed Employment outcomes were significant for both conditions (p=0.012), and close to significance for the treatment condition (p=0.063). 24

+ Results Interestingly, participants in the feedback condition who received social security or subsistence benefits made the most steady and consistent progress with interpersonal relationships (p=.025), social role performance (p=.021), and mental health functioning (p=.028). 25

+ Using Client-Focused Methods in Program Evaluation

+ Tasks of the Evaluator Verify that implemented programs provide needed services Determine which programs produce the most favorable outcomes Select the programs that offer the most needed types of services Determine if program interventions are the cause of the desired changes Provide information to maintain and improve quality (Posavac & Carey, 2007)

+ Verify that Implemented Programs Provide Needed Services “The most fundamental problem with programs is that some are either never implemented as planned or are implemented in such a diluted fashion that people in need receive no or minimal benefit.” (p. 4, Posavac & Carey, 2007) 28

+ Using Client-Focused Methods to Conduct Program Evaluations Assessing the need for the program Examining the process of meeting consumer needs Assessing what the program has actually achieved Evaluating program outcomes 29

+ Considerations for Implementing a Client-Focused Approach On-going Monitoring: Attrition is a big concern o Start with a bigger N o Consider short durations of monitoring o Make agreements about the logistics of routine monitoring (e.g. at office, on-line, weekly, every other week, etc.) Delivering Feedback: Consider agency and consumer resources (on-line, USPS, Log in) Implementing Feedback: Suggest a routine time for consumer and provider to meet to discuss the feedback 30

+ Contact Information Saara Grizzell l.usu.edu