1. Use of the Outcome Questionnaire-45 in a Psychology Training Clinic
John Fitzgerald, PhD.
The Psychology Centre, Hamilton
Acknowledging the conscientious support of Philippa Thomas and Jessica Taylor, and all the interns and students who have been on placement with us at the Centre.

2. The Psychology Centre, Hamilton
The Psychology Centre (TPC) is the operational arm of the Waikato Clinical Psychology Educational Trust. The Centre opened for business late in 2000.
All 2nd year (intermediate) students on the University of Waikato Clinical Programme complete a placement (2 days per week, 13 weeks). Four students each semester.
All interns complete a placement (4 days per week, 23 weeks). Four each semester.
Clients from two main sources,
Community/self referral (seen by 2nd Yrs and two interns)
Health Waikato’s Adult Mental Health Service (seen by two interns)
We see clients of all ages and with a wide range of concerns. The only clients we decline are,
Those who are assessed as being too complex to be seen by an intern even under close supervision – no more than 10 per year.
Clients who refuse to allow direct supervisor involvement, either by videotaping or sessions or ‘live supervision’ – no more than 5 per year.
The Psychology Centre, Hamilton

3. Measuring Outcomes and Monitoring Change
Trauer (2010), in Outcome Measurement in Mental Health, suggests we measure outcomes to,
Provide information to managers
Assist in decision making
Incorporate the consumer’s perspective
These are often limited in their client focus, actuarial, and macro-oriented.
In training, as in routine clinical practice, we strive to focus more on,
Providing data to therapists and their clients
Supporting collaborative therapy decisions
Facilitating client engagement
This is more about recognising and monitoring change.
Lambert (2010), in Prevention of Treatment Failure, focuses primarily on the collection and use of clinical data aimed at improving the effects of psychotherapy, and the prevention of poor/negative outcomes. This is what we are training students to understand and apply.

4. How Good are we at Measuring Therapeutic Outcomes?
Hatfield & Ogles (2004)
Surveyed 2,000 clinical practitioners in APA
37% used some form of outcome assessment
Most common were BDI, GAS, CBCL – these are NOT outcome measures
OQ-45 (Lambert, et al., 1994) only used by 6% of respondents
Why? Too much paper work, time consuming, extra burden on the client
Bickman, Rosof-Williams, Salzer, et al., (2000)
Surveyed 539 C&A mental health workers in USA
Few rated standardised outcome data as desirable or valued
Psychologists no different from other groups
Outcome assessment is an ongoing process, not one that should be relegated to the final session.
Two US studies
Additional paper work
But it is the client’s time
?? Extra burden ???
I would have thought that finding out how the client is doing IS the work of therapy .. Not an adjunct or add-on!
So what are they doing to identify change? To understand what specific elements of what they are doing are the most useful?

5. More Reasons to be Concerned About Measuring Change.
Code of Ethics for Psychologists Working in Aotearoa/New Zealand (2002)
Principle 2: Responsible Caring
Practice Implication 2.1.8: When it is clear that the client is not benefitting, a psychologist initiates a termination of the professional relationship.
How do we judge that “the client is not benefitting”? Measuring outcomes does nothing to address this imperative.
Core Competencies for the Practice of Psychology in New Zealand (2009)
Clinical Scope: Intervention
Knowledge – “Knowledge of how to critically evaluate interventions and modify them when change is required”
Skills – “Critical evaluation of strengths, weaknesses and limitations during interventions” (emphasis added)

6. Do No Harm!
There is an “assumption … that as psychotherapy is only talking … no possible harm can ensue.” (Nutt & Sharpe, 2008)
An increasing body of research shows us that about 10% of clients get worse after starting therapy (Jarrett, 2008)
When is a procedure harmful? (Francis, 2009)
when it is significantly detrimental
when it does not do what it purports to do
inaction in the face of imminent harm
Do we have adequate monitoring strategies for recognising each of these eventualities?

7. Outline of the project Purposes Standard Measures
Model good practice processes for students/interns (Scientist-Practitioner)
Integrate standardised monitoring procedures into individualised case management
Establish a practice base for future consultation (DHB, PHO, etc.)
Standard Measures
Outcome Questionnaire-45 introduced when managing a brief therapy evaluation for Waikato PHO, January 2008
Helping Alliance Questionnaire-II and Patient Health Questionnaire-4 since January 2011.
Took several months for processes to ‘bed-in’
Measures not routinely collected at commencement
Difficulty integrating with case-specific measures
Limited data management systems and data integration

8. Measures and protocol
Outcome Questionnaire - 45 (OQ45.2; Lambert, Hansen, Umphress et al., 1996) – 45 item self-report questionnaire exploring Symptom Distress (anxiety, affective disorders, adjustment disorders, and stress), Interpersonal Relations (e.g. loneliness, conflict, and family problems), and Social Role (difficulties in role as worker, homemaker, student, etc).
Helping Alliance Questionnaire - II (HAq-II; Luborsky, Barber, Siqueland, et al.,1996) – the standard client/therapist completed measure against which newer measures are evaluated. The HAq-II has 19-item client and therapist rated forms.
Patient Health Questionnaire for Depression and Anxiety - 4 (PHQ-4; Kroenke, Spitzer, Williams, & Löwe, 2009). 4-item criterion-referenced screening tool used to identify elevated levels of either anxiety (2 questions, GAD-2) or depression (2 questions, PHQ-2). The PHQ-4 is part of a series of abbreviated measures developed by Spitzer and colleagues, and is a public domain instrument.
Session
Measure 1
OQ45, PHQ-4 2
HAq-II 3
OQ-45 4 - 5
OQ45, HAq-II
Every 2nd session
OQ45
End of therapy
OQ45, HAq-II, PHQ-4
Measures are administered at the start of the session, except for the first session.

9. Descriptive Results
N=241, clients in our OQ45 database. This is about 80% of clients seen - excludes age<18yrs, clients seen for ‘assessment only’, ACC referrals.
Age: mean=36.4y, SD=12.9y, range 18-77y.
Gender: Male=72(30%), Female=169(70%)
198 (82%), clients completed service. 43 clients still ‘open’.
Overall DNA/Canc rate=20.6%

10. Community clients (n=134, 68%)
All clients
AMH clients (n=64, 32%)
Community clients (n=134, 68%)
‘Closed’ Cases
N=198
Total attended
1,852
703
1,149
Mean (SD)
9.4 (6.7)
10.1 (5.6)
8.6 (3.0)
Range
0-38
1-26
Total DNA
483
170
313
2.4 (3.0)
2.7 (3.0)
2.3 (3.0)
0-17
0-9
Male
n=56 (28%)
494
236
258
8.8 (6.6)
10.3 (5.1)
7.8 (7.4)
0-29
2-21
118 50 68
2.1 (2.8)
2.2 (2.3)
2.1 (3.1)
0-12
Female
n=142 (72%)
1,358
467
891
9.6 (6.8)
11.4 (6.0)
8.8 (7.0)
365
120
245
2.6 (3.1)
2.9 (3.4)
2.4 (2.9)

11. American samples (Lambert et al., 2003)
Cross-Cultural norms
de Jong, Nugter, Polak, et al. (2007) have explored the validity of the OQ-45 in Holland.
American samples (Lambert et al., 2003)
Dutch samples
(de Jong et al., 2007)
The Psychology Centre
Community
Clinical
Community referred
AMH referred
All clients SD
25.4 (12)
49.4 (15)
22.2 (10)
48.9 (16)
46.6 (12.9)
48.6 (13.2)
47.3 (13.0) IR
10.2 (6)
19.7 (6)
8.4 (5)
16.8 (7)
17.3 (6.4)
18.7 (5.6)
17.8 (6.2) SR
9.6 (4)
14.1 (5)
8.1 (3)
13.6 (6)
13.2 (5.0)
13.7 (5.2)
13.3 (5.1)
Total
45.2 (19)
83.1 (22)
38.7 (16)
79.5 (25)
77.0 (20.3)
81.0 (20.5)
78.4 (20.4)

12. OQ-45 scale means and SDs All clients AMH clients Community clients In
Out
Symptom Distress
47.3 (13.0)
32.5 (13.9)
48.6 (13.2)
35.3 (14.2)
46.6 (12.9)
30.8 (13.4)
Interpersonal Relations
17.8 (6.2)
13.6 (6.4)
18.7 (5.6)
15.5 (6.3)
17.3 (6.4)
12.4 (6.2)
Social Role
13.3 (5.1)
9.8
(4.8)
13.7 (5.2)
10.1 (4.9)
13.2 (5.0)
9.5
(4.7)
Total Score
78.4 (20.4)
55.8 (23.1)
81.0 (20.5)
60.9 (23.2)
77.0 (20.3)
52.7 (22.5)
Clinical threshold scores: SD=36, IR=15, SR=12, Total=63
Reliable Change Index (RCI) for OQ-45 Total Score is 14 points
All scale In-Out comparisons within each client grouping were statistically significant
Intake SR was the only significant difference between genders (t=-2.17, p=.03)
Mean change in the Total Score for the three groups were 21.9, 18.7, 23.9 respectively. Each of these is greater than the RCI threshold.

13. Progress Monitoring
Available on-line OQ-Analyst assists in the monitoring of client progress, and helps identify when therapy is not progressing … (we do not use this … yet).
However, it has been useful in tracking progress (RH), relapse (GD), and the impact of specific events in treatment (KS and RK).
OQ45 scores are discussed in supervision, and with the client.
OQ45 scores are mapped as one way of monitoring and highlighting change.

14. Helping Alliance Questionnaire-II
Commenced use in 2011 because of clear reported association between therapy outcomes and the quality of the therapeutic relationship (see Crits-Christoph & Gallop, 2006; Lambert, 2010), which we had no formal way of monitoring.
Proving difficult to collect. Wording of scale is ‘difficult’ in places (e.g., “From time to time we both talk about the same important events in my past”)
Intake: n=40,
Client mean=5.25(0.4), range
Therapist mean=4.82(0.4), range
Outcome: n=24,
Client mean=5.43(0.4), range
Therapist mean=5.17(0.4), range
Suggested interpretation –
Client means increase during course of therapy
Therapist means increase
We have used a threshold of 1SD either side of the mean as a ‘flag’
Intake – client rating (quality of therapeutic alliance) – 2 low, 5 high
Intake – therapist – 9 low, 0 high
Outcome – client – 1 low, 7 high
Outcome – therapist – 2 low, 1 high

15. OQ45 x HAQ-II
We need to be cautious not to assume causality, and be aware that our sample is still small.
At the point of intake there was a negative association between the OQ45 Symptom Distress subscale (and Total Score) and the therapists rating of the alliance (r=-0.51).
At the point of outcome there was a negative association between the client ratings on the HAq-II and OQ45 Social Role at intake (r=-0.47) and outcome (r=-0.45), and Symptom Distress at outcome (r=-0.56).
At the point of outcome there was a negative association between the therapist ratings on the HAq-II and all OQ45 subscales at outcome.
All associations were significant at p=<.05, and all contributed to significant relationships between the HAq-II and the OQ45 Total Score.

16. Patient Health Questionnaire for Depression and Anxiety-4
This is an ultra-brief screening tool that we are evaluating, and using in another research project.
The measure often gets ‘forgotten’.
Suggested interpretation: scoring 3 or more on either the two anxiety items or the two depression items indicates a need for specific assessment.
Intake: n=58, anxiety mean=3.56(1.6), depression mean=2.87(1.8)
Outcome: n=18, anxiety mean=2.22(1.9), depression mean=1.61(1.5)

17. OQ45 x PHQ-4
Depression – significant relationship between PHQ-depression and BDI-II (r=0.62, p<.05), but not the depression items on the DASS.
Anxiety – significant relationship between PHQ-anxiety and anxiety items on the DASS (r=0.74, p<.05), but not the BAI.
Kim, Beretvas & Sherry (2010) identify
two factors in the OQ45 associated with depression (Negative Self-Worth, 10 items; Loss of Interest, 5 items). There was a significant association between the PHQ-depression score and 11/15 OQ45 depression items.
one factor in the OQ45 associated with anxiety (Anxiety/Somatization, 8 items). There was a significant association between the PHQ-anxiety score and 5/8 OQ45 anxiety items.

18. What We Have Learned So Far …
Process
Collecting standardised monitoring data is difficult
Of 512 OQ45s, 53 (10.3%) contained errors.
Pro-rating errors - 22 (4.2%), largely resolved with training.
Arithmetic errors – 31 (6.1%), ongoing.
Collating and managing progress data in ‘real-time’ is hard, but important.
What helps?
Using the data (this is very IMPORTANT)
Integration of data into supervision
Engaging with the client

19. What We Have Learned So Far …
Our norms for the OQ45 are broadly comparable to those found overseas … so we will continue using it
The therapy that our students/interns are able to offer seems to be generally effective (in reducing OQ45 scores), Consumer Feedback suggests it is also useful to our clients.
There are some interesting associations between the OQ45 and the HAq-II scores. While not particularly happy with the HAq-II we are keen to continue monitoring Therapeutic Alliance, and working with it as a metric within therapy.