1. Measuring Change and Strengthening Practice in Therapeutic Residential Care Edith Loch, General Manager Client Outcomes, MacKillop Family Services Doug Faircloth, Managing Director, Verso Consulting OPEN Sector Research Symposium October 2017

2. Presentation Overview
TRC Monitoring Project overview
Outcomes monitoring system
Key Findings
How we connect meaning to the data
Using the data in practice
Monitoring program effect

3. TRC Monitoring Project
Methodology based on Victorian Evaluation of TRC Pilots ( ) using validated tools – HoNOSCA, SDQ and Brann Likert
Utilises a web based portal developed by Verso and The Carers Phone
11 MacKillop TRC homes in Victoria and 1 in NSW. Data collection commenced in Feb 2016 (approx. 30,000 sets of entries to date)
Provides point in time (real time) data and time sequenced data that will track progress over time; Residential staff record their immediate observations of each young person using rotating questions – 3 times per day (40 secs to 1.5 mins to populate)
The questions are based on the Brann Likert scales that measures behavioural change
Responses are viewed in graph form for each young person in real time with parameters for review being customized
Staff consider multiple and related data sets to support reflective practice and case work
Expert analysis and practice workshops have been provided by Dr Peter Brann
HONOSCA questions are cross referenced on the screen to the relevant reports/daily observations entered by the residential workers.

4. Daily observations – questions asked
42 different questions - clustered over the week across 3 shifts per day
Every shift:
relationship with carers, risk taking, unhappy/tearful, aggression
Every day:
Morning: was the young person in the TRC home as authorised last night, what time did they return home, how did they sleep, what time did they go to sleep?
Afternoon: did the young person attend school as expected, was the absence authorised etc., recreation, physical activity, peer relationships, connection to community, hygiene, sense of self
Evening: Nutrition, risk taking and type, contact with family, connection with community
Every week:
Connection to culture and Indigenous connection

12. Add in same info by distribution??

13. What is the young person’s sense of self today?

14. Key Findings: demographics
42 young people in MacKillop TRCs in scope
Differing gender, age mix and placement history in MacKillop TRC homes compared to the statewide TRC evaluation population as follows:
a higher proportion of males (49% compared to 34% in evaluation population)
the average age at entry to MacKillop TRC is 15.4 years compared to 13 years
Prior to entry to the MacKillop TRCs, our young people had higher numbers of prior placements/placement instability
TRC program model fidelity is an issue

15. Outcomes Measures: HoNOSCA

16. Outcomes Measures: SDQ
We have collected Strengths and Difficulties Questionnaire (SDQ) data and compared these to the SDQ scores for the children and young people in the DHS TRC Pilot Evaluation.
Five domains are reported:
peer problems,
hyperactivity,
conduct problems,
emotional problem
pro-social functioning.

17. Outcome Measures: Brann Likert Scales
The base line data and early time sequenced data reporting indicates that the children and young people in the MFS TRC program have elevated symptom severity above entry into specialist mental health services and are showing improvements at a similar and comparable rate of children and young people in the Tasmanian and Victorian TRC evaluations

18. Time series: Quality of sleep

19. Time series: Best quality of family contact

20. Time series: Evidence of risk taking

21. How we connect meaning to the data
Data analysis is used to support reflective practice, assessment and therapeutic plans
Multi-disciplinary workshops led Dr Peter Brann workshops
explored how to use data, its meaning and how to put it into practice to ensure good outcomes 

22. Case examples: sharing data with young person
The sleeping pattern graph was shared and discussed with the young person resulting in a meaningful discussion and self reflection and an action plan with strategies to self regulate that has resulted in positive improvement in sleep and school attendance.
Staff feel that it is more “kid friendly” to show the graphs and through this conversation with her also they identified her anxiety prior to school.   They regularly talk this through with her now and have purchased a trampoline that she uses before school to self-regulate.

23. Further practice examples
Use of data to inform whole of house strategies
The data on time of return home was used to develop strategies with approval of the care team and DHHS around transport of young people in a TRC home. As a result  they have seen young people getting home earlier and to sleep earlier. The data has been available to track and support this.
Mediating staff views and responses
The examination of the Verso data by staff in reflective practice has identified the different staff views on a range of issues, e.g. what level of drug use is rated as serious or minor etc. This have lead to conversation regarding different views and responses of staff.   Examination of the data lead to this realisation and conversation.

24. Key learning
The data collection and reporting has been a contributing factor in galvanizing an organisational commitment, common language and a focus on the program elements of therapeutic care
The data has been integral to reflective practice by care staff, therapeutic specialist and management
The structure of answering the daily questions has resulted in care staff reflecting on a wider range of positive and problematic issues than previously (without being asked the questions)
Strong governance structures have now been put in place across the organisation to consider and monitor the essential elements and client outcomes
This system has demonstrated that research and evaluation tools can be successfully embedded in a residential setting through IT applications

25. For further information
Edith Loch, General Manager Client Outcomes, MacKillop Family Services: Doug Faircloth, Managing Director, Verso Consulting: