Dr Alex Hassett and Anna Neal May 2011 The Importance of Measuring Outcomes Related to Looked After Young People’s Emotional Wellbeing and Mental Health Dr Alex Hassett and Anna Neal May 2011

Aims of the Workshop Understanding why outcomes for looked after young people need to be measured and how this can be done The SDQ – what it measures and how it measures it GBOs – what it measures and how it measures it Uses of outcomes data for different stakeholders Limitations and things to bear in mind when using these measures Experience of scoring and interpreting Questions and answers

Data Collection on the Emotional Health of Looked After Children Since April 2008 all Local Authorities in England have been required to provide information on the emotional and behavioural health of children and young people aged 4 – 16 years who have been in care for more than one year. This data is collected through the Parent / Carer version of the SDQ, from which the Total Difficulties Score for each child is submitted to the Department for Education (formerly the Department of Children, Schools and Families) through the SSDA903 data return. The LAC CAMHS Expert Group has reviewed the SDQ process currently followed in Kent and subsequently recommended a new model for data collection. This model aims to result in an increased number of completed questionnaires and also a more practice driven approach, resulting in better outcomes.

Uses of outcome evaluation Case evaluation: To provide information about individual children and their families. Practitioner evaluation: To provide information about outcomes for the range of children and families seen by an individual clinician Service evaluation: To provide information about the outcomes of particular projects or services Strategy evaluation: To provide information about the impact of a CAMHS strategy From www.corc.uk.net

Example Uses of Outcome Data and Relevant Stakeholders Data for Information Analysis of data as an indicator of team performance / typical outcome for specific groups of children and young people according to treatment modality Data for Judgement Analysis of data to evidence positive strategic development and inform changes to practice Data for Professional & Personal Development Analysis of data to facilitate reflection on individual cases and inform changes to clinical practice. Analysis of data to map progress and trajectories of change for children and young people E.g. Commissioners / Service Managers E.g. Service Managers / Clinicians / Service Users E.g. Clinicians / Service Users

What is the SDQ? - Versions The SDQ was devised by Robert Goodman (1997, 1998) and is a brief behavioural screening tool used to measure the severity and impact of difficulties. There are Parent rated, Self (young person) rated and Teacher rated versions of the tool. Within the Parent and Teacher rated SDQs, there are various versions for Parents / Teachers of children and young people of different ages. The Self rated SDQ is intended for use with young people aged 11 to 17 years old.

What is the SDQ? - Subscales Section 1 consists of 25 items about psychological attributes which are divided into 5 sub-scales; Emotional Symptoms, Conduct Problems, Hyperactivity, Peer Problems and Prosocial. Some statements are positive and some are negative. Each statement is rated using a 3 point Likert Scale ranging from not true to somewhat true to certainly true. The score for each response varies across items.

What is the SDQ – Total Difficulties Score A ‘Total Difficulties Score’ (TDS) can be generated for each version of the SDQ, by summing the scores of four scales (all except the ‘Prosocial’ scale), to give a score between 0-40 The TDS can be used to classify scores into ‘normal’, ‘borderline’, and ‘abnormal’ ranges. An abnormal score on one or more TDS can be used to identify ‘cases’ with mental health disorders (however, 10% of a community sample will score in the abnormal range for any given score and a further 10% will score in the borderline range). 

SDQ Data Total Difficulties Score A Total Difficulties Score (TDS) can be used to quantify severity of difficulties and is generated by summing the scores of four scales (all except the ‘Prosocial’ scale) on each version of the SDQ, to give a score between 0-40 ‘NORMAL’ ‘BORDERLINE’ ‘ABNORMAL’ Parent Rated SDQ 0-13 14-16 17-40 Self Rated 0-15 16-19 20-40 Teacher Rated 0-11 12-15 16-40

What is the SDQ? – Impact Supplement Several versions of the SDQ are available with an ‘Impact Supplement’ on the back page. Respondents are asked whether they think the young person has a problem, and if so, enquires further about chronicity, distress, social impairment, and burden to others. This provides useful additional information for clinicians and researchers with an interest in psychiatric caseness and the determinants of service use.

What is the SDQ? – Impact Supplement Items on overall distress and social impairment can be summed to generate an impact score between 0-10 on the Parent and Self rated versions and 0-6 on the Teacher rated version of the SDQ.

What is the SDQ? – Impact Supplement Responses are scored accordingly; Not at all =0, Only a little = 0, Quite a lot = 1, A great deal = 2 When respondents have answered no to the first question on the impact supplement (when they do not perceive the child to have emotional or behavioural difficulties) they are asked not to complete the following questions on resultant distress or impairment and so the impact score is automatically scored as 0. Impact scores can be used to classify cases into ‘normal’, ‘borderline’, and ‘abnormal’ ranges. A total impact score of 2 or more is abnormal, 1 is borderline and 0 is normal.

Not at all / Only a little SDQ Data Impact Score Quantifies the degree of impact caused by the child’s difficulties on the child / family / classroom Items on overall distress and social impairment are be summed to generate an impact score Parent and Self rated versions 0 -10, Teacher rated version 0-6 Not at all / Only a little Quite a lot A great deal ‘Overall distress’ & ‘Social impairment’ All versions 1 2 ∑ ‘Overall distress’ & ‘Social impairment’ scores ‘NORMAL’ ‘BORDERLINE’ ‘ABNORMAL’ 2(+)

SDQ Sensitivity, Reliability and Validity The items contained in the SDQ are focussed on behavioural descriptions which aim to capture detail about emotional, conduct, hyperactivity and peer problems. The SDQ has good psychometric properties with normative data available based on populations from Britain, Sweden, Finland, Germany, Australia and the USA. Over 50 different language versions of the SDQ are available.

SDQ Sensitivity SDQ are not particularly sensitive to capturing detail about psychosis, eating disorders and learning disability. SDQs are not intended to be used with children and young people with severe levels of learning disability. If these types of problems are suspected, alternative or at least additional measures may need to be utilised e.g. Clinician rated measures such as HoNOSCA, Eating Disorder Examination (EDE), LD specific measures such as Sheffield Learning Disabilities Outcome Measure (SLDOM), Nisonger Child Behaviour Rating Form (Nisonger CBRF)...

SDQ Data Problem Improvement & Service Usefulness (Time 2 / Follow Up SDQs) This data can be used to give an overview of how the service is typically perceived by young people / parents / teachers in terms of improving problems and helpfulness in other ways Since coming to the clinic, are your child’s problems... Has coming to the clinic been helpful in other ways, e.g. Providing information or making the problems more bearable... Much worse A bit worse About the same A bit better Much better Not at all Only a little Quite a lot A great deal

What is the GBO? The Goals Based Outcome measure (GBO) was developed jointly by CORC and Hertfordshire Partnership NHS Trust as an attempt to capture progress towards a goal in the work with children, young people and their families. Goals are rated at Time point 1 (usually prior to intervention) and again at Time point 2 (Time 1 plus 6 months or at the end of intervention; whichever is first) The respondent uses a ten point scale to rate how close they feel to achieving their goal (0= goal not met at all, 10 = goal reached).

What Is the GBO? Goals are set by the young person (aged 11-16) or their family, rather than the practitioner / clinician. As such, the measure gives a different perspective to clinical outcome measures and can measure different sorts of change that might not always be captured using only behavioural or symptom based measures (e.g. Parent / carer confidence at managing a young person’s difficulties). GBOs enable measurement of the effectiveness of an intervention across a variety of settings and with a variety of service users. The goals could be those of a young person in individual therapy, through to the goals of a staff team in a care home receiving consultation from a service.

What is the GBO? For many people the first step is identifying some potential goals - many of the more goal focused therapies have developed techniques to help with goal setting. Good examples are the ‘miracle question’ used in solution focused therapy or by asking what a person might change if they were given ‘three wishes’. The key rule is that the person setting the goal is the person doing the work. Therefore, an example of an acceptable goal for a staff team might be around managing the behaviour of a child if the work is on what they can do differently. However, it would be unacceptable for the team to set a behaviour change goal if the focus of the work is individual therapy with the child in question.

GBO Protocol Set the goals over the first three sessions of the intervention/assessment Record up to three goals Once a goal is agreed, record how close the client feels they are to reaching the goal (this is the time one (T1) rating) Record the rating on a scale from zero to ten where ‘zero’ means the goal is not met in any way, ‘ten’ means the goal is met completely and a rating of ‘five’ means they are half way to reaching the goal. At the end of the intervention – or after six months – (whichever is first), record again how close to reaching the goal the client now feels they are (this is the time two (T2) rating) Again, record the rating on a scale from zero to ten where ‘zero’ means the goal is not met in any way, ‘ten’ means the goal is met completely and a rating of ‘five’ means they are half way to reaching the goal.

GBO Data Goal one (low mood) T1 = 2/10, T2 = 7/10, therefore GBO = 7 – 2 = 5 Goal two (self harm) T1 = 3/10, T2 = 5/10, therefore GBO = 5 – 3 = 2 Goal one (Young Person) T1 = 1/10, T2 = 7/10, therefore GBO = 7 – 1 = 6 Goal two (Parents) T1 = 3/10, T2 = 9/10, therefore GBO = 9 – 3 = 6 The larger the GBO (difference) score is, the greater the degree of perceived movement / change. If the score is a positive number, movement has been in the desired direction; towards achieving the goal, if it is a negative number, movement has been away from the goal. If goals are collected from multiple perspectives, CORC advise they should be ranked in order of respondent; the child / young person mother father other family member Consideration would need to be given to where in the order goals set by a carer should be placed. This would partly depend on who the other respondents were e.g. Child, biological parent etc...

Subjectivity and Perverse Goal Setting Goals, by their nature, are subjective. The difficulty with such subjective measures is that their scientific validity is difficult to establish – as a young person moves towards a goal it is difficult to be sure that what they rate on the ten point scale reflects a “true” shift The strength is that in much work with young people it is their subjective view of change that is arguably a vitally important measure of success. It is easy to be seduced into ‘collaborating’ with clients in perverse ways, to set ‘easy’ goals that are more achievable – not to help provide users with a sense of achievement, but to make outcomes look good. This process can equally be at play from the young person’s / carer’s side; if they feel that setting complex goals may lead to them receiving a ‘better’ service, or if they fear that showing progress towards a goal may lead to a useful service being stopped.

References www.corc.uk.net/ Department for Education (2011). Guidance on Data Collection on the Emotional Health of Looked After Children. Retrieved 29th March 2011 from: http://media.education.gov.uk/assets/files/pdf/s/statistical%20returns_sdq%20guidance%20update%20dec%202008.pdf Goodman, R. (1997). The Strengths and Difficulties Questionnaire: A Research Note. Journal of Child Psychology and Psychiatry, 38, 581-586 Goodman, R., Meltzer, H. & Bailey, V. (1998). The Strengths and Difficulties Questionnaire: A pilot study on the validity of the self-report version. European Child and Adolescent Psychiatry, 7, 125-130 Goodman, R. (1999). The extended version of the Strengths and Difficulties Questionnaire as a guide to child psychiatric caseness and consequent burden. Journal of Child Psychology and Psychiatry, 40,791-801 Goodman, R., Renfrew, D. & Mullick, M. (2000). Predicting type of psychiatric disorder from Strengths and Difficulties Questionnaire (SDQ) scores in child mental health clinics in London and Dhaka. European Child and Adolescent Psychiatry, 9, 129-134

References Goodman, R., Ford, T., Simmons, H., Gatward, R. & Meltzer H (2000). Using the Strengths and Difficulties Questionnaire (SDQ) to screen for child psychiatric disorders in a community sample. British Journal of Psychiatry, 177, 534-539 Goodman, R. (2001). Psychometric properties of the Strengths and Difficulties Questionnaire (SDQ). Journal of the American Academy of Child and Adolescent Psychiatry, 40, 1337-1345 Mathai, J., Anderson, P. & Bourne, A. (2002). The Strengths and Difficulties Questionnaire (SDQ) as a screening measure prior to admission to a Child and Adolescent Mental Health Service (CAMHS). Australian e-Journal for the Advancement of Mental Health, 1:3 Mathai, J., Anderson, P. & Bourne, A. (2003). Use of the Strengths and Difficulties Questionnaire as an outcome measure in a child and adolescent mental health service. Australasian Psychiatry, 11, 334-337 Meltzer, H., Gatward, R., Corbin, T., Goodman, R. & Ford, T. (2003). The Mental Health of Young People Looked After by Local Authorities in England. Retrieved 29th March 2011 from http://www.esds.ac.uk/doc/5280/mrdoc/pdf/5280userguide.pdf http://www.rcpsych.ac.uk/clinicalservicestandards/centreforqualityimprovement/qnic/qnicrom/qnicromoutcometools.aspx www.sdqinfo.org/