Assessment of Intervention Outcomes Workshop: Thursday 7th February IYW Conference, Llandrindod Wells Dr Dave Daley & Dr Tracey Bywater

2 Overview of workshop Rationale for conducting your own evaluations How to manage the numbers Suggested evaluation material Rationale for conducting your own evaluations How to manage the numbers Suggested evaluation material

3 Why evaluate clinical work? NHS Trusts need require you to conduct evaluation It is imperative to be able to demonstrate improvement associated with your groups Helps to maintain funding and/or win new funding NHS Trusts need require you to conduct evaluation It is imperative to be able to demonstrate improvement associated with your groups Helps to maintain funding and/or win new funding

4 Is it difficult to evaluate outcome? The most important decisions to make when considering evaluation are: 1) Design 2) Evaluation measures The most important decisions to make when considering evaluation are: 1) Design 2) Evaluation measures

5 1. Design It is very important that you have baseline measures (before) and outcome measures (after) It is also important that you have the same measures on everyone It is very important that you have baseline measures (before) and outcome measures (after) It is also important that you have the same measures on everyone

6 What can we do with our evaluation scores I? Even simple spreadsheet programmes like Excel will allow you to conduct simple statistics For evaluation purposes you are most interested in change from start to end Even simple spreadsheet programmes like Excel will allow you to conduct simple statistics For evaluation purposes you are most interested in change from start to end

7 Pros & Cons of change scores Change scores are most useful Easy and simple way of evaluating change However recent suggestion that change scores on SDQ are less sensitive However formula is complicated, has not yet been fully validated and is not recommended for evaluations of small groups Change scores are most useful Easy and simple way of evaluating change However recent suggestion that change scores on SDQ are less sensitive However formula is complicated, has not yet been fully validated and is not recommended for evaluations of small groups

8 What can we do with our evaluation scores II? Change scores should demonstrate improvements in child behaviour outcome You can also begin to examine whether your groups are more effective for particular types of parents Change scores should demonstrate improvements in child behaviour outcome You can also begin to examine whether your groups are more effective for particular types of parents

9 What can we do with our evaluation scores III? Rather than do lots of fancy statistics, it is also interesting to examine scores at outcome Most evaluation measures will have scores which indicate clinical concern It is very easy but also meaningful to examine the percentage of children and/or parents who score above cut off scores before intervention and after intervention Rather than do lots of fancy statistics, it is also interesting to examine scores at outcome Most evaluation measures will have scores which indicate clinical concern It is very easy but also meaningful to examine the percentage of children and/or parents who score above cut off scores before intervention and after intervention

10 2. Evaluation Measures Which ones? Pathfinders project uses: Demographic Questionnaire Beck Depression Inventory Eyberg Child Behaviour Inventory Strengths & Difficulties Questionnaire x 2 Parenting Scale Overt Hostility Scale Teacher Questionnaire/Report Which ones? Pathfinders project uses: Demographic Questionnaire Beck Depression Inventory Eyberg Child Behaviour Inventory Strengths & Difficulties Questionnaire x 2 Parenting Scale Overt Hostility Scale Teacher Questionnaire/Report

11 Demographics What do you need to know? Do you want to compare outcomes of:- older versus younger parents? Boys versus girls? Different areas? Any other ideas? What do you need to know? Do you want to compare outcomes of:- older versus younger parents? Boys versus girls? Different areas? Any other ideas?

12 The Eyberg Child Behaviour Inventory 36 items, ages 2-16 years, 10 minutes to complete Problem score has a minimum of 0, maximum of 36 (yes/no answers, problem or not) Intensity scale - minimum score of 36, maximum 252 (scale of 1 – 7, where 1 = never and 7 = always) Clinical cut-off scores, 127 intensity, or 11 problem Purchase from 36 items, ages 2-16 years, 10 minutes to complete Problem score has a minimum of 0, maximum of 36 (yes/no answers, problem or not) Intensity scale - minimum score of 36, maximum 252 (scale of 1 – 7, where 1 = never and 7 = always) Clinical cut-off scores, 127 intensity, or 11 problem Purchase from

13 Strengths & Difficulties Questionnaire FREE, website: 10 minutes to complete, 25 items: Emotional symptoms Conduct problems Hyperactivity Peer relationship problems Prosocial behaviour An additional impact supplement Parent & teacher Normal, borderline & atypical ranges/cut-off FREE, website: 10 minutes to complete, 25 items: Emotional symptoms Conduct problems Hyperactivity Peer relationship problems Prosocial behaviour An additional impact supplement Parent & teacher Normal, borderline & atypical ranges/cut-off

14 Beck Depression Inventory 21-item elf-report inventory measuring the severity of characteristic attitudes & symptoms associated with depression Each item contains four possible responses which range in severity from 0 ( I do not feel sad) to 3 ( I am so sad or unhappy that I cant stand it) Score of = mild to moderate depression Score of = moderate to severe depression 21-item elf-report inventory measuring the severity of characteristic attitudes & symptoms associated with depression Each item contains four possible responses which range in severity from 0 ( I do not feel sad) to 3 ( I am so sad or unhappy that I cant stand it) Score of = mild to moderate depression Score of = moderate to severe depression

15 Statistics with little maths Excel Graphs Mean & SD Change scores Effect sizes Excel Graphs Mean & SD Change scores Effect sizes

16 Welsh Sure Start Data

17 Effect sizes Interven fup1 Control fup1 Differencepooled SD effect sizes BDI ECBI-P ECBI-I Cohens 1988 guidelines: difference between means divided by pooled SD. 0.3 = clinically useful change, 0.5 medium effect, 0.8 = large effect

18 Any questions?????