Evaluation and combined methods research Geoff Lindsay ARM

What is combined (mixed) methods research? Can it be justified? Examples of studies Impact Conclusions overview

Quantitative and qualitative Designed to optimise strengths, limit weaknesses Research question(s) driven –Major and subordinate Nature of combined methods

If only quantitative –Why? Are they satisfied If only qualitative –How many? Representative? What works but also –reasons –and what context(s) 4

designs Exploratory –Focus groups –Identifying the research questions in detail ? Explanatory –Combination of quant & qual –More complex sequential designs –Convergence –Which is emphasised – quant or qual? Eg follow up of quant by quall to identify ‘why?’ Triangulation –concurrent 5

Complexity of the project –Programme evaluations produce a number of questions –require a number of methods to address Method variation within a paradigm –E.g. Focus groups → individual interviews –Observations → interviews → focus groups 6

teams Necessary and enhance CM But need to ensure a real ‘team’ not a ‘collection’ for optimising the research –Parallel, semi-detached approaches can work if managed well –Some examples: 7

Pathfinder , national roll-out (PEIP) Study of parenting programmes for which there is prior evidence of efficacy –Webster-Stratton’s Incredible Years –Triple P –Strengthening Families, Strengthening Communities –Families and Schools Together (FAST) –Strengthening Families Programme Parents with child aged 8-13 presenting or at risk of having behavioural difficulties 8 1. Parenting Early Intervention Programme (PEIP)

Pathfinder To examine the effectiveness of the roll out of each programme across LAs in England –The effectiveness of each programme when rolled out –The effectiveness of this delivery model To identify implications for future development across the country Processes as well as outcomes Our results led to DfE funding ALL LAs to implement parenting programmes PEIP –Repeat for national roll-out – all English LAs 9 Aims of the evaluation

methods Quantitative –Parent completed pre- and post-parenting group questionnaires –Numbers of groups/parents supported Qualitative –Over 400 semi-structured interviews with parents, facilitators, strategic and operational leads, head teachers – 10

Evidence - efficacy Randomized control trials Extensive evidence (though varied across programmes) for efficacy –Outcomes under optimal conditions The PEIP provided evidence for effectiveness under real life conditions, on a national scale

Design of PEIP [Pathfinder (Wave 1, )] 47 LAs: –All 23 Wave 2 LAs, 24 Wave 3 LAs Pre-, post- parenting group measures plus 1- year follow up Interviews with parents, professionals Analysis of the programmes

Quantitative Pre- and post group questionnaires to parents –Child behaviour (SDQ) –Parent well being Warwick and Edinburgh Mental Well–being Scale –Parenting skills Parenting Scale

Did PEIP target the ‘right’ parents? Most (91%) were the biological parents to the target child, and 85% were female Wide range of minority ethnic groups (19% vs 8% nationally) High proportion single parents (44% vs. 24% national statistics) and two-thirds living in rented property (63% vs. 27%) 54% no qualifications or only some GCSEs, but 20% with HE qualifications and 11% with degrees High levels of support needs: 49% had seen GP in last 6 months and 21% had seen a social worker Low levels of mental well-being: 75% of parents scored below the national median.

Did PEIP target the right children? Mean age 8.6 years (but wide range: 37% age 0-7; 54% age 8-13; 9% age 14+) 62% boys 49% entitled to FSM (16% nationally) 12% had statements of SEN (3% nationally) 31% reported additional educational support in school High % had behavioural, emotional and social difficulties with a negative impact on everyday life (home, friendships, classroom learning, leisure and burden on family) e.g. 61% vs national 13% with conduct problems

SDQ scores at pre-group (% abnormal) nationalPEIP Emotional symptoms Conduct problems Hyperactivity Peer problems Total difficulties Prosocial scale Impact score p <.001 in all cases

Did the parent and child outcomes improve after the groups? All effects are statistically highly significant Effect size is a standardised way of presenting the change in outcomes: <0.2 small, 0.5 medium; 0.8+ large. Parent outcomes show the largest effects because directly targeted

Quantitative data: conclusions Clear evidence of success of the PEIP –Parent and child improvements across a range of important dimensions –Parents very positive about the group experience (those who completed) –Maintained positive outcomes one year later

The Qualitative data A total of 429 interviews with LA strategic leads and/or operational leads; other professionals involved in parenting support,; parenting group facilitators; school representatives & parents Changes noted in parents’ own behaviour included: –Setting boundaries and applying them consistently –Giving more praise –Keeping calm and not shouting –Giving instructions to children in clear terms so they understand what is required –Spending more time in talking to their children –Having more ‘family time’

Positive changes seen in children’s behaviour included: –Increased confidence –More considerate of other people’s feelings –More calm and more open –Improved attitude towards parents and siblings –More often compliant when asked to do things –Improved school attendance

LA factors & quality outcomes Strategic leadership & operational co-ordination combined supported roll-out of the PEIP in LAs. Where these were not in place, the PEIP was less efficient in organising groups & reaching parents –Strategic leadership, including the existence of a parenting strategy, helped establish the PEIP and support it through the roll out –Strategic leadership meshed the PEIP with existing priorities & infrastructure.

LA factors & quality outcomes Operational co-ordination helped PEIP to be delivered across an LA. Models differed – some LAs had one central co-ordinator & others divided the co-ordination role on a geographic basis. –However, some central co-ordination enabled LA- wide oversight of processes & outcomes. –The PEIP LA which returned most pre & post course CEDAR booklets by far adopted a model of area co- ordinators taking responsibility for their area, but were also linked to a strategic centre with oversight.

Outcomes Overall PEIP was effective on all our measures –And improvements were maintained one year later All 4 programmes were effective –with some differences between them Large variation between LAs in numbers of parents supported - cost effectiveness varied as a result Average cost per parent was £1658 for those completing, lowest was £534 in one LA Important factors identified to guide policy and practice Very successful government initiative and clear evidence in support of use of these programmes by LAs

Reference Lindsay, G., Strand, S. & Davis, H. (2011). A comparison of the effectiveness of three parenting programmes in improving parenting skills, parent mental well being and children’s behaviour when implemented on a large scale in community settings in 18 English local authorities: The Parenting Early Intervention Pathfinder (PEIP), BMC Public Health 2011, 11:962 doi: /