1. Evaluating the Outcomes of Health and Social Care Partnerships: the POET Approach. Helen Dickinson h.e.dickinson@bham.ac.ukh.e.dickinson@bham.ac.uk Health Services Management Centre, April 2007

2. Doubts over value of £3bn Sure Start The first major evaluation of the government's flagship £3bn Sure Start programme… has revealed no overall improvement in the areas targeted by the initiative. Although some Sure Start schemes were successful, an independent study … revealed that Sure Start as a whole failed to boost youngsters' development, language and behaviour. It also showed children of teenage mothers did worse in Sure Start areas than elsewhere.

3. Doubts over value of £3bn Sure Start cont. The findings… represent only an early snapshot of the programme's effectiveness, and academics involved in the £20m evaluation emphasise that they do not mean the scheme, which varies widely around the country, will not succeed in helping children in deprived areas in the long term

4. Outline of session  Policy Context  What is partnership working?  The evaluation challenge  The POET approach  Findings so far…

5. The Prevalence of Partnerships  5,500 individual partnership bodies.  60 types of different partnerships  Partnerships may spend around £15-20 billion per year.  Approx 75,000 places on partnership boards  Underestimate of the extent of partnership working? ( Sullivan and Skelcher, 2002)

6. The welfare state Five Giants2006 equivalent WantSocial security DiseaseNHS IgnoranceEducation IdlenessEmployment/leisure SqualorHousing/regeneration

7. Government by hierarchy Government Health Social Security

8. Services organised around the person Citizen NHS HousingIncome

9. Evidence-based policy and practice  ‘What counts is what works’  I.e. government commitment to make policy and promote practices on the basis of solid evidence, rather than by ideological fiat.  The Cabinet Office declares in the Magenta Book (official guidance notes for policy evaluation and analysis), ‘evidence-based principles are at the heart of the Government’s reform agenda for better policy making and policy implementation’ (Cabinet Office, 2003: 17).

10. The policy context (in theory) Partnership Better Services? Better Outcomes?

11. A loss of confidence in the concept?  Academic critiques of the partnership concept.  Areas with long-lasting and high profile relationships recently encountered difficulties. ‘Partnerships also bring risks. Working across organisational boundaries brings complexity and ambiguity that can generate confusion and weaken accountability. The principle of accountability for public money applies as much to partnerships as to corporate bodies. The public needs assurance that public money is spent wisely in partnerships and it should be confident that its quality of life will improve as a result of this form of working’ (Audit Commission, 2005: 2).

12. Outcomes, process and partnership working  Systematic review of 491 papers found a dearth of evidence to support the notion that joint working between the NHS and social services is ‘effective’. (Cameron & Lart, 2003)  Literature review of health and social care partnerships (Dowling et al, 2004) found: - Evaluations centred on process and organisational issues, - Little emphasis on user outcomes  Based on an assumption that if process is smooth then expected outcomes will be delivered?  Or, selecting appropriate outcomes to evaluate partnerships is a fundamentally difficult task?

13. The policy context questioned Partnership Better Services? Better Outcomes? (Do they? How? For whom? In what contexts?)

14. What is partnership working? (1)  Partnership is a very UK-based (New Labour?) term for an internationally encountered phenomenon relating to the relationships between bodies working together to deliver services.  Not only health and social care manifestation. Other terminology includes:  Joint-working, seamless- working, joined-up thinking, interprofessional working, multi-professional working, integrated teams, multi-agency working, interagency working, collaboration and interdisciplinary working.

15. What is partnership working? (2) “Joint working arrangement of partners who are otherwise independent bodies and these co-operate to achieve a common goal” (Audit Commission, 1998) Number of different forms of relationships which can exist between organisations:  Hierarchies  Markets  Networks “Methodological anarchy and definitional chaos” (Ling, 2000: 83). Partnerships

16. Partnership map (1) Health Health & LA in & LA in & & social civic ‘whole’ LA care society system Merger Partnership organisation Joint management Co-ordination Consultation Information Depth Breadth

17. Partnership map (2) Health Health & LA in & LA in & & social civic ‘whole’ LA care society system Merger Partnership organisation Joint management Co-ordination Consultation Information Depth Breadth Adult services Children’s services

18. Group Exercises

19. The evaluation challenge  How is ‘partnership’ defined?  How do we know if partnerships are successful? What is it that defines success?  Complex policy initiatives in complex contexts  Multiple stakeholders and perspectives (individual, organisational, service user, carer, governmental…)  Proving you’ve prevented something?  Long-term programmes/outcomes  What would have happened anyway? Etc etc etc.

20. Aims of POET project  Do partnerships improve outcomes for service users?  Design a generic evaluation toolkit that can test this assumption  Strong theoretical underpinning

21. POET: Partnership Outcomes Evaluation Toolkit 1. Online staff survey - Tests the ‘health’ of the partnership - Identifies developmental and problem areas - Highlights areas for celebration - Surfaces all the underlying ‘partnership theories’. 2.User survey - Tests whether the ‘correct’ outcomes have been identified. - To what extent have these outcomes been met? - Retroductive techniques identify generative mechanisms

22. Effective partnership working: the three levels I O S From Glasby (2003) Individual Organisational Structural

23. POET  Formative through process-based evaluation by staff members. Highlights areas that need improvement (often fairly simple) and allow celebration where things have gone well). Provides a benchmark to check back against over time period.  Summative through work with service users and carers – what is it the partnership specifically does for those who use its services?

24. Partnership map Health Health & LA in & LA in & & social civic ‘whole’ LA care society system Merger Partnership organisation Joint management Co-ordination Consultation Information Depth Breadth

25. Lessons so far…for staff members  Working with different professionals, from different backgrounds with their associated values is difficult, but can be very rewarding in an individual sense through learning about other professions and expanding knowledge of client group.  “Not sure why we are here......feel it was just because we were easy to move and education had to put someone here!”  “However there are blips - health staff have lines of accountability other than to integrated service management”.

26. Lessons so far…outcomes  Conceptualising outcomes is a difficult task. - Culturally difficult to do – i.e., thinking in such a different way. However, a number of people did value this – took arguments away from structures and relates to outcomes.  Number of the conceptualised outcomes relate more to organisational and structural issues that user outcomes per se. -Integrated management system -Integrated IT system -Pooled budget -More team meetings etc

27. Lessons so far…service user pathways  ‘Deep’ partnership relations strength in forming care pathways around the service user – but tensions in relating to wider partners (integration is always simultaneously a process of fragmentation).  Internalisation of complexity within teams – needs quite specific support structures.

30. Lessons so far…impact on users  “I am more aware of what is available and don't feel isolated and left to deal with it. Thank you”.  “Received brilliant services, clearly explained and very helpful. We miss our key-worker”.  “it is an early stage to provide any real evidence for it. However children and their parents like this approach. It has ability to coordinate the service provision and reduce duplication”.

31. Key Lessons  Where can relationships be most effective – where is the core business / overlap of business?  Time periods – when would you expect to see impacts.  Partnerships are necessary under some circumstances and for particular client groups as a consequence of organisational forms.  However, this way of working can be complex and difficult, requiring quite different mechanisms of training and support.  What about wider links? Who to include is always a concurrent decision about who to exclude.  This more ‘nuanced’ view of working in this way is often what is missing from policy guidance.

32. The F word! (finance that is)  A number of the partnerships talked about financial matters.  In the private sector most partnership working is about ££ (reducing duplication, larger budget = more bargaining power etc).  Not a problem if partnerships are about saving money – but be more clear about this, otherwise risk creating expectations that can’t deliver on.

33. If partnership is the answer – then what is the question?

34. Next steps  Complete field testing and refinement of POET.  Write up findings of field testing for dissemination (journals, conferences, books)  Launch tool nationally in conjunction with the Care Services Improvement Partnership (CSIP) – will be made freely available for organisations to use through http://hsmcfs3.bham.ac.uk/questionnaire http://hsmcfs3.bham.ac.uk/questionnaire