1. The Jack Brockhoff Child Health and Wellbeing Program, The McCaughey Centre, Melbourne School of Population Health, University of Melbourne EVALUATING COMPLEX INTER-AGENCY PARTNERSHIPS: A MIXED METHODS APPROACH Australasian Evaluation Society September 1st 2011 Elisha Riggs, Karen Block, Deborah Warr, Lisa Gibbs

2. Overview of presentation Background –Social disadvantage and partnerships –Challenges for evaluating partnerships Evaluative research –Ucan2 partnership –Methodological approach –Findings and partnership map –Challenges for partnership evaluation –Conclusions

3. ‘The Australian Government’s vision of a socially inclusive society is one in which all Australians feel valued and have the opportunity to participate fully in the life of our society’ (www.socialinclusion.gov.au)www.socialinclusion.gov.au WHO Commission on SDoH: redistribute resources across organisations and systems; invest in collaborative action Integrated approaches make good sense given the complex and shifting circumstances of disadvantage ‘More is achieved, more efficiently, and to greater effect, when individuals and organisations work together’ (Stokes and Tyler 1997) ‘Joined-up’ approaches can coordinate and channel services and support (Blair 1997, Burnett & Appleton 2004) Promoting social inclusion through partnerships

4. Challenges for evaluating partnerships Evaluation of partnerships aiming to address complex social issues such as social exclusion is critical Evaluations provide opportunities for internal monitoring to promote the success and sustainability of the partnerships and for organisations and policy-makers to demonstrate the value of the invested resources Remains ongoing challenges in developing sufficiently complex evaluation frameworks that account for the complexity in partnerships

5. Ucan2 partnership Inter-agency partnership program aimed at promoting social inclusion opportunities for newly arrived refugee youth Multiple evaluation components: –Organisational partnership evaluation –Student evaluation Qualitative component Wellbeing survey Social network analysis Economic evaluation

6. Theoretical framework Partnerships require varying degrees of cooperation between partner organisations The Partnership Continuum conceptualises progressive scales of integration (VicHealth) Levels of integration range from networking (most informal), through to coordinating, cooperating and collaborating (the most formal, involves higher degrees of planning and sharing of resources) Partnership Continuum

7. Methodological approach Several distinct components –Organisational ethnography – 2 years, 6 month lead-in –Document analysis –VicHealth’s The Partnership Analysis Tool – baseline and 18 month follow-up (selected based on relevant criteria) –Semi-structured interviews – baseline and 18 month follow-up –Mapping the partnership

8. The checklist The highest possible aggregated score is 140 and the following legend suggests how scores can be interpreted: 92-140 A partnership based on genuine collaboration has been established. The challenge is to maintain its impetus and build on the current success. 50-91 The partnership is moving in the right direction but it will need more attention if it going to be really successful 0-49 The whole idea of the partnership should be rigorously questioned (VicHealth)

9. Findings Overall partnership checklist score: middle category Secondary analysis: ‘program development’: highest category and ‘program delivery’: middle category Interviews illuminated checklist results and provided deeper insight into the partnership Delivery group: Collaborating Management group: Coordinating Informed a rigorous understanding of the partnership levels within and across the inter- agency partnership

10. Follow-up No change in checklist scores Dramatic shift in qualitative results Would not have identified changes over time if just relying on checklist data –Swift action to improve governance structures –Higher levels of trust between partners –Shift in wanting ‘equal’ partnership towards identifying the need for leadership I don't think it's anything about personalities, it's more about processes. I think it's about trusting the processes and the complexities that are involved (Program Development)

11. Mapping the partnership

12. Addressing the complexity of partnership evaluation Relationships are non-linear and multi- dimensional, constantly repositioning themselves within a changing and evolving environment Inter-agency partnerships need to recognise and acknowledge that the partnership they have established comprises individuals with different backgrounds Evaluative processes must adjust to these complexities

13. Addressing the complexity of partnership evaluation (cont) Importance of linking evaluation to theory No single evaluation tool identified that captures the complexities and realities of working in complex partnerships Using the Partnership Continuum and mapping the partnership provide reflexive opportunities

14. Conclusions Increasing governmental support for partnerships to address complex problems Evaluative research involves theory development and testing Overall there is a synergistic effect of using multiple data collection methods A mixed method approach is required in order to develop conceptual understandings of the complexity of real-world partnerships

15. For further information: Elisha Riggs, Research Fellow The University of Melbourne eriggs@unimelb.edu.au