Australasian Evaluation Society Conference Sydney August-September 2011 Strength of partnerships: a mixed methods approach to evaluation Presented by: Date: Geraldine Marsh Virginia Lewis Kate Silburn Jenny Macmillan

Centre for Health Systems Development (CHSD) at the Australian Institute for Primary Care & Ageing (AIPCA) at La Trobe University has been evaluating initiatives that involve partnering at many levels. Examples of the different types and levels of partnering initiatives we have evaluated include:  Hospital Admission Risk Programs (HARP)  Early Intervention for Chronic Disease projects  Community Initiatives for National Suicide Prevention Strategy  Two large-scale evaluations of statewide primary health system development strategies in Australia: Primary Care Partnerships (Vic) and Connecting Healthcare in Communities (CHIC) 2

Role of Government  Governments have an important role in creating or enabling the foundations of effective systems for organised primary health care service delivery (Takach, Gauthier, Sims-Kastelein, & Kaye, 2010; World Health Organisation, 2008).  This requires deliberate and sophisticated policy processes that maximise the potential within complex systems and the capacity within these systems for innovation (McDaniel & Driebe, 2001; Swantson, et al., 2010; World Health Organisation, 2008).

Partnering and Primary Health Care Reform Current reform is based on setting up (new) collaborative structures and relationships  Other geographically based formal and informal governance and organisational arrangements exist  Evaluative research is required to assess the emerging ML/LHN and their relationships to existing (or other developing) partnerships and the effects on PHC delivery and consumer outcomes 4

How do we assess value of partnering? Evaluating partnering (collaboration, etc.) is important to demonstrate it adds value and is worth investment  Attributing the contribution of relationships between organisations to health service delivery and health outcomes is difficult  First step is being able to capture the nature of the partnership and describe its “effectiveness” in a systematic way

Measures Literature on barriers and facilitators leads to an understanding of characteristics of functional partnerships, general consensus about what good partnerships look like. But how to capture systematically and usefully is difficult  Descriptive qualitative review  Network mapping technique  Questionnaires based on presence of characteristics  VicHealth Partnerships Analysis Tool  Nuffield Partnership Assessment Tool (Hardy, Hudson and Waddington, 2003)

Commonly used questionnaires VicHealth Partnerships Analysis Tool A tool for establishing, developing and maintaining productive partnerships 7 domains rated on a 5-point Agree/Disagree scale: 1. Determining the need for the partnership 2. Choosing partners 3. Making sure partnerships work 4. Planning collaborative action 5. Implementing collaborative action 6. Minimising the barriers to partnerships 7. Reflecting on and continuing the partnership Different possible methods of completion: Leaders present views for discussion; Individually completed; as a group activity

Nuffield Partnership Assessment Tool (Hardy, Hudson & Waddington, 2003) Six domains rated on 4-point Agree-Disagree scale 1. recognise and accept the need for partnership 2. develop clarity and realism of purpose 3. ensure commitment and ownership 4. develop and maintain trust 5. create clear and robust partnership working arrangements 6. monitor, measure and learn Completed individually with post-completion discussion

Potential issues  Each partner weights criteria for judging success differently (Toulemonde et al., 1998)  Sensitivity issues (e.g. Funnell, 2006; own experience leading to modified response options for original VicHealth tool)  Ratings tend to be “generous”  Discussion after completion of the tool is constrained by the ratings given

An alternative approach Through extensive experience in evaluating partnerships CHSD has developed:  A mixed-method design for assessing partnerships in health systems  A generalised approach where focus of strategy/funding is on building capacity through inter-organisational (sector) processes

Procedure  Data collection involves a facilitated group interview ( partnership/governance group)  Questions are asked about the stage of development or progress: How well developed are each of the following in your Partnership Council? e.g. clear goals for PC  A facilitated discussion occurs Two facilitators are present to allow post-completion review and moderation if required. Also enables accurate documentation of comments during discussion.

 Questions (16) reflect domains represented in Nuffield and VicHealth  Includes references also to the underlying logic of the initiative/program In the case of statewide primary health reform, some references to the health system within the catchment  Also collect a brief version of VicHealth Partnerships Analysis Tool (individually completed ) and a single overarching development question (consensus) 12

Response options relate to progress towards achievement of the characteristic:  We won’t be doing this  Haven’t started yet  Currently planning this  Developing this  This is reasonably well developed  This is very well developed/ sustainable  No group consensus reached

 Generality of the questions allows for flexibility and diversity in implementation of strategies, while facilitated discussion gathers specific information reflecting differences in local implementation the “why” is the same, the “where are we up to” is on the same scale, but the “how” is different  Analysis of relationships between measures supports validity and reliability of the approach

15 Partnership Council ratings: “Clear goals for the PC ”

16 “Working so partners are involved in planning / setting priorities for collaborative action”

Benefits of this Approach  Provides quantitative and qualitative data about the characteristics of the partnership  Immediate information is used by partnership to plan future actions – feedback supports strategic planning  Immediate information is used by funder/policy makers to identify areas where additional support and/or modification to strategy may be required  Qualitative information strengthens understanding and interpretation of quant data 17

 Quantitative and qualitative data allows comparison of the characteristics of a partnership over time.  Contributes useful evidence to support development of the partnership and the policy/strategy  Data is about progress towards the strategic goal of increased system capacity through partnership.  Requires experienced and skilled facilitators – for the process of reaching consensus, and with ability to assess quality of evidence provided for ratings

Conclusion  The mixed method design can be applied to different settings to provide robust quantitative indicators with a rich level of interpretive information.  The data obtained through the facilitated group interview contributes useful evidence to support development of the partnership and the policy/strategy.

Associate Professor Virginia Lewis or Geraldine Marsh Centre for Health Systems Development Australian Institute for Primary Care & Ageing La Trobe University Contact:

Partnership Analysis Tool (Brief version – 20 items) 21 1 = ‘Strongly disagree’ 2 = ‘Disagree’ 3 = ‘Not sure’ 4 = ‘Agree’ 5 = ‘Strongly agree’