A framework for evaluating partnerships HOW A COMPREHENSIVE APPROACH CAN STRENGTHEN UNDERSTANDING AND APPLICATION

Overview Partnerships as delivery of policy programs have long history Development of effective frameworks and tools been lagging Evaluators face technical challenges Within international development also face conceptual challenge Case example of a Partnership Evaluation Framework Use of framework can strengthen understanding

A brief history and definitional issues Not a new concept No single, common definition – confusion (Carnwell and Carson, 2004; Dowling et al, 2004; Caplan et al, 2007) Different drivers and underlying theories Relatively recent emergence in international development (Piciotto, 2007)

Current evaluation approaches Few studies to assess performance or effectiveness (Brinkerhoff, 2002; Jorgensen, 2006; Serafin, 2008) Focus primarily on outputs (Caplan et al, 2007) No agreement about elements to be assessed – different elements emphasised by different authors

Partnership assessment tools Include some of the elements emphasised in literature Primarily for developmental purposes Beneficial (Sunderland et al, 2009) Do NOT perform well across all elements (Halliday et al, 2004)

Partnership evaluation frameworks Emerged in response to perceived shortfalls Range of domains + sub dimensions More comprehensive: developmental and evaluative Outcomes Questions, evidence criteria Mixed methods

Evaluation challenges Technical - need to adjust methods (Conlin & Stittat, 2006; Jobin, 2008) Conceptually (international development) Uptake in short space of time Broad scale and scope, and more complex Promotion of sovereignty (Picciotto, 2007) Still grappling with purpose Involving partners Practical participatory evaluation Transformative participatory evaluation (Cousins and Whitmore, 1998)

Case example Review - Australia and Indonesia Partnership Maternal and Neonatal Health New modality Review – 2010; 2 years into partnership Effectiveness and clarification of partnership

Case example Partners still grappling with concept No shared view I wanted a tool or framework to further develop the concept and increase understanding Partnership Assessment Tool (Hardy, 2003) promising BUT… Frameworks more comprehensive BUT…

Case example Developed a framework based on mix of: Elements highlighted by the partners Salient and commonly used features from literature Drew heavily on Brinkerhoff (2002) Plus Caplan et al (2007), Jobin (2008),

Partnership Evaluation Framework Hind, 2010 PREREQUISITES STRUCTURE PROCESS OUTCOMES Enabling environment Political attitudes Preoccupations and priorities Commitment Expectations Stability of environment Formal partnership dimensions Legitimacy Resources Responsibilities Decision-making Contribution Compliance Partnership practice Nature of Interaction Capacity development Mutuality and equality Effectiveness Achieving desired results – including Influence on: Administrative capacity Service delivery capacity Community capacity Responsiveness Partnership performance Relationships Performance of representatives Conflict resolution Creating and strengthening success factors Drivers: Organisational & individual Incentives Obligations Efficiency Make good use of partner resources Benefits Costs Added value Informal partnership dimensions Social capital Reputation Trust Institutional elements Common objectives – public policy purpose Shared governance Written agreement

Areas of assessment Intended to develop assessment areas participatory Accepted a pragmatic approach Practical Contextually responsive Consequential (Datta, 1997)

Areas for assessment Prerequisites Enabling environment Extent to which: partnership is supported politically; has the backing of key stakeholders; there are real opportunities for collaboration; partners tolerate and encourage power-sharing partnership meets current priorities partners are able to focus on the partnership given other work/priorities/demands partners provide leadership; there are champions; partners are willing to adapt; partners are future oriented there are clear and reasonable expectations; expectations are similar between partners environment is stable and conducive to collaborative working; key individuals are in place to facilitate partnership; external context is understood Drivers: organisational and individual purpose, mandate, vision of partner organisations supports/facilitates partnership partners have inbuilt organisational culture, processes, systems, demands to facilitate partnership partners have legal and/or organisational requirements to partner decision making processes are clear and sufficient to facilitate partnership partners have processes in place to address lapses in commitment Institutional elements there are common, shared goals linked to relevant public policies; partners articulate what they want to achieve from the partnership formal and informal governance structures are in place and work; processes and structures have been developed collaboratively the partnership is articulated in writing yet flexible to adapt as required Areas for assessment

Application of framework Chose all domains and dimensions Mixed methods Document review Semi-structured interviews Observational visits Informal interaction with beneficiaries Workshops Presentations by partners Facilitated group discussion Partner surveys using 5-point ranking scale Narrative writing

Effectiveness of the Framework Sufficient breadth to: address issues re: time, resources, data availability answer all salient evaluation questions make judgement using most not all dimensions Could be used flexibly within the local context Drew on literature and salient features identified by partners

Did it help strengthen the understanding of partnership? Workshops with partners Opportunity to discuss and reflect on partnership New insights + promoted broader thinking Some further development in understanding Discussions during report drafting Opportunity to examine the sub dimensions and their relevance advanced thinking of concept BUT… More in-depth discussions were limited so potential constrained

Conclusion Evaluating partnerships is essential Still in formative stage Concept poses a challenge therefore is a challenge for evaluators The framework was robust Need to adapt for local situation The partnership evaluation framework + techniques can help progress concept

Partnership assessment tools The Partnership Analysis Tool: VicHealth: http://www.vichealth.vic.gov.au/en/Publications/VicHealth-General-Publications/Partnerships-Analysis-Tool.aspx The Partnership Self-Assessment Tool: Center for the Advancement of Collaborative Strategies in Health; www.partnershiptool.net The Partnership Assessment Tool: Hardy et el, 2003, http://www.communities.gov.uk/documents/localgovernment/pdf/135112.pdf

Partnership evaluation frameworks Brinkerhoff (2002) – domains linked through flow chart Caplan et al (2007) – the ‘drivers’: external, organisational + individual contexts Jobin (2008) –transaction cost theory; allows consideration of counterfactual; social capital