Evaluation Collaboration: Opportunities and Challenges Michael Quinn Patton Utilization-Focused Evaluation 19 May, 2010 Evaluating the Haiti Response: Encouraging Improved System-wide Collaboration A joint OECD/DAC Evalnet-UNEG-ALNAP Roundtable Meeting

VISION and PERCEPTION THOMAS’ THEORM What is perceived as real is real in its consequences. 2

Degrees of working together 1. Networking: sharing information and ideas 2. Cooperating: helping distinct members accomplish their separate individual goals. 3. Coordinating: working separately on shared goals. 4. Collaborating: working together toward a common goal but maintaining separate resources/responsibilitie s 5. Partnering : Shared goals, shared decisions, shared resources within a single entity. 3

4

Please number a sheet of paper:

Question 1. What is current state of working together in humanitarian evaluation (current baseline)? 0. Not at all working together 1. Networking: sharing information and ideas 2. Cooperating: Helping distinct members accomplish their separate individual goals. 3. Coordinating: working separately on shared goals. 4. Collaborating: Working together toward a common goal but maintaining separate resources and responsibilities 5. Partnering: Shared goals, shared decisions, shared resources within a single entity. 6

Question 2. From your perspective, what is the desired level of working together for the Haitian relief evaluation that should be the goal of this meeting? 0. Not at all working together 1. Networking: sharing information and ideas 2. Cooperating: Helping distinct members accomplish their separate individual goals. 3. Coordinating: working separately on shared goals. 4. Collaborating: Working together toward a common goal but maintaining separate resources and responsibilities 5. Partnering: Shared goals, shared decisions, shared resources within a single entity. 7

8 Question 3. At what level of working together are you personally committed on behalf of your organization? 0. Not at all working together 1. Networking: sharing information and ideas 2. Cooperating: Helping distinct members accomplish their separate individual goals. 3. Coordinating: working separately on shared goals. 4. Collaborating: Working together toward a common goal but maintaining separate resources and responsibilities 5. Partnering: Shared goals, shared decisions, shared resources within a single entity.

Evaluation Framework The WORKING TOGETHER continuum as an evaluation framework for evaluating the Haitian relief evaluation. Potentially different degrees of working together for different evaluation purposes. 9

Diverse Types of Evaluation Different uses for different primary intended users 10

Accountability Were funds used appropriately? Each organization accountable to its own stakeholders for basic accountability. What can be done together? Criteria for accountability, for example: * What is appropriate use of relief funds? * What is timely relief? * What is coordinated relief? 12

Other Evaluation Purposes Improvement of relief efforts in the short term Learning and sharing lessons Development of Haiti in the long term Overall judgment about Haitian relief 13

Speaking Truth to Power 14

Aboutthis bookAboutthis book Preview this bookPreview this book Shake Hands with the Devil By Roméo DallaireShake Hands with the Devil By Roméo Dallaire This is a preview. The total pages displayed will be limited. LL Loading... Loading... 15

New Directions for Evaluation Enhancing Disaster and Emergency Preparedness, Response, and Recovery Through Evaluation edited by Liesel Ashley Ritchie and Wayne MacDonald

New Directions for Evaluation 1.Enhancing Disaster and Emergency Preparedness, Response, and Recovery Through Evaluation Liesel Ashley Ritchie and Wayne MacDonald 2. Real-Time Evaluation in Humanitarian Emergencies Emery Brusset, John Cosgrave, Wayne MacDonald 3. The Interagency Health and Nutrition Evaluation Initiative in Humanitarian Crises: Moving From Single-Agency to Joint, Sectorwide Evaluations Olga Bornemisza, André Griekspoor, Nadine Ezard and Egbert Sondorp

4. Save the Children’s Approach to Emergency Evaluation and Learning: Evolution in Policy and Practice Megan Steinke-Chase and Danielle Tranzillo 5. Logic Modeling as a Tool to Prepare to Evaluate Disaster and Emergency Preparedness, Response, and Recovery in Schools Kathy Zantal-Wiener and Thomas J. Horwood

6. Evolution of a Monitoring and Evaluation System in Disaster Recovery: Learning from the Katrina Aid Today National Case Management Consortium Amanda Janis, Kelly M. Stiefel &Celine C. Carbullido 7. Disasters, Crises, and Unique Populations: Suggestions for Survey Research Patric R. Spence, Kenneth A. Lachlan 8. Evaluation of Disaster and Emergency Management: Do No Harm, But Do Better Liesel Ashley Ritchie & Wayne MacDonald

FOCUS and PRIORITIES Utilization-Focused Evaluation lessons 1.Less is more: The dangers and delusions of comprehensiveness. Quality first. 2.Ask important questions: Better to get weaker data on important questions than harder data on unimportant questions. 2. Stay focused on use: actionable findings for intended uses by primary intended users. 20

Understanding and Creating CONTEXT “Social scientists need to recognize that individual behavior is strongly affected by the context in which interactions take place rather than being simply a result of individual differences.” Elinor Ostrom 2009 Nobel Prize in Economics for her analysis of economic governance, especially the commons. 21

The Central Role of Trust “Crucial role of trust among participants as the most efficient mechanism to enhance transactional outcomes…. Empirical studies confirm the important role of trust in overcoming social dilemmas.” Reference: American Economic Review 100 (June 2010):

The Central Role of Trust “Thus, it is not only that individuals adopt norms but also that the structure of the situation generates sufficient information about the likely behavior of others to be trustworthy reciprocators who will bear their share of the costs of overcoming a dilemma.” 23

Elinor Ostrom’s Conclusion “Extensive empirical research leads me to argue that … a core goal of public policy should be to facilitate the development of institutions that bring out the best in humans. We need to ask how diverse polycentric institutions help or hinder the innovativeness, learning, adapting, trustworthiness, levels of cooperation of participants, and the achievement of more effective, equitable, and sustainable outcomes at multiple scales.” 24

Evaluation Theory of Change Process Use – Beyond findings’ use How an evaluation is conducted has an impact beyond the findings that come from the evaluation. For example, evaluation questions carry messages about what matters, what’s important. 25

26

Lessons on effective evaluation collaborations Research on  Shared Measurement Platforms  Comparative Performance Systems, and  Adaptive Learning Systems The research was based on six months of interviews and research by FSG Social Impact Advisors. They examined 20 efforts to develop shared approaches to performance, outcome or impact measurement across multiple organizations. Reference: Kramer, Mark, Marcie Parkhurst, & Lalitha Vaidyanathan (2009). Breakthroughs in Shared Measurement and Social Impact. FSG Social Impact Advisors. 27

Eight factors important to effective evaluation collaborations 1.Strong leadership and substantial funding throughout a multi-year development period. 2.Broad engagement in the design process by many organizations in the field, with clear expectations about confidentiality or transparency 3.Voluntary participation open to all relevant organizations 4.Effective use of web-based technology 28

Eight factors important to effective evaluation collaborations 5. Independence from funders in devising indicators and managing the system 6. Ongoing staffing to provide training, facilitation, and to review the accuracy of all data 7. Testing and continually improving the system through user feedback 8. In more advanced systems, a facilitated process for participants to gather periodically to share results, learn from each other, and coordinate their efforts. 29

Adaptive Learning Systems  The most important lesson learned: the power of breakthroughs to promote a systemic and adaptive approach to solving social problems.  Adaptive Learning Systems offer a new vision that goes beyond capacity building for individual organizations. 30

Adaptive Learning Systems  Breakthroughs offer ways to increase the efficiency, knowledge, and effectiveness of the entire system of interrelated organizations that affect complex interactions and working together.  Adaptive Learning Systems provide a collaborative process for all participating organizations to learn, support each other’s efforts, and improve over time. 31

Adaptive Learning Systems “We believe that shared measurement systems can help move the sector beyond fragmented and disconnected efforts…by creating a new degree of coordination and learning that can magnify the impact of funders and grantees alike.” 32

Adaptive Learning Systems Attentive to complexity concepts and understandings:  COMPLEX DYNAMIC INTERACTIONS  EMERGENCE  ADAPTABILITY  CO-EVOLUTION  DEALING WITH UNCERTAINTY  SYSTEMS CHANGE  INNOVATION For more on complexity and aid, see Ramalingam et al, (2008) Exploring the science of complexity, ODI Working Paper

34 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use

Elinor Ostrom on Complexity To explain the world of interactions and outcomes occurring at multiple levels, we also have to be willing to deal with complexity instead of rejecting it. Some mathematical models are very useful for explaining outcomes in particular settings. We should continue to use simple models where they capture enough of the core underlying structure and incentives that they usefully predict outcomes. When the world we are trying to explain and improve, however, is not well described by a simple model, we must continue to improve our frameworks and theories so as to be able to understand complexity and not simply reject it. 35

Major Capacity Development Purpose Question Is the collaborative system to be designed only for Haitian relief, or is it to become the foundation for future evaluations of disaster relief? What’s the VISION for the system of evaluation collaboration being built? 36

Results of the Baseline Survey THOMAS’ THEORM What is perceived as real is real in its consequences. 37