Future Trends in Evaluation Live Webinar 1st July 2010 DevInfo 1

This series of webinars are based on the book published by UNICEF in partnership with key international institutions Authors: 40 global evaluation leaders Partnership: UNICEF, WB, UNDP, WFP, UNIFEM, IDEAS, IOCE, DevInfo

Available for free download at www.mymande.org

2010 Monthly webinars: 4 Saraswathi Menon Finbar O’Brien The Role of the United Nations in Fostering National Ownership and Capacities in Evaluation 22nd June 2010 Michael Quinn Patton Marco Segone Future trends in evaluation. Moving from policies to results by developing national capacities for country-led monitoring and evaluation systems 1st July 2010 Caroline Heider Craig Russon Evaluating policies and their results The role of policy analysis in over-coming the implementation challenge September 2010 Belen Sanz Human Rights and gender in evaluation October 2010 Jean Quesnel Linda G. Morra Imas The professionalization of evaluation Professionalizing development evaluators. Travelling the road to results November 2010 Oscar A. Garcia Angela Bester Joint evaluation of the role and contribution of the UN system in South Africa. Lessons learned December 2010 4

2011 Monthly webinars: Michael Bamberger Institutionalizing impact evaluation. A key element in strengthening country-led monitoring and evaluation systems January 2011 Hallie Preskill Alexey Kuzmin Exploring effective strategies for facilitating evaluation capacity development Use of evaluation training in evaluation capacity building February 2011 Michael Quinn Patton Utilization-focused evaluations March 2011 David Fetterman Rita O’Sullivan (TBC) Empowerment evaluation Collaborative evaluation. Creating environments to engage stakeholders in evaluation April 2011 Indran Naidoo The monitoring and evaluation in South Africa. Many purposes, multiple system May 2011 Manuel Fernando Castro Diego Dorado (TBC) Building a results-based management and evaluation system in Colombia June 2011 Velayuthan Sivagnanasothy National monitoring and evaluation system in Sri Lanka. Experiences, good practices, challenges and the way forward July 2011 David Rider Smith Policies, institutions and personalities. Lessons from Uganda’s experience in monitoring and evaluation September 2011

The recording of the previous webinars are available at www. mymande The recording of the previous webinars are available at www.mymande.org 1 2

The recording of the previous webinars are available at www. mymande The recording of the previous webinars are available at www.mymande.org

The recording of the previous webinars are available at www. mymande The recording of the previous webinars are available at www.mymande.org

Agenda 9h30 – 9h35 Welcome and introduction 9h35 – 9h50 Michael Quinn Patton, Founder and Director, Utilization- focused evaluation, and former President of the American Evaluation Association 9h50 – 10h05 Marco Segone, Systemic management, UNICEF Evaluation Office 10h05 – 10h25 Questions and Answers Moderator: Abigail Taylor, Knowledge Management Specialist, UNICEF Evaluation Office 10h25 – 10h30 Wrap-up

Questions and Answers 1 3 2 10

Keynote speakers QUINN PATTON, Michael former President of the American Evaluation Association and author of six evaluation books including Utilization-Focused Evaluation; Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use; and Qualitative Research and Evaluation Methods. 11

Trends in EVALUATION UNICEF Webinar Michael Quinn Patton Michael Q. Patton, UNICEF, July 1, 2010

Evolving Understandings I keep changing what I said. Any person who is intellectually alive changes his ideas. If anyone at a university is teaching the same thing they were teaching five years ago, either the field is dead, or they haven’t been thinking. Noam Chomsky “The Professor Provaocateur,” The New York Times Magazine, Nov. 2, 2003: 13. Michael Q. Patton, UNICEF, July 1, 2010

TRENDS 1. Globalization of the Profession IOCE IDEAS Challenge: Non-professionals doing evaluation Diversity of evaluation Michael Q. Patton, UNICEF, July 1, 2010

Evaluation Standards and Culture Utility Feasibility Propriety Accuracy NDE: International Perspectives on Evaluation Standards, No. 104, 2004. Michael Q. Patton, UNICEF, July 1, 2010

2. Evaluation as a Transdiscipline Evaluative Thinking IDRC 1. Globalization of the Profession 2. Evaluation as a Transdiscipline Evaluative Thinking IDRC Michael Q. Patton, UNICEF, July 1, 2010

1. Globalization of the Profession 2. Evaluation as a Transdiscipline 3. Increased political interest in accountability, performance indicators, and transparency Michael Q. Patton, UNICEF, July 1, 2010

SPEAKING TRUTH TO POWER

This is a preview. The total pages displayed will be limited. L   Loading... Loading...   About this book Preview this book Shake Hands with the Devil  By Roméo Dallaire This is a preview. The total pages displayed will be limited. L  

Trends Essential skills – beyond just methods Opportunities: IPDET 4. Growing emphasis on evaluation capacity-building and skill development. Essential skills – beyond just methods Opportunities: IPDET The Evaluators’ Institute Professional associations Michael Q. Patton, UNICEF, July 1, 2010

Trends 5. Continuing debate about what constitutes methodological rigour Michael Q. Patton, UNICEF, July 1, 2010

GOLD STANDARD DEBATE: METHODOLOGICAL APPROPRIATENESS not Methodological orthodoxy or rigidity

Mixed Methods Matching methods to evaluation questions, purposes, and resources; Contingency-based evaluation Michael Q. Patton, UNICEF, July 1, 2010

Trends 6. Systems thinking and complexity concepts as frameworks for evaluation Michael Q. Patton, UNICEF, July 1, 2010

“Using Systems Concepts in Evaluation” edited by AEA monograph: “Using Systems Concepts in Evaluation” edited by Bob Williams and Iraq Iman Michael Q. Patton, UNICEF, July 1, 2010

Seeing Through A Complexity Lens Michael Q. Patton, UNICEF, July 1, 2010

Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use Michael Quinn Patton 2010, Guilford Press

Complexity Concepts Emergence Openness Dynamic High uncertainty Nonlinear: Small actions, large impacts Co-evolution Adaptation Unanticipated consequences Michael Q. Patton, UNICEF, July 1, 2010

Unanticipated developments as an expected aspect of innovation and a focus and challenge for evaluation. Michael Q. Patton, UNICEF, July 1, 2010

Beyond just formative and summative to Developmental Evaluation Michael Q. Patton, UNICEF, July 1, 2010

What is being developed? Development not just Improvement. Core questions: What is developing? What is being developed? Michael Q. Patton, UNICEF, July 1, 2010

Challenge Matching the evaluation process and design to the nature of the situation: Contingency-based Evaluation Michael Q. Patton, UNICEF, July 1, 2010

Contingency-based Evaluation Situational analysis & responsiveness Context sensitivity Clarify and focus on intended users: stakeholder analysis Clarify and focus on intended uses Methodological appropriateness Criteria for evaluating the evaluation: credibility, meaningfulness, utilty Michael Q. Patton, UNICEF, July 1, 2010

1. Globalization of the profession Summary: Six Trends 1. Globalization of the profession 2. Evaluation as a transdiscipline. 3. Increased political importance of M & E 4. Growing emphasis on capacity-building and essential skills 5. Debate about methods: Gold standard debate 6. Using systems thinking and complexity concepts Michael Q. Patton, UNICEF, July 1, 2010

January 2009 References Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. Michael Quinn Patton, 2010, Guilford Press. http://www.guilford.com/cgi-bin/cartscript.cgi?page=pr/patton.htm&dir=research/res_eval&cart_id=435936.20323 Utilization-Focused Evaluation, 4th ed., Michael Quinn Patton, Sage Publications, 2008. Michael Quinn Patton UNICEF WebinarJuly 1, 2010

References “Future Trends in Evaluation.” Pp. 44-57 in From Policies to Results: Developing capacities for country M & E systems, Marco Segone (ed.) UNICEF, 2010. Michael Q. Patton, UNICEF, July 1, 2010

Keynote speakers SEGONE, Marco is responsible for the decentralized evaluation portfolio at the UNICEF Evaluation Office, serves as a Senior Advisor to the IOCE Board and was IOCE Vice-President. 40

National evaluation capacity development Key elements for a conceptual framework Marco Segone, Systemic Strengthening, UNICEF Evaluation Office

WHY

In line with new international development paradigm… National ownership and capacity development: the key ingredients to development Managing for results Mutual accounta bility Harmoni zation Alignment Ownership Paris Declaration on Aid Effectiveness

… the UN General Assembly … national Governments have the primary responsibility for coordinating external assistance and evaluating its contribution to national priorities mandated the United Nations system to promote national ownership and capacity development, and to make system-wide progress in harmonizing evaluation practices UNICEF

… and UN Agencies Boards request. 2009 UNICEF ExBoard: emphasizes that programme countries should have greater ownership and leadership in the evaluation of all forms of assistance underlines the importance of increasing the participation of national counterparts and strengthening national capacity in evaluation encourages UNICEF to use national evaluation systems, where available, to continue to incorporate capacity-building mechanisms into programme design and implementation and ensure that evaluations are responsive to national demand UNICEF

HOW

A Systemic approach to Capacity Development. Individual Level (skills, knowledge, experience) UNICEF

Individual Level Demand side Capacity to strategically plan evaluations, and to identify the key evaluation questions Capacity to manage evaluation for independence and credibility Capacity to use evaluation Supply side (capacity to conduct an evaluation): Behavioural independence Independence of mind & integrity Knowledge and respect of evaluation standards Agreed evaluation processes & products Professional competences Formal education (Masters) Specialized training Professional Conferences and meeting On the job training (country-led evaluations) Community of Practices and networking

A Systemic approach to Capacity Development. Institutional Level (policies, procedures, frameworks) Individual Level (skills, knowledge, experience) UNICEF

Institutional Level Evaluation culture Set of values and attitudes supporting evaluative (critical) thinking within an organization Institutional commitment to learning from evaluation, support evidence-based policy debate and demand for accountability. Individual more self-directed learners and use information to act; take higher risks but also develop a greater sense of personal accountability and responsibility; consult, coach, and support each other more. Protective culture (Remove repercussions on careers) Understanding of the foundations and principles of Monitoring and Evaluation Institutionalizing independence, credibility and utility Evaluation policies Work programme and budget Independence & adequacy of budget Conduct of evaluations Institutional endorsement of standards In-built Quality Assurance systems

A Systemic approach to Capacity Development. Institutional Level (policies, procedures, frameworks) Enabling Environment (policies, legislation, power relations, social norms) Individual Level (skills, knowledge, experience) UNICEF

Enabling Environment Public administration committed to manage for results and accountability Transparency Results-based public budgeting Evidence-based policy making Strong civil society rights holders able to demand for and monitor quality of public services Strong national evaluation association foster indigenous demand for M&E, and strengthen indigenous supply

We need a mind shift to do things differently... We should always aim at strengthening National ownership and leadership, and not undermining it: Selecting topics of mutual interest Implement evaluations jointly with Governments, as a first step towards country-led evaluation Hiring local experts to the maximum extent possible Not assuming there are weak evaluation capacities, even if none is immediately apparent Co-ordinating with other UN agencies and key international stakeholders Invest in the country-led process, even if it may require additional efforts and less control

Questions and Answers 1 3 2 54

The discussion goes on at www.mymande.org 1 2

The discussion goes on at www.mymande.org

The discussion goes on at www.mymande.org

Next webinar, September, same time Caroline Heider Director of the Office of Evaluation of the World Food Programme (WFP) and Vice Chair of the UN Evaluation Group Craig Russon Senior Evaluation Officer, International Labour Organization (ILO) 58