1. WELCOME to the webinar “The Professionalization of Evaluation” This Live Webinar will start at 9:30 AM, New York time. All microphones & webcams are disabled and we will only enable microphones during the Q&A portion. Therefore, you will not hear any sound/noise till the beginning of the webinar.

2. The Professionalization of Evaluation Live Webinar 30 th November 2010

3. This series of webinars are based on the book published by UNICEF in partnership with key international institutions  Authors: 40 global evaluation leaders  Partnership: UNICEF, WB, UNDP, WFP, UNIFEM, IDEAS, IOCE, DevInfo

4. Available for free download at www.mymande.org www.mymande.org

5. Webinars on National evaluation capacity development Saraswathi Menon Finbar O’Brien The Role of the United Nations in Fostering National Ownership and Capacities in Evaluation 22 nd June 2010 Michael Quinn Patton Marco Segone Future trends in evaluation. Moving from policies to results by developing national capacities for country- led monitoring and evaluation systems 1 st July 2010 Caroline Heider Craig Russon Evaluating policies and their results The role of policy analysis in over-coming the implementation challenge 16 th September 2010 Belen Sanz Luque Florence E. Etta Human Rights and gender in evaluation 22 nd October 2010 Jean Serge Quesnel Linda G. Morra Imas Robert Picciotto The professionalization of evaluation 30th November 2010 Oscar A. Garcia Angela Bester Joint evaluation of the role and contribution of the UN system in South Africa. Lessons learned 14 th December 2010

6. Michael BambergerInstitutionalizing impact evaluation. A key element in strengthening country-led monitoring and evaluation systems January 2011 Hallie Preskill Alexey Kuzmin Exploring effective strategies for facilitating evaluation capacity development Use of evaluation training in evaluation capacity building February 2011 Michael Quinn PattonUtilization-focused evaluations March 2011 David Fetterman Rita O’Sullivan (TBC) Empowerment evaluation Collaborative evaluation. Creating environments to engage stakeholders in evaluation April 2011 Indran Naidoo The monitoring and evaluation in South Africa. Many purposes, multiple system May 2011 Manuel Fernando Castro Diego Dorado (TBC) Building a results-based management and evaluation system in Colombia June 2011 Velayuthan Sivagnanasothy National monitoring and evaluation system in Sri Lanka. Experiences, good practices, challenges and the way forward July 2011 David Rider Smith Policies, institutions and personalities. Lessons from Uganda’s experience in monitoring and evaluation September 2011 Webinars on National evaluation capacity development

7. Webinars on Emerging Practices on Development Evaluation: SpeakersTitleWhen Zenda Ofir, Former President, African Evaluation Association Shiva Kumar, Independent Consultant from India Developing Countries Perspective on Emerging Practices in Development Evaluation 13 October 2010 Fred Carden, International Development Research Center (IDRC) Good Practices in Evaluating Policy Influence16 November 2010 Jared Raynor, Consultant of TCC groupEvaluating Networks and Partnerships7 December 2010 Peter Morgan, Independent ConsultantEvaluating Capacity DevelopmentJanuary 2011 Charles Lusthaus, Universalia, Associate Professor in the Department of Administration and Policy Studies, McGill University; & partner in Universalia Management Group; Faculty Advisor to the Centre for Educational Leadership, McGill University Evaluating Organizational PerformanceFebruary 2011 Steve Rochlin, Director and AccountAbility's US Representative Evaluating InnovationMarch 2011 Alastair Bradstock, Business Development Director of the International Institute for Environment and Development (IIED) Evaluating Sustainable DevelopmentApril 2011

8. The recording of the previous webinars are available at www.mymande.orgwww.mymande.org 1 2

9. The recording of the previous webinars are available at www.mymande.orgwww.mymande.org

10. The recording of the previous webinars are available at www.mymande.orgwww.mymande.org

11. Professionalization of evaluation Why to professionalize evaluation? What do we mean with “professionalization” of evaluation? How to “professionalize” evaluation, taking into consideration different contexts, risks and opportunities?

12. Keynote Speakers Jean Serge QUESNEL Senior Facilitator, UNSSC, Adjunct Professor, ENAP Quebec and Carleton University, and former Director of Evaluation at UNICEF, IADB and CIDA. Robert PICCIOTTO, Visiting Professor, King’s College, London and former Director General, Evaluation, the World Bank. Linda G. MORRA IMAS Advisor for evaluation capacity development, The World Bank, and Co- Director, International Program for Development Evaluation Training (IPDET).

13. Agenda 9h30 – 9h35Welcome and introduction Marco Segone, Systemic management, UNICEF Evaluation Office 9h35 – 9h50 Linda G. Morra Imas, Advisor for evaluation capacity development, The World Bank, and Co-Director, International Program for Development Evaluation Training (IPDET) 9h50 – 10h05 Robert Picciotto, Visiting Professor, King’s College, London and former Director General, Evaluation, the World Bank 10h05 – 10h20Jean Serge Quesnel, Senior Facilitator, UNSSC, Adjunct Professor, ENAP Québec and Carleton University, and former Director of Evaluation at UNICEF, IADB and CIDA 10h20 – 10h50Questions and Answers 10h50 – 11h00 Wrap-up Marco Segone

14. Questions and Answers 1 3 2

15. “Professionalizing Development Evaluators: Travelling the Road to Results" Linda Morra Imas World Bank Advisor & Consultant Co-Director, IPDET IDEAS Board of Directors 30 November 2010

16. PROFESSIONALISM WHY DO IT? PROFESSIONALISM L. Morra Imas 30 NOV 2010 16

17. L. Morra Imas 30 NOV 2010 17 Who Is Diving In? National & Regional Evaluation Associations Universities Development Organizations Standard-Setting Organizations IDEAS Global Competencies

18. L. Morra Imas 30 NOV 2010 18 Bottom Line: Time has come  Evaluation competencies are here to stay  Issues can be addressed

19. But…how many evaluators work only in a single context? L. Morra Imas 30 NOV 201019

20. A Call for a Global Standard Premise Core competencies exist for evaluation professionals-- across organizations, nations, and regions; having strong evaluation skills is paramount- - add local requirements L. Morra Imas 30 NOV 201020

21. L. Morra Imas 30 NOV 201021 IDEAS Initiative Origin: IDEAS Board Meeting July 2008 Volunteer IDEAS committee formed with great interest- 60 volunteers Membership survey conducted and results presented at 2009 Global Assembly Initiative endorsed and expanded at GA Draft competencies presented Sept. 2010 Membership vote upcoming 2011 Global Assembly

22. INVITATION TO REVIEW AND COMMENT Linda Morra Imas L. Morra Imas 30 NOV 201022 International Development Evaluation Association www.ideas-int.org

23. Finally, consider…  Is recognition of evaluation as a specific discipline and its stature more likely to increase with global competencies? L. Morra Imas 30 NOV 201023

24. Evaluation: a profession? Robert Picciotto 24

25. Evaluation internationalization demands professionalization Canada has taken steps towards evaluators’ accreditation but no similar action is currently on evaluation associations’ agendas within Europe or the developing world Yet evaluation is now called upon to carry out increasingly complex regional, multi-country and development evaluations across borders This calls for global upgrading of evaluation skills and harmonization of competency standards – hallmarks of professionalization 25

26. Expert opinion about evaluation professionalization is divided. For Michael Patton evaluation is already “a demanding and challenging profession” Ernest House concurs: he views evaluation as a “new profession” On the other hand, Lincoln calls evaluation “a fledging profession” Rossi, Lipsey and Freeman disagree: “evaluation is not a profession at all” 26

27. Yet, evaluation has matured as a knowledge occupation… Evaluation is now a discipline in its own right since it has acquired: −a agreed well defined body of expert knowledge −a recognized set of specialized skills −broadly ethical guidelines Evaluation is also a trans-discipline: its concepts and tools support the social sciences The demand for high quality, independent evaluation is growing rapidly around the world 27

28. … and evaluators display a growing sense of self-identity and collegiality.  The “paradigm wars” are over: advocates of qualitative methods agree that quantitative methods complement qualitative methods and vice versa  National and regional evaluation guidelines share: a commitment to systematic inquiry a respect for diverse peoples and cultures an eagerness to achieve results an unwavering public interest orientation 28

29. Unfortunately evaluation independence is under chronic threat Absent the self management regime of a profession, most evaluators do not enjoy adequate autonomy or independence: Decision makers in the public, private and voluntary sectors tend to view evaluation as a management tool Evaluators within organizations are rarely protected from external or internal pressure Evaluation consultants are fee dependent and subject to the whims of commissioners 29

30. What is professionalism? In late Middle English usage a profession is: “a declaration of belief in and obedience to a religion” or “a vow made by a person entering a religious order” In contemporary usage a profession is: “any occupation as a means of earning a living”. “a body of qualified persons in a specific occupation or field” “ a vocation, a calling, esp. one requiring advanced knowledge or training in some branch of learning or science” 30

31. But there is more to it than that: the professions enjoy special privileges… a franchise to operate with autonomy in the public interest significant influence over clients and society specialized knowledge and expert practice honed by experience esoteric or mystifying language that excludes the uninitiated access to specialized education and training programs membership in associations that promulgate and enforce standards plentiful job prospects good remuneration 31

32. … and professionalism would put order in a chaotic market place Much progress has been made, at least at the national level, towards agreed evaluation guidelines and competency standards Yet, quacks and amateurs feel free to use the evaluator title and to market their services This is incompatible with the increasingly demanding quality requirements evinced by decision makers and evaluation commissioners all over the world 32

33. This underlies the rationale for controlled designation of evaluators Evaluators certified by a legitimate body have demonstrated the competency to perform to a specified standard of competency Two approaches to competencies: −Input based models (Germany, UKES and EES) −Outcome based model (CES) A first step consists of accreditation of education courses and training programs 33

34. Reasonable controls over entry would pave the way for professionalization The history of professions demonstrates that prestige and influence would result:  A recognized place in the upper regions of the occupational ladder  High and rising demand for services  Substantial monetary rewards  Public respectability 34

35. But this is not automatic: professionalization must be earned It calls for specific actions by the occupational group so that public trust is gained through:  A credible body of advanced knowledge and proven practices  Ethical safeguards  Verifiable competencies  An effective self management regime  Reliable professional credentials 35

36. The rewards to society would be large…  The professions are critical to the effective working of modern economies  They act as counterweight to the coercive power of the bureaucratic state and the tyrannies of private interests  A franchise to practice in the public interest implies safeguards that distinguish professionals from amateurs and charlatans 36

37. …but the risks are substantial and must be managed Monopolistic practices: over-dependency of vulnerable consumers supply of unnecessary services unwarranted privileges prohibitive fees De-professionalization due to: Misguided professional governance Coercive state intervention Subversion or capture by private interests (e.g. funding with strings) 37

38. Are we a profession? Not yet… We exercise limited influence We are still few in numbers and receive modest monetary rewards Mixed quality of services Scarcity of education and training Slow convergence of standards Limited autonomy Meager progress towards accreditation 38

39. Has the time come? Both evaluation and professionalism seek to shelter their adherents from vested interests (independence) Both value work quality ahead of narrow efficiency considerations (excellence) Both put the general welfare ahead of economic gain (public interest) Equally, professionalism needs evaluation for improved learning and increased accountability 39

40. What specifically is to be done?  Increased supply of high quality education and training  Accelerated harmonization of ethical, quality and competency standards  Increased professional autonomy  Accreditation of training programs  Designation of evaluators 40

41. Do you agree with this diagnostic? If you don’t please explain where I went wrong If you do, would you endorse your evaluation association adopting a comprehensive professionalization agenda? Would the impact of professionalization on your own activities as an evaluator be positive or not? Would you be prepared to volunteer to make professionalization happen? 41

42. Role of Evaluation Associations and Networks in the Professionalization of Evaluation Jean Serge Quesnel 30 November 2010

43. Evaluation Associations/Networks Some 111 Evaluation Associations/Networks presently exists in the world. Most were created since year 2000 This exponential movement is in itself a strong indication of the need for systematizing and professionalizing the discipline of evaluation

44. International Evaluation Groups

45. Regional & National Associations & Networks 2010 International RFÉIDEASOICE Regional Sub- regional National ReLAC IPEN AfrEAAES EES NESE ACE 14 countries 8 countries 41 countries 14 countries 17 countries Sub- national SWEP AEA, CES SQEP, SEA NWEA, WREN CoEsa

46. Evolving roles of Evaluation Networks & Associations The following slides present four generations of Evaluation Networks and Associations The roles that Evaluation Networks and Associations play in support of the professionalization of evaluation vary according to the level of their respective institutional maturity.

47. 1st Generation Aims at creating an awareness Rallies providers and users of evaluation Creates a community of evaluophiles Facilitates professional exchanges Offers informal apprenticeship

48. 2nd Generation Demystifies evaluation Fosters harmonization of concepts et tools Allows an ownership of evaluation approaches and techniques Offers basic training in evaluation Rallies a community of practitioners and users of evaluation

49. 3rd Generation Promotes the good use of the evaluation function via the sharing of real life experiences Fosters the quality of evaluation products Facilitates working groups on new themes offering a challenge to evaluation practice (R&D) Provides methodological guides Sets norms and standards Offers specialized training in evaluation

50. 4th Generation Assumes a normative role Sets a code of conduct for evaluators Identifies professional competencies Offers a competency-based training program Rules and bestows professional designations

51. 51 Professionalization? EVALUATION is a DISTINCT PROFESSION is in the making. Over the years, Evaluation Associations and Networks have developed Evaluation Norms Evaluation Standards Evaluation Policies Evaluation Code of Ethic Initiatives for professionalizing evaluation through professional recognition is occurring in different regions as well greater formalization of evaluation policies by large evaluation commissioners Discussions on competency profiles for evaluators are at the top of the agenda in evaluation seminars and conferences.

52. 52 Why Professionalization? The ASSUMPTION If evaluation is professionalized – then, qualified and RECOGNIZED EVALUATORS will conduct evaluations and guide the evaluation function. THEN the evaluation practice will gain in terms of: - Rigor- Credibility - Quality- Use

53. Levels of professionalization 1.Credentialisation Demonstrates having the minimum of competencies required 2.Certification Successfully has passed the professional examination requirements 3.Licence Legally authorised to practice the profession

54. Elements of Professionalization A Will to Improve the professionalism of Evaluation Norms and Standards Deontology Competency Profiles Benchmark Job Descriptions PolicyPolicy Core Training Quality Assurance Professional Recognition Learning & Improving

55. 55 Basic Competencies Management Skills Personal Attributes Interpersonal Skills Technical And Professional skills Knowledge of the context Competencies As soon as Evaluation Associations have agreed on core competencies, de facto, they will determine who is a “professional evaluator”.

56. 56 Six Core Competencies for UN Evaluation Staff 1. Work Experience 2. Evaluation Design 3. Data Collection And Analysis 4. Reporting 5. Managing the Evaluation Process 6. Ethics Criteria Unique relevance for Evaluation positions System-wide applicability Mandatory compliance at all levels of positions

57. 57 UNEG - Professionalization UNEG is committed to the professionalization and harmonization of the evaluation function within the UN System: www.unevaluation.orgwww.unevaluation.org 2005 UNEG Norms and Standards for Evaluation in the UN System 2005 Self-assessment of UN agencies’ level of compliance with UNEG Norms and Standards. http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=21 http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=22 2005/2006 Peer Reviews of the Evaluation Function in the UN System using the UNEG N&S as a framework. (UNDP, UNICEF, others) 2006 Competencies profile and generic JD for evaluators in the UN System. http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=86 http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=85 http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=84 http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=83 October 2007: Ethical Guidelines for Evaluation http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=102 October 2007: Code of Conduct for Evaluation in the UN System http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=100

58. On going Work Regional Associations are adopting or revising Norms, Standards and Codes of Conduct for Evaluation. IDEAS is inviting you to collaborate in the process of defining global competencies for evaluators. IOCE is undertaking a tally of existing norms and standards for evaluation. This will provide a sound baseline for eventual harmonization. Is this not progress toward professionalization?

59. Questions and Answers 1 3 2

60. Evaluating Networks and Partnerships, 7 December 2010 Jared RAYNOR, TCC Group

61. Joint evaluation of the role & contribution of the UN system in South Africa, 14 December 2010 Oscar A. GARCIA Senior Evaluation Adviser, UNDP Evaluation office. Angela BESTER Director, Deliotte and Touche, South Africa.

62. The recording will be available at www.mymande.org www.mymande.org 1 2