1. Independent Evaluation Office UNEG EPE - Strengthening decentralized evaluation functions New York March 11, 2015 Alexandra Chambel and Hicham Daoudi, Evaluation Advisers

2. Background to the evaluation decentralized function

3. Milestones  2009: Evaluation Policy includes decentralized country programme evaluations o Final evaluation at the end of each programme cycle o 100% geographical coverage  2011: Evaluation Branch introduces evaluation quality assessment (EQA) tool to assess final evaluation reports  2013: Revised evaluation policy creates an independent Evaluation Office (EO) reporting directly to the Executive Board o EO is independent from: operational, management and decision-making functions = impartial – objective o Comprehensive biennial budgeted evaluation plan o New role for RO M&E Advisers o Quality assurance system 3

4. Where is the EO within UNFPA? 4 Country Offices Executive Director Office of Audit and Investigation Services (OAIS) Office of the Executive Director (OED) Arab States Regional Office (ASRO) Asia/ Pacific Regional Office (APRO) Eastern Europe/ Central Asia Reg. Office (EECARO) Deputy Executive Director (Programme) Deputy Executive Director (Management) Latin America/ Caribbean Reg. Office (LACRO) Evaluation Office Division for Human Resources (DHR) Division for Management Services (DMS) Programme Division (PD) Technical Division (TD) Information and External Relations Division (IERD) Southern/ Western Asia Sub-Regional Office Pacific Sub-Regional Office Central Asia Sub-Regional Office Caribbean Sub-Regional Office Country Offices Country Offices Country Offices Procurement Services (PSB) Management Information Systems (MIS) UNFPA Executive Board Western/ Central Africa Regional Office (WESARO) Office Ethics Eastern/ Southern Africa Sub-Regional Office (ESARO) Country Offices Country Offices Office of the Security Coordinator (OSC)

5. Challenges  Organizational challenges o Limitations to impartiality due to operational duties of M&E staff o Low capacity of M&E staff o M&E focal points with limited time devoted to the management of evaluations  Other challenges o Limited budget allocated to country programme evaluations o Uneven access to qualified evaluation expertise o Risk of conflict of interest with national consultants Low quality of decentralized country programme evaluations 5

6. Decentralized evaluation support capacity 2013 UNFPA Global East and Southern Africa West and Central Africa Arab States Asia and the Pacific Eastern Europe and Central Asia Latin America and the Caribbean M&E Advisers in Regional Offices 611 (vacant)1111 M&E Officers 46 (38%) 14 (64%) 14 (61%) 5 (33%) 8 (33%) 0 (0%) 5 (24%) M&E Focal Points 768910161716 6

7. Addressing the challenges  Development of methodological guidance o Methodological handbook o Regional training workshops with country office staff o EO regional focal points o Ad hoc advice provided to M&E staff  Biennial budgeted evaluation plan for the evaluation function o Led to increased budgets allocated to evaluation at country level o Rationalization of evaluation planning  Additional quality assurance mechanisms introduced to complement existing quality assessment tool 7

8. PART 1 Designing the Evaluation PART 1 Designing the Evaluation PART 2 Conducting the Evaluation PART 2 Conducting the Evaluation PART 3 Tools & Resources PART 3 Tools & Resources Toolkit Templates Outlines Glossary Elements of Theory http://www.unfpa.org/admin-resource/how-design-and-conduct-country- programme-evaluation-unfpa Handbook on How to Design and Conduct a Country Programme Evaluation at UNFPA 8

9. Methodology Training Workshops YEARPARTICIPANTSLEVEL OF SATISFACTION (%) extremely useful usefulslightly useful AFRICA2012297030 APRO20133562299 EECARO2013193862 LACROMarch 2014288020 ARAB STATES May in 2015 9

10.  The EQA grid is used to express an objective judgment on key aspects of the evaluation report, as well as on the overall quality of the evaluation report  The EQA grid is structured along eight assessment criteria which cover the key quality-related aspects of an evaluation report  The evaluation report is reviewed against each criteria and is awarded an assessment level (Very Good, Good, Poor, Unsatisfactory). The reviewer also provides detailed comments for each criteria 10 The evaluation quality assessment grid

11. 1 Structure & clarity of reporting 2 Completenes s and brevity of executive summary 3 Justification of the design & methodological approach 4 Reliability of data 5 Soundness of the analysis & credibility of the findings 6 Validity of the conclusions 7 Usefulness of the recommen- dations 8 Meeting the needs stated in the ToR 11 The evaluation quality assessment criteria

12.  Country Office Evaluation Managers o Prepare ToR (using guidance provided by EO) o Identify and pre-select qualified evaluators o Ensure quality of evaluation deliverables in consultation with evaluation reference groups o Conduct own quality assessment of draft final evaluation reports  Regional Office M&E Advisers o Assist country offices to prepare ToR and submit final draft to EO o Take part in evaluation reference groups o Assist CO in conducting QA of draft final evaluation reports  Evaluation Office regional focal points o Review and approve ToR for CPE o Pre-qualify consultants for evaluation teams o Assess the quality of final evaluation reports o Provide ad hoc guidance 12 UNFPA Evaluation Quality Assurance processes

13. EVALUATION RESOURCES

14. Reference Documents  UNFPA revised Evaluation policy: http://www.unfpa.org/admin- resource/executive-board-united-nations-development-programme- united-nations-population-fund-1http://www.unfpa.org/admin- resource/executive-board-united-nations-development-programme- united-nations-population-fund-1  Transitional budgeted biennial evaluation plan 2014-2015: http://www.unfpa.org/admin-resource/transitional-biennial- budgeted-evaluation-plan http://www.unfpa.org/admin-resource/transitional-biennial- budgeted-evaluation-plan  Handbook on How to Design and Conduct a Country Programme Evaluation at UNFPA: http://www.unfpa.org/admin-resource/how- design-and-conduct-country-programme-evaluation-unfpahttp://www.unfpa.org/admin-resource/how- design-and-conduct-country-programme-evaluation-unfpa  2012 Evaluation Quality Assessment report: http://www.unfpa.org/admin-resource/evaluation-quality-assessment http://www.unfpa.org/admin-resource/evaluation-quality-assessment  UNFPA evaluation database is publically available at: http://web2.unfpa.org/public/about/oversight/evaluations/ http://web2.unfpa.org/public/about/oversight/evaluations/ 14