1. IDEAS Delhi, Workshop – DAC April 2005 1 Independent Evaluation « Quality Assessment of SDC’s External Evaluation Reports » Anne Bichsel Evaluation + Controlling Division

2. IDEAS Delhi, Workshop – DAC April 2005 2 Content 1.Objectives of the Quality Assessment 2.Procedures and Method 3.Findings 4.Recommendations

3. IDEAS Delhi, Workshop – DAC April 2005 3 1. Objectives ot the Quality Assessment To assess the quality of SDC‘s external evaluation reports To contribute to improving future performance and achieving a higher evaluation quality of SDC‘s external evaluations

4. IDEAS Delhi, Workshop – DAC April 2005 4 2. Procedures and Method (I) Step I: Random Sample of 12 external evaluation reports Step 2: devise a list of evaluation standards on the basis of - „DAC Minimum Sufficient Evaluation Standards“ - SEVAL Standards - Key questions in Approach Paper

5. IDEAS Delhi, Workshop – DAC April 2005 5 2. Procedures and Method (II) List of evaluation standards with 4 categories Utility (9 standards): readable, accessible, timely evaluations with a good summary Feasibility (3 standards): realistic well thought-out evaluations Propriety (4 standards): ethical aspect of evaluations Accuracy (7 standards): proper scientific methods and procedures

6. IDEAS Delhi, Workshop – DAC April 2005 6 2. Procedures and Method (III) Step 3: review documents, conduct interviews with desk managers and evaluators Step 4: work through all the standards for the sample of evaluations. Result: 12 fact sheets Step 5: analysis and comparison Step 6: conclusions and recommendations

7. IDEAS Delhi, Workshop – DAC April 2005 7 3. Findings (I) Strengths of external evaluation reports Identification and participation of stakeholders Timely reporting Cost effectiveness Complete and balanced assessment

8. IDEAS Delhi, Workshop – DAC April 2005 8 3. Findings (II) Weaknesses of external evaluations reports Selection procedures of evaluation team Formal written agreement Comprehensive and clear reporting Description of evaluation purpose and objectives Description of evaluation procedures and methods Making findings available

9. IDEAS Delhi, Workshop – DAC April 2005 9 4. Recommendations (I)  Draft good, realistic and comprehensive TORs Clear purpose Clear, focused, concise and understandable objectives with three to four questions Agreement on TORs among SDC divisions Realistic expectations in terms of resources and timeframe Beware of questions at the impact level

10. IDEAS Delhi, Workshop – DAC April 2005 10 4. Recommendations (II)  Ensure more competitive and open selection procedures Aim at balance in desired knowhow and independence of evaluators Place more weight on evaluation knowhow (methods) More open and transparent selection procedures More competition Careful team building in matching personalities, skills, division of labor

11. IDEAS Delhi, Workshop – DAC April 2005 11 4. Recommendations (III)  Improve the conditions for the utilisation of external evaluations Accompany evaluations closely for maximum learning potential Continuity on the part of desk managers Reports need to be more accessible to a wider audience Enhance the quality of evaluation reports Make results of external evaluations more widely available

12. IDEAS Delhi, Workshop – DAC April 2005 12 4. Recommendations (IV)  Enhance the conditions for high quality external evaluations Evaluations are a challenging business, hence: Provide support and training for desk managers Training in evaluation Support in drafting TORs and commissioning evaluations Support in obtaining high quality reports

13. IDEAS Delhi, Workshop – DAC April 2005 13 In sum: less is more !

14. IDEAS Delhi, Workshop – DAC April 2005 14 Follow-up by the Core Learning Group: Evaluation & Controlling Net H-BereichO-BereichE-BereichM-BereichF-BereichA-Bereich Christoph Jakob Kuno Schläfli Peter Meier Regula Bäbler Markus Glatz Ivo Angehrn Alexandre Kobel Direktor DEZA Stellv. Direktor Gerhard Siegfried Samuel Wälty Anne Bichsel Sekretariat (je 50%) Regula Herlan Christa Rohner jeweilige Bereichsleitung

15. IDEAS Delhi, Workshop – DAC April 2005 15 Steps of an Independent Evaluation COSTRA Comité Stratégique TORs&contracts Implementation Draft Report Debriefing Final Report Approach Paper Core Learning Partners (CLP) Approach Paper Core Learning Partners (CLP) Agreement at Completion Point Dissemination to Broad Learning Partners (BLP) Agreement at Completion Point Dissemination to Broad Learning Partners (BLP) Senior Management Standpoint discussed in COSTRA I. II. III. IV. COSTRA Decision 0.

16. IDEAS Delhi, Workshop – DAC April 2005 16 Measures to improve quality Communicated evaluation results / recommendations in all departments Clarified and communicated the palette of instruments Developing toolkit for commissioners of evaluations  Intraweb: Description of Process / examples  Principles to ensure successful Use of Evaluations Improving training courses: curriculum and availability (1 day training Ext. Evals) Backstopping and Quality Control by the E&C Net

17. IDEAS Delhi, Workshop – DAC April 2005 17 Independent Evaluation External Evaluation External Review Self-Evaluation Expert Opinion Palette of Instruments

18. IDEAS Delhi, Workshop – DAC April 2005 18 Independent Evaluations since 2001 2 – 5 annually outside line management ( E&C Division) - issues of interest across departments - selected country programs aim for the same DAC Executive Registration External Evaluations 5 – 10 annually within - to generate knowledge within the department - of interest across divisions standards Summary in data management External Reviews according to operational needs line management - ongoing operations - in the context of program cycle management system (DMS)

19. IDEAS Delhi, Workshop – DAC April 2005 19

20. IDEAS Delhi, Workshop – DAC April 2005 20