2. Constructing a Monitoring & Evaluation Plan Presented by: Cindy Banyai, Ph.D. Executive Director, efocus Institute IREX Community Solutions Program September 8 – September 15, 2011

3. Learning Objectives Identify key steps in constructing an M&E plan –Determining an evaluation schedule –Building off the logframe –Selecting indicators –Designing evaluation questions –Using evaluation data 2 2011 IREX CSP, Cindy Banyai, refocusinstitute.com

4. SIGNIFICANCE OF EVALUATION 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 3

5. Evaluation is important because… it provides valuable information to stakeholders and administrators reduces waste and inefficiency improves communication between important parts of a policy/program/project 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 4

6. Why evaluate? Accountability –Disclose findings to stakeholders –Use findings to inform resource allocation and other decisions –Metrics matched to meaningful outputs and outcomes under the control of responsible organization –Compare performance to ex ante commitments and targets –Use methods that obtain internal validity of measurement –Ensure credibility of analysis 5 2011 IREX CSP, Cindy Banyai, refocusinstitute.com

7. Why evaluate? Learning –Generate knowledge about magnitude and determinants of project performance –Leading project/program designers and implementer to refine designs and introduce improvements into future efforts –Requires careful selection of evaluation questions to test fundamental assumptions underlying project designs, methods that generate findings that are internally and externally valid –Develop systems to share findings widely and facilitate integration of the evaluation conclusions and recommendations into decision-making (USAID 2011) 6 2011 IREX CSP, Cindy Banyai, refocusinstitute.com

8. Evaluation Schedule Determine when all evaluation activities will occur throughout the PMC –Monitoring activities must occur at regular intervals (yearly, quarterly, bi-annually) depending on timeframe of policy/program/project 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 7

9. Project Management Cycle (PMC) 8 Based on Miyoshi, 2008 2011 IREX CSP, Cindy Banyai, refocusinstitute.com

10. STEPS IN EVALUATION 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 9

11. Steps in evaluation 1. Confirm purpose of evaluation 2. Organize information 3. Design evaluation work plan 4. Collect data 5.Analyze and interpret data 6. Report evaluation results 7. Give feedback 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 10

12. 1 - Confirm purpose of evaluation What is the evaluation objective – DAC 5? What is the purpose of the evaluation results? Who are the potential users of the evaluation results? 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 11

13. DAC 5 Criteria and logic model 12 Miyoshi, 2008 2011 IREX CSP, Cindy Banyai, refocusinstitute.com

14. 2 - Organize information Gather materials on policy/program/project to be evaluated Identify necessary documents, informants 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 13

15. 3 - Design evaluation work plan Select indicators Determine evaluation questions, basis for judgment, data needed Choose appropriate evaluation approach and data collection methods –Consider time, resources available 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 14

16. Indicators and Measurement Indicators –To understand phenomena –Must be direct and valid Measurement –To make indicators measurable or operational –Must be practical and reliable Examples –Increase in knowledge: the average score of a nation-wide test –Better living standards: increase in income 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 15

17. Selecting indicators SMART –Specific –Measureable –Attainable –Relevant –Timely ( AusGUIDElines 2003) 16 2011 IREX CSP, Cindy Banyai, refocusinstitute.com

18. Indicators must also be… Adequate –Represent objectives and avoid having too many indicators for only one purpose 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 17

19. Building off the logframe Use logframe to categorize and demonstrate connections between indicators 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 18

20. Logframe 19 Miyoshi, 2009; based on JICA 2004 2011 IREX CSP, Cindy Banyai, refocusinstitute.com

21. Evaluation Questions Refer to supplement with question prompts based on DAC 5 Tailor to suit specific needs of your evaluation –Follow supplemental worksheet 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 20

22. Types of evaluation questions 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 21 Miyoshi, 2008

23. 22 Types of Evaluation & Evaluation Questions Ex-anteMid-termTerminalEx-post Normative Question: Performance measurement Baseline study●●● Descriptive Question Process evaluation ○●● Cause effect relationship: DAC 5 Criteria Relevance●●● Impact○ △ ○/ ●● Effectiveness○○/ ●●● Efficiency○●● Sustainability○ △ ●● ○ Prediction ● Actual, △ Optional Miyoshi, 2008

24. 4 - Collect data Gather information according to selected methods –Qualitative, quantitative methods Surveys, interviews –Participatory evaluation Focus groups, projects 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 23

25. 5 - Analyze and interpret data Organize collected data Analyze results –Process statistics from quantitative data –Build narratives or code and render statistics of qualitative data Interpret analysis –Answer evaluation questions –Compare analysis to indictors and benchmarks Make recommendations, draw lessons from process 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 24

26. 6 - Report evaluation results Put together analysis and interpretation of evaluation Convey to potential users 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 25

27. Reporting Report summaries to external stakeholders Public reporting Internal documents and data management –Establishing management information systems to use evaluation data 26 2011 IREX CSP, Cindy Banyai, refocusinstitute.com

28. 7 - Give feedback Use results to improve management of ongoing policies/programs/projects Feed results into planning new policies/programs/projects Add reports to systems to improve organizational learning and accountability 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 27

29. Thanks for your attention! Please send questions & comments to photoevaluation@refocusinstitute. com cindy.banyai@gmail.com

30. References AusGUIDElines (2003). The Logical Framework Approach. Canberra: Australian Agency for International Development. Banyai, Cindy. 2010. Community capacity and governance – New approaches to development and evaluation. PhD diss., Ritsumeikan Asia Pacific University. International Fund for Agricultural Development (2010). Rural Poverty Report 2011: New realities, new challenges: new opportunities for tomorrow’s generation. Rome International Fund for Agricultural Development. Japan International Cooperation Agency - Office of Evaluation and Post Project Monitoring, Planning and Evaluation Department (2004). JICA Evaluation Handbook: Practical Methods for Evaluation. Tokyo: Japan International Cooperation Agency. Miyoshi, Koichi. (2008). What is Evaluation?. In Miyoshi, Koichi (Ed). Hyoka-ron wo Manabu Hito no tameni (For People Learning Evaluation Theory). (pp. 1- 16). Tokyo. Sekaishisosha. Patton, Michael Quinn (2002). Qualitative Research & Evaluation Methods. Thousand Oaks: Sage Publications, Inc. Weiss, Carol H. (1998). Evaluation. Upper Saddle River, NJ, USA: Prentice- Hall, Inc. 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 29