Constructing a Monitoring & Evaluation Plan Presented by: Cindy Banyai, Ph.D. Executive Director, efocus Institute IREX Community Solutions Program September 8 – September 15, 2011

Learning Objectives Identify key steps in constructing an M&E plan –Determining an evaluation schedule –Building off the logframe –Selecting indicators –Designing evaluation questions –Using evaluation data IREX CSP, Cindy Banyai, refocusinstitute.com

SIGNIFICANCE OF EVALUATION 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 3

Evaluation is important because… it provides valuable information to stakeholders and administrators reduces waste and inefficiency improves communication between important parts of a policy/program/project 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 4

Why evaluate? Accountability –Disclose findings to stakeholders –Use findings to inform resource allocation and other decisions –Metrics matched to meaningful outputs and outcomes under the control of responsible organization –Compare performance to ex ante commitments and targets –Use methods that obtain internal validity of measurement –Ensure credibility of analysis IREX CSP, Cindy Banyai, refocusinstitute.com

Why evaluate? Learning –Generate knowledge about magnitude and determinants of project performance –Leading project/program designers and implementer to refine designs and introduce improvements into future efforts –Requires careful selection of evaluation questions to test fundamental assumptions underlying project designs, methods that generate findings that are internally and externally valid –Develop systems to share findings widely and facilitate integration of the evaluation conclusions and recommendations into decision-making (USAID 2011) IREX CSP, Cindy Banyai, refocusinstitute.com

Evaluation Schedule Determine when all evaluation activities will occur throughout the PMC –Monitoring activities must occur at regular intervals (yearly, quarterly, bi-annually) depending on timeframe of policy/program/project 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 7

Project Management Cycle (PMC) 8 Based on Miyoshi, IREX CSP, Cindy Banyai, refocusinstitute.com

STEPS IN EVALUATION 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 9

Steps in evaluation 1. Confirm purpose of evaluation 2. Organize information 3. Design evaluation work plan 4. Collect data 5.Analyze and interpret data 6. Report evaluation results 7. Give feedback 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 10

1 - Confirm purpose of evaluation What is the evaluation objective – DAC 5? What is the purpose of the evaluation results? Who are the potential users of the evaluation results? 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 11

DAC 5 Criteria and logic model 12 Miyoshi, IREX CSP, Cindy Banyai, refocusinstitute.com

2 - Organize information Gather materials on policy/program/project to be evaluated Identify necessary documents, informants 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 13

3 - Design evaluation work plan Select indicators Determine evaluation questions, basis for judgment, data needed Choose appropriate evaluation approach and data collection methods –Consider time, resources available 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 14

Indicators and Measurement Indicators –To understand phenomena –Must be direct and valid Measurement –To make indicators measurable or operational –Must be practical and reliable Examples –Increase in knowledge: the average score of a nation-wide test –Better living standards: increase in income 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 15

Selecting indicators SMART –Specific –Measureable –Attainable –Relevant –Timely ( AusGUIDElines 2003) IREX CSP, Cindy Banyai, refocusinstitute.com

Indicators must also be… Adequate –Represent objectives and avoid having too many indicators for only one purpose 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 17

Building off the logframe Use logframe to categorize and demonstrate connections between indicators 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 18

Logframe 19 Miyoshi, 2009; based on JICA IREX CSP, Cindy Banyai, refocusinstitute.com

Evaluation Questions Refer to supplement with question prompts based on DAC 5 Tailor to suit specific needs of your evaluation –Follow supplemental worksheet 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 20

Types of evaluation questions 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 21 Miyoshi, 2008

22 Types of Evaluation & Evaluation Questions Ex-anteMid-termTerminalEx-post Normative Question: Performance measurement Baseline study●●● Descriptive Question Process evaluation ○●● Cause effect relationship: DAC 5 Criteria Relevance●●● Impact○ △ ○/ ●● Effectiveness○○/ ●●● Efficiency○●● Sustainability○ △ ●● ○ Prediction ● Actual, △ Optional Miyoshi, 2008

4 - Collect data Gather information according to selected methods –Qualitative, quantitative methods Surveys, interviews –Participatory evaluation Focus groups, projects 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 23

5 - Analyze and interpret data Organize collected data Analyze results –Process statistics from quantitative data –Build narratives or code and render statistics of qualitative data Interpret analysis –Answer evaluation questions –Compare analysis to indictors and benchmarks Make recommendations, draw lessons from process 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 24

6 - Report evaluation results Put together analysis and interpretation of evaluation Convey to potential users 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 25

Reporting Report summaries to external stakeholders Public reporting Internal documents and data management –Establishing management information systems to use evaluation data IREX CSP, Cindy Banyai, refocusinstitute.com

7 - Give feedback Use results to improve management of ongoing policies/programs/projects Feed results into planning new policies/programs/projects Add reports to systems to improve organizational learning and accountability 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 27

Thanks for your attention! Please send questions & comments to com

References AusGUIDElines (2003). The Logical Framework Approach. Canberra: Australian Agency for International Development. Banyai, Cindy Community capacity and governance – New approaches to development and evaluation. PhD diss., Ritsumeikan Asia Pacific University. International Fund for Agricultural Development (2010). Rural Poverty Report 2011: New realities, new challenges: new opportunities for tomorrow’s generation. Rome International Fund for Agricultural Development. Japan International Cooperation Agency - Office of Evaluation and Post Project Monitoring, Planning and Evaluation Department (2004). JICA Evaluation Handbook: Practical Methods for Evaluation. Tokyo: Japan International Cooperation Agency. Miyoshi, Koichi. (2008). What is Evaluation?. In Miyoshi, Koichi (Ed). Hyoka-ron wo Manabu Hito no tameni (For People Learning Evaluation Theory). (pp ). Tokyo. Sekaishisosha. Patton, Michael Quinn (2002). Qualitative Research & Evaluation Methods. Thousand Oaks: Sage Publications, Inc. Weiss, Carol H. (1998). Evaluation. Upper Saddle River, NJ, USA: Prentice- Hall, Inc IREX CSP, Cindy Banyai, refocusinstitute.com 29