Monitoring Progress & Evaluating Impact Presented by: Cindy Banyai, Ph.D. Executive Director, efocus Institute IREX Community Solutions Program September 1, 2011 – September 8, 2011

Learning Objective Identify and employ key methods and tools for monitoring progress and evaluating impact IREX CSP, Cindy Banyai, refocusinstitute.com

Project Management Cycle (PMC) 3 Based on Miyoshi, IREX CSP, Cindy Banyai, refocusinstitute.com

Monitoring Progress Monitoring - mid-term and terminal evaluations –Examine implementation process -- assess inputs through outputs –Measure performance – assess outputs to intermediate outcomes 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 4

Evaluating Impact Impact evaluation – measures relationship between intermediate outcomes and end outcomes Sustainability evaluation – measures long- term impacts, periphery effects of program 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 5

USAID on Impact Evaluation Measures the change in a development outcome that is attributable to a defined intervention Based on models of cause and effect Requires a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for observed change Comparisons between beneficiaries that are randomly assigned to either a treatment or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured (USAID 2011) 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 6

DAC 5 Evaluation Criteria Efficiency - A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results. Effectiveness - The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance IREX CSP, Cindy Banyai, refocusinstitute.com 7

DAC 5 Evaluation Criteria (con’t) Impact - Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Sustainability - The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long- term benefits. The resilience to risk of the net benefit flows over time IREX CSP, Cindy Banyai, refocusinstitute.com 8

DAC 5 Evaluation Criteria (con’t) Relevance - The extent to which the objectives of a development intervention are consistent with beneficiaries’ requirements, country needs, global priorities and partners’ and donors’ policies. (DAC 2002) 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 9

10 Relationship between Evaluation Criteria & Logic Model End Outcomes Intermediate Outcomes Outputs Inputs impact effectiveness efficiency Relevance sustainability Activities Measuring performance Examining process Assessment of 5 criteria Measuring performance Examining implementing process Miyoshi 2008

TOOLS FOR MONITORING AND EVALUATION 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 11

Traditional types of data gathering Surveys Tests Interviews Observation Documents 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 12

Participatory types of data gathering Focus groups – cooperative inquiry Training Action research project –Photographs –Video –Plays/storytelling –Metaphor drawing –sculpture IREX CSP, Cindy Banyai, refocusinstitute.com

Quantitative data Need counterfactual - situation a participating subject would have experienced had he or she not been exposed to the program Baseline or pre-program data used to assess impact –Predict outcomes that might result from the program (as in ex ante evaluations) –Make before-and-after comparisons (also called reflexive comparisons) (WB 2010) 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 14

Quantitative impact evaluation Ex-ante impact evaluation - measures intended impacts of future policies/programs/projects, predicts program impacts. –Looks at a potentially targeted area’s current situation –May involve simulations based on assumptions about how the economy works –Based on structural models of the economic environment facing potential participants 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 15

Quantitative impact evaluation Ex-post impact evaluations - measure actual impacts accrued by beneficiaries attributable to policy/program/project –Examines immediate benefits, reflects reality –Sometimes miss the mechanisms underlying the program’s impact on the population, which structural models aim to capture (WB 2010) 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 16

Quantitative methods for ex-post impact evaluation Randomized evaluations Matching methods, specifically propensity score matching (PSM) Double-difference (DD) methods Instrumental variable (IV) methods Regression discontinuity (RD) design and pipeline methods Distributional impacts Structural and other modeling approaches (WB 2010) 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 17

Qualitative data 3 main sources of qualitative data – open- ended interviews, observations, documents (Patton 2002) Emerging sources – photographs, video, artistic expressions (Hesse-Beiber and Leavy 2008) Can be coded, transformed into statistics 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 18

Qualitative research Looks at a phenomenon, situation in-depth and detail The researcher is the instrument – validity relies on her skill, competence and rigor –Internal validity – when qualitative data is corroborated from multiple sources 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 19

Quantitative vs. Qualitative Need both to provide depth and understanding of situation –Numbers balance narrative accounts –Stories and visuals put a human touch on numbers 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 20

Determine approach and tools Use these questions to guide your selection –Who is the information for and who will use the findings? –What kinds of information are needed? –How is the information to be used? For what purposes are the evaluation being done? –When is the information needed? –What resources are available to conduct the evaluation? –Given the answers to the proceeding questions, which methods are appropriate? (Patton 2002) 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 21

Parting thoughts on M&E Collect the data you need. Use the data you collect. Doing so improves your efficiency and the clarity of your evaluations, in addition to freeing up time for other program activities IREX CSP, Cindy Banyai, refocusinstitute.com 22

Thanks for your attention! Please send questions & comments to com

References Banyai, Cindy. (2010). Community capacity and governance – New approaches to development and evaluation. PhD diss., Ritsumeikan Asia Pacific University. Bureau for Policy, Planning, and Learning – United States Agency for International Development [USAID]. (2011.) Evaluation Policy. Washington, D.C.: USAID. Development Assistance Committee [DAC] of the Organization for Economic Cooperation and Development [OECD]. (2002). Glossary of Terms in Evaluation and Results Based Management. Paris: OECD. Hesse-Beiber, Sharlene Nagy and Leavy, Patricia. (2008). Handbook of Emergent Methods. New York: Guilford Press. Japan International Cooperation Agency - Office of Evaluation and Post Project Monitoring, Planning and Evaluation Department (2004). JICA Evaluation Handbook: Practical Methods for Evaluation. Tokyo: Japan International Cooperation Agency. Khandaker, Shahidur R., Koolwal, Gayatri B., Samad, Hussain A. [WB] (2010). Handbook on Impact Evaluation: Quantitative Methods and Practices. Washington, D.C.: World Bank. Miyoshi, Koichi. (2008). What is Evaluation?. In Miyoshi, Koichi (Ed). Hyoka-ron wo Manabu Hito no tameni (For People Learning Evaluation Theory). (pp. 1-16). Tokyo. Sekaishisosha. Patton, Michael Quinn (2002). Qualitative Research & Evaluation Methods. Thousand Oaks: Sage Publications, Inc IREX CSP, Cindy Banyai, refocusinstitute.com 24