Monitoring and Evaluation Presentation by Kanu Negi Office of Development Effectiveness (ODE), DFAT and David Goodwins Coffey International Development

I.Key annual DFAT performance reporting—assessments at multiple levels: Investment/project—Aid Quality Checks Country/regional program—Aid Program Performance Reports Overall Australian aid program—the Performance of Australian Aid II. Role of ODE in quality assuring aid performance assessments Aid program evaluation Independent oversight the performance reporting on the Australian aid program III.M&E expectations in DFAT’s aid program: Investment Design Guidelines DFAT Monitoring and Evaluation Standards IV.Guidance and tips on program evaluation and program logic, including program logic used in ODE’s trade facilitation evaluation V.Guidance on dealing with attributing investments to results Presentation structure

Program Logic Major steps in Program Evaluation Program Logic Program Logic

Trade Facilitation Program Logic (used in ODE evaluation)

To what extent did the investment lead to changes (positive or negative). Used in tandem with contribution which looks at the confluence of many factors influencing a change (what contribution did the investment make to that change?) Two approaches are often used: 1.The intervention is unpacked and quantitative approaches (e.g. statistical inference) are used to look at causes for change for specific components 2.Mixed method case study approaches are used such as contribution analysis. Dealing with the Attribution problem

Why is attribution often difficult?

What effects (outcomes) are caused by DFAT investments? › Quasi – experimental randomised control trials – scientific method – expensive, works with “closed systems”. Uses cases that have no investment compared to those that have. › Case studies and contribution analysis – uses case studies and research to test your program theory (theory of change). Does you program theory hold true in the real world? › Can the results of your analysis be generalised for other situations? › What works under what situations? Cause and effect

1.Develop a detailed program logic model clearly articulating assumptions and external influencing factors. Map out it out as a systems diagram showing where the investments will contribute. What are the pre- conditions? 2.Assess the existing evidence on results – helped by making sure you have a baseline. Break down the intervention into measurable components or cases. Use quantitative and qualitative data (mixed methods) and identify strong and weak evidence. 3.Assess alternative explanations for results – other factors that could have produced the results 4.Assemble the possible reasons for the performance of the investment – describe how the contribution made could be influencing outcomes. Check assumptions. 5.Seek additional evidence to fill in any gaps – e.g. interviews – qualitative studies 6.Revise and strengthen the performance case study Contribution Analysis

1.Study the most successful cases and the least successful cases of an intervention 2.Use these extremes to work out what works and what doesn’t and why. Document the context, mechanism and outcomes for each case. 3.Use the lessons from the most successful and least successful cases to strengthen the investment to improve the overall results. Case study approach to support contribution analysis – Success case method