1. International engagement: m & e meeting Monitoring & Evaluation: an introduction for practitioners Liz Allen

2. Source: Subsetum; http://www.flickr.com/photos/subsetsum/ monitoring & evaluation can play a vital role in the management & improvement of an activity, organisation or process

3. Monitoring & evaluation: why do it? Theory & practice M & E: challenges Keeping it real: practical & proportionality my talk Monitoring & evaluation: an introduction for practitioners

4. o accountability & validation o management of funding initiatives o strategy & planning o policy & advocacy o learning my funder requires it … I want my programme to work well … I want to do more in the future … I want to tell people about what we do … & show the benefits … I want to find out what works … rationale Monitoring & evaluation: rationale

5. Theory

6. Source: Subsetum; http://www.flickr.com/photos/subsetsum/ o M & E framework o Monitoring o Set up review o Process review o Formative review o Summative review

7. Source: Subsetum; http://www.flickr.com/photos/subsetsum/ o M & E framework o Monitoring o Set up review o Process review o Formative review o Summative review o Inputs/activities o outputs o outcomes o … impact? o (longer term impact)

8. Source: Subsetum; http://www.flickr.com/photos/subsetsum/ o M & E framework o Monitoring o Set up review o Process review o Formative review o Summative review o inputs/activities o outputs o outcomes o … impact? (longer term impact) Things to do & things to ‘measure/track’

9. Theory & formal frameworks o Logical framework o Outcome mapping o Payback framework o Theories of change o Results-based management

10. Monitoring & evaluation: rationale M & E in project cycle Monitoring & review points INPUTS & OUTPUTS ‘ M & E Framework’ design Set-up/process review INPUTS Project start Taking stock OUTPUTS/OUTCOMES Learning & lessons for next time

11. Frameworks good for o keeping outcomes in mind o ‘forces’ definition of: o project objectives o indicators o appropriate data collection method & timings o organisation o simplification Monitoring & evaluation: rationale

12. Frameworks less good o keeping outcomes in mind o ‘forces’ definition of: o project objectives o indicators o appropriate data collection method & timings o organisation o simplification Monitoring & evaluation: rationale o accommodating externalities & context o dealing with the unexpected o understanding process

13. Practice doing m & e

14. (my) basic principles o involvement of stakeholders o definition of objectives/outcomes & associated indicators & method o integrated into project plan from start o properly resourced o practical (usable) & proportionate Monitoring & evaluation: doing m & e

15. (my) basic principles o involvement of stakeholders o definition of objectives/outcomes & associated indicators & method o integrated into project plan from start o properly resourced o practical (usable) & proportionate Monitoring & evaluation: doing m & e

16. an indicator “quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement.” “ something that helps us to understand where we are, where we are going and how far we are from the goal... They are bits of information that summarize the characteristics of systems or highlight what is happening in a system. “ Introduction to LFA: defining indicators

17. SMART & SPICED indicators SMART Specific Measurable Achievable Relevant Timebound SPICED Subjective Participatory Interpreted/communicable Cross-checked Empowering Diverse & disaggregated Introduction to LFA: defining indicators

18. “Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.” attributed to Albert Einstein theoretical physicist, philosopher & author 1879-1955

19. Quantitative o access & participation data (‘penetration’) o attendance & visitor data (‘reach’) o ‘audience’ characteristics (e.g. demographics) o consumer characteristics o scientific output assessment (e.g. publications) o ‘alt metrics’ o amount raised/ follow-on funding o structured opinion & feedback

20. Qualitative o interviews o open questions (in questionnaire) o focus groups o ethnography o participatory research o observation o comment/bulletin boards o visitor books

21. Monitoring & evaluation: rationale & overview Theory & practice M & E: challenges Keeping it real: practical & proportionality my talk Monitoring & evaluation: an introduction for practitioners

22. the elusive ‘impact’ time frame involved serendipity attribution & contribution ‘ripple effects’ counter-factual value of ‘negative’ findings

23. and closer to home … getting agreement on goals, objectives, indicators & method

24. and resourcing m & e properly … adequately, systematically, proportionately…

25. Monitoring & evaluation: rationale & overview Theory & practice M & E: challenges Keeping it real: practical & proportionality my talk Monitoring & evaluation: an introduction for practitioners

26. Understand stakeholder & audience requirements Be prospective: build m & e in from the start Choose right method & tailor Resourcing: ensure access to key data & information & someone to manage this Consider option for trends & benchmarks Keep it real: be proportionate & practical – measures can evolve Be flexible – learning is iterative & should be part of the process Beware over-monitoring & evaluation! Summary Monitoring & evaluation: keeping it real

27. Wellcome Trust’s Indicators of Progress OutcomesKey indicators of progress Discoveries Applications Engagement Research leaders Research environment Influence 1. significant advances in the generation of new knowledge 2. contribute to discoveries with tangible impacts on health 3. contribute to the development of enabling technologies, products and devices 4. uptake of research into policy and practice 5. enhanced level of informed debate in biomedicine 6. significant engagement of key audiences & increased reach 7. develop a cadre of research leaders 8. evidence of significant career progression among those we support 9. key contributions to the creation, development and maintenance of major research resources 10. contributions to the growth of centres of excellence 11. significant impact on science funding & policy developments 12. significant impact on global research priorities and processes

28. Wellcome Trust’s Indicators of Progress OutcomesKey indicators of progress Discoveries Applications Engagement Research leaders Research environment Influence 1. significant advances in the generation of new knowledge 2. contribute to discoveries with tangible impacts on health 3. contribute to the development of enabling technologies, products and devices 4. uptake of research into policy and practice 5. enhanced level of informed debate in biomedicine 6. significant engagement of key audiences & increased reach 7. develop a cadre of research leaders 8. evidence of significant career progression among those we support 9. key contributions to the creation, development and maintenance of major research resources 10. contributions to the growth of centres of excellence 11. significant impact on science funding & policy developments 12. significant impact on global research priorities and processes

33. Wellcome Trust’s Indicators of Progress OutcomesKey indicators of progress Discoveries Applications Engagement Research leaders Research environment Influence 1. significant advances in the generation of new knowledge 2. contribute to discoveries with tangible impacts on health 3. contribute to the development of enabling technologies, products and devices 4. uptake of research into policy and practice 5. enhanced level of informed debate in biomedicine 6. significant engagement of key audiences & increased reach 7. develop a cadre of research leaders 8. evidence of significant career progression among those we support 9. key contributions to the creation, development and maintenance of major research resources 10. contributions to the growth of centres of excellence 11. significant impact on science funding & policy developments 12. significant impact on global research priorities and processes

36. l.allen@wellcome.ac.uk Questions

37. Suggested resources Ahmed, S. & Palermo, A-G (2010) Community Engagement in Research: Frameworks for Education and Peer Review. American Journal of Public Health, 100, 8: 1380-1387 Matthew, B. & Ross, L. (2010) Research Methods: a practical guide for the social sciences. Pearson Education Limited NORAD (2008) Results Management in Norwegian Development Cooperation: a practical guide. http://www.norad.no/en/tools-and-publications/publications/publication?key=109837 Stein, D. & Valters, C. (2012) Understanding ‘Theory of Change’ in international al development : a review of existing knowledge http://www2.lse.ac.uk/internationalDevelopment/research/JSRP/downloads/ToC_Lit_Review.pdf Vogel, I. (2012) Review of the use of ‘Theory of Change’ in international development. For DfID UK Web Center for Social Research methods http://socialresearchmethods.net