International engagement: m & e meeting Monitoring & Evaluation: an introduction for practitioners Liz Allen

Source: Subsetum; monitoring & evaluation can play a vital role in the management & improvement of an activity, organisation or process

Monitoring & evaluation: why do it? Theory & practice M & E: challenges Keeping it real: practical & proportionality my talk Monitoring & evaluation: an introduction for practitioners

o accountability & validation o management of funding initiatives o strategy & planning o policy & advocacy o learning my funder requires it … I want my programme to work well … I want to do more in the future … I want to tell people about what we do … & show the benefits … I want to find out what works … rationale Monitoring & evaluation: rationale

Theory

Source: Subsetum; o M & E framework o Monitoring o Set up review o Process review o Formative review o Summative review

Source: Subsetum; o M & E framework o Monitoring o Set up review o Process review o Formative review o Summative review o Inputs/activities o outputs o outcomes o … impact? o (longer term impact)

Source: Subsetum; o M & E framework o Monitoring o Set up review o Process review o Formative review o Summative review o inputs/activities o outputs o outcomes o … impact? (longer term impact) Things to do & things to ‘measure/track’

Theory & formal frameworks o Logical framework o Outcome mapping o Payback framework o Theories of change o Results-based management

Monitoring & evaluation: rationale M & E in project cycle Monitoring & review points INPUTS & OUTPUTS ‘ M & E Framework’ design Set-up/process review INPUTS Project start Taking stock OUTPUTS/OUTCOMES Learning & lessons for next time

Frameworks good for o keeping outcomes in mind o ‘forces’ definition of: o project objectives o indicators o appropriate data collection method & timings o organisation o simplification Monitoring & evaluation: rationale

Frameworks less good o keeping outcomes in mind o ‘forces’ definition of: o project objectives o indicators o appropriate data collection method & timings o organisation o simplification Monitoring & evaluation: rationale o accommodating externalities & context o dealing with the unexpected o understanding process

Practice doing m & e

(my) basic principles o involvement of stakeholders o definition of objectives/outcomes & associated indicators & method o integrated into project plan from start o properly resourced o practical (usable) & proportionate Monitoring & evaluation: doing m & e

(my) basic principles o involvement of stakeholders o definition of objectives/outcomes & associated indicators & method o integrated into project plan from start o properly resourced o practical (usable) & proportionate Monitoring & evaluation: doing m & e

an indicator “quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement.” “ something that helps us to understand where we are, where we are going and how far we are from the goal... They are bits of information that summarize the characteristics of systems or highlight what is happening in a system. “ Introduction to LFA: defining indicators

SMART & SPICED indicators SMART Specific Measurable Achievable Relevant Timebound SPICED Subjective Participatory Interpreted/communicable Cross-checked Empowering Diverse & disaggregated Introduction to LFA: defining indicators

“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.” attributed to Albert Einstein theoretical physicist, philosopher & author

Quantitative o access & participation data (‘penetration’) o attendance & visitor data (‘reach’) o ‘audience’ characteristics (e.g. demographics) o consumer characteristics o scientific output assessment (e.g. publications) o ‘alt metrics’ o amount raised/ follow-on funding o structured opinion & feedback

Qualitative o interviews o open questions (in questionnaire) o focus groups o ethnography o participatory research o observation o comment/bulletin boards o visitor books

Monitoring & evaluation: rationale & overview Theory & practice M & E: challenges Keeping it real: practical & proportionality my talk Monitoring & evaluation: an introduction for practitioners

the elusive ‘impact’ time frame involved serendipity attribution & contribution ‘ripple effects’ counter-factual value of ‘negative’ findings

and closer to home … getting agreement on goals, objectives, indicators & method

and resourcing m & e properly … adequately, systematically, proportionately…

Monitoring & evaluation: rationale & overview Theory & practice M & E: challenges Keeping it real: practical & proportionality my talk Monitoring & evaluation: an introduction for practitioners

Understand stakeholder & audience requirements Be prospective: build m & e in from the start Choose right method & tailor Resourcing: ensure access to key data & information & someone to manage this Consider option for trends & benchmarks Keep it real: be proportionate & practical – measures can evolve Be flexible – learning is iterative & should be part of the process Beware over-monitoring & evaluation! Summary Monitoring & evaluation: keeping it real

Wellcome Trust’s Indicators of Progress OutcomesKey indicators of progress Discoveries Applications Engagement Research leaders Research environment Influence 1. significant advances in the generation of new knowledge 2. contribute to discoveries with tangible impacts on health 3. contribute to the development of enabling technologies, products and devices 4. uptake of research into policy and practice 5. enhanced level of informed debate in biomedicine 6. significant engagement of key audiences & increased reach 7. develop a cadre of research leaders 8. evidence of significant career progression among those we support 9. key contributions to the creation, development and maintenance of major research resources 10. contributions to the growth of centres of excellence 11. significant impact on science funding & policy developments 12. significant impact on global research priorities and processes

Wellcome Trust’s Indicators of Progress OutcomesKey indicators of progress Discoveries Applications Engagement Research leaders Research environment Influence 1. significant advances in the generation of new knowledge 2. contribute to discoveries with tangible impacts on health 3. contribute to the development of enabling technologies, products and devices 4. uptake of research into policy and practice 5. enhanced level of informed debate in biomedicine 6. significant engagement of key audiences & increased reach 7. develop a cadre of research leaders 8. evidence of significant career progression among those we support 9. key contributions to the creation, development and maintenance of major research resources 10. contributions to the growth of centres of excellence 11. significant impact on science funding & policy developments 12. significant impact on global research priorities and processes

Wellcome Trust’s Indicators of Progress OutcomesKey indicators of progress Discoveries Applications Engagement Research leaders Research environment Influence 1. significant advances in the generation of new knowledge 2. contribute to discoveries with tangible impacts on health 3. contribute to the development of enabling technologies, products and devices 4. uptake of research into policy and practice 5. enhanced level of informed debate in biomedicine 6. significant engagement of key audiences & increased reach 7. develop a cadre of research leaders 8. evidence of significant career progression among those we support 9. key contributions to the creation, development and maintenance of major research resources 10. contributions to the growth of centres of excellence 11. significant impact on science funding & policy developments 12. significant impact on global research priorities and processes

Questions

Suggested resources Ahmed, S. & Palermo, A-G (2010) Community Engagement in Research: Frameworks for Education and Peer Review. American Journal of Public Health, 100, 8: Matthew, B. & Ross, L. (2010) Research Methods: a practical guide for the social sciences. Pearson Education Limited NORAD (2008) Results Management in Norwegian Development Cooperation: a practical guide. Stein, D. & Valters, C. (2012) Understanding ‘Theory of Change’ in international al development : a review of existing knowledge Vogel, I. (2012) Review of the use of ‘Theory of Change’ in international development. For DfID UK Web Center for Social Research methods