1. Developing a common Monitoring and Evaluation (M&E) framework for the IASC Guidelines on Mental Health and Psychosocial Support (MHPSS) in emergency settings Jura Augustinavicius, MSc MHS Doctoral Student Johns Hopkins Bloomberg School of Public Health

2. Outline Key Monitoring and Evaluation (M&E) concepts Main components of Logical Frameworks (Logframes) IASC Need for a common M&E framework Review –Aims –Introduction of Draft I of the Common M&E Framework –Methods –Results Conclusions

3. Key Concepts: M&E Functions of M&E: 1.Process Evaluations: How well was the project implemented? 2.Monitoring outputs and outcomes: Did changes occur in the target population? 3.Impact evaluation: Can observed changes be attributed to the project or programme (i.e. cause and effect)?

4. Key Concepts: Logframes A systematic tool for designing, planning, implementing, and monitoring and evaluating a project or programme –Can help organize thinking –Identify and assess risks inherent in project design and implementation –Measure project progress –Develop consensus and communicate a project’s intent and strategy

5. PROJECT STRATEGY GOAL: The overall good to which the programme contributes (and to which other factors and programmes may also contribute)

6. PROJECT STRATEGY GOAL : The overall good to which the programme contributes (and to which other factors and programmes may also contribute) OUTCOMES: The specific parts of the overall goal that the project will achieve and that the project is held responsible for

7. PROJECT STRATEGY GOAL : The overall good to which the programme contributes (and to which other factors and programmes may also contribute) OUTCOME : The specific part of the overall goal that the project will achieve and that the project is held responsible for OUTPUTS: The project intervention; Direct and immediate products of activities

8. PROJECT STRATEGY GOAL : The overall good to which the programme contributes (and to which other factors and programmes may also contribute) OUTCOME : The specific part of the overall goal that the project will achieve and that the project is held responsible for OUTPUTS : The project intervention; Direct and immediate products of activities ACTIVITIES: What people who are employed by the project do to assist the community

9. PROJECT STRATEGY GOAL : The overall good to which the programme contributes (and to which other factors and programmes may also contribute) OUTCOME : The specific part of the overall goal that the project will achieve and that the project is held responsible for OUTPUTS : The project intervention; Direct and immediate products of activities ACTIVITIES : What people who are employed by the project do to assist the community INPUTS: Major resources required to carry out activities

10. PROJECT STRATEGY ASSUMPTIONS: Variables that are outside of the control of the project but that affect the success of the project Use “if, then” logic Should be generated based on past experience with similar programs GOAL : The overall good to which the programme contributes (and to which other factors and programmes may also contribute) OUTCOME : The specific part of the overall goal that the project will achieve and that the project is held responsible for OUTPUTS : The project intervention; Direct and immediate products of activities ACTIVITIES : What people who are employed by the project do to assist the community INPUTS : Major resources required to carry out activities

11. PROJECT STRATEGY INDICATORS : Used to measure progress towards achieving the project strategy ASSUMPTIONS: Variables that are outside of the control of the project but that affect the success of the project GOAL : The overall good to which the programme contributes (and to which other factors and programmes may also contribute) OUTCOME : The specific part of the overall goal that the project will achieve and that the project is held responsible for OUTPUTS : The project intervention; Direct and immediate products of activities ACTIVITIES : What people who are employed by the project do to assist the community INPUTS : Major resources required to carry out activities

12. PROJECT STRATEGY INDICATORS : Used to measure progress towards achieving the project strategy MEANS OF VERIFICATION: Sources of indicator data ASSUMPTIONS : Variables that are outside of the control of the project but that affect the success of the project GOAL : The overall good to which the programme contributes (and to which other factors and programmes may also contribute) OUTCOME : The specific part of the overall goal that the project will achieve and that the project is held responsible for OUTPUTS : The project intervention; Direct and immediate products of activities ACTIVITIES : What people who are employed by the project do to assist the community INPUTS : Major resources required to carry out activities

13. IASC IASC guidelines were developed in 2007 Published consensus on MHPSS in emergencies IASC MHPSS Reference Group continues to meet to disseminate and implement the guidelines and to develop MHPSS products 30+ member organization

14. A Common M&E Framework Systematic M&E can offer a means to understand the relative benefits of MHPSS interventions Consolidate goals, outcomes, and indicators for MHPSS activities across organizations Offers a common platform from which to advocate for prioritizing MHPSS in humanitarian settings

15. Aim To identify the most commonly applied strategies to monitor and evaluate MHPSS interventions in LMICs using a logframe approach

16. Methods 1. Review of Logframes and Theories of Change Documents collected from IASC RG MHPSS member organizations across humanitarian sectors and anonymized

17. Methods 1. Review of Logframes and Theories of Change Documents collected from IASC RG MHPSS member organizations across humanitarian sectors and anonymized 2. Narrative Review EMBASE, PILOTS, PsycInfo, PubMed/MEDLINE, and WHO regional databases

18. Methods 1. Review of Logframes and Theories of Change Documents collected from IASC RG MHPSS member organizations across humanitarian sectors and anonymized 2. Narrative Review EMBASE, PILOTS, PsycInfo, PubMed/MEDLINE, and WHO regional databases Standardized instructions for double data extraction of: Goals Outcomes Indicators Means of Verification Top-down: Mapping outcomes and indicators to outcomes from the preliminary IASC RG framework Bottom-up: Qualitative (thematic) analysis to identify common themes at each logframe level

19. Goal: Enhanced mental health and psychosocial wellbeing/reduced psychological and social suffering Outcomes: Community-focused (1) Emergency responses are safe, dignified, participatory, community owned, socially and culturally acceptable (2) People are safe, protected and human rights violations are addressed (3) Existing social, community, and family systems promote an environment for recovery and human development Person-focused (4) Communities and families support people with mental health and psychosocial problems (5) People with mental health and psychosocial problems access appropriate focused care Underlying core principles: 1. Human rights and equity; 2. Participation; 3. Do no harm; 4. Integrated services and supports; 5. Building on available resources and capacities; 6. Multi- layered supports Developed based on input from workshops in Sri Lanka, Lebanon, and Uganda

20. Results: Logframe & TOC Review 39 Documents reviewed Logframes (84.6%)

21. Results: Logframe & TOC Review

22. Results: Narrative Review 88 Articles and Book Chapters Studies published between 1992-2013 Study Population Children and adolescents (31%) Adults (32%) Combination of children and adults (23%) Unknown (14%)

23. Results: Narrative Review

24. Qualitative Analysis Results Theme 1: Increasing resilience, promoting social wellbeing, and preventing mental health and psychosocial problems Theme 2: Reducing ill mental health, psychosocial symptoms, and associated functional impairment through care Theme 3: Building capacity to identify, intervene on, and monitor MHPSS problems Theme 4: Promoting optimal human development within existing social systems Theme 5: Macro level goals that seek to build peace between groups after conflict and to address structural problems within societies Theme 6: Protection of vulnerable groups, such as women, children, the elderly, and people with disabilities

25. Qualitative Analysis Results Theme 1: Increasing resilience, promoting social wellbeing, and preventing mental health and psychosocial problems Theme 2: Reducing ill mental health, psychosocial symptoms, and associated functional impairment through care Theme 3: Building capacity to identify, intervene on, and monitor MHPSS problems Theme 4: Promoting optimal human development within existing social systems Theme 5: Macro level goals that seek to build peace between groups after conflict and to address structural problems within societies Theme 6: Protection of vulnerable groups, such as women, children, the elderly, and people with disabilities

26. Qualitative Analysis Results Theme 1: Increasing resilience, promoting social wellbeing, and preventing mental health and psychosocial problems Theme 2: Reducing ill mental health, psychosocial symptoms, and associated functional impairment through care Theme 3: Building capacity to identify, intervene on, and monitor MHPSS problems Theme 4: Promoting optimal human development within existing social systems Theme 5: Macro level goals that seek to build peace between groups after conflict and to address structural problems within societies Theme 6: Protection of vulnerable groups, such as women, children, the elderly, and people with disabilities

27. Qualitative Analysis Results Theme 1: Increasing resilience, promoting social wellbeing, and preventing mental health and psychosocial problems Theme 2: Reducing ill mental health, psychosocial symptoms, and associated functional impairment through care Theme 3: Building capacity to identify, intervene on, and monitor MHPSS problems Theme 4: Promoting optimal human development within existing social systems Theme 5: Macro level goals that seek to build peace between groups after conflict and to address structural problems within societies Theme 6: Protection of vulnerable groups, such as women, children, the elderly, and people with disabilities

28. Qualitative Analysis Results Theme 1: Increasing resilience, promoting social wellbeing, and preventing mental health and psychosocial problems Theme 2: Reducing ill mental health, psychosocial symptoms, and associated functional impairment through care Theme 3: Building capacity to identify, intervene on, and monitor MHPSS problems Theme 4: Promoting optimal human development within existing social systems Theme 5: Macro level goals that seek to build peace between groups after conflict and to address structural problems within societies Theme 6: Protection of vulnerable groups, such as women, children, the elderly, and people with disabilities

29. Qualitative Analysis Results Theme 1: Increasing resilience, promoting social wellbeing, and preventing mental health and psychosocial problems Theme 2: Reducing ill mental health, psychosocial symptoms, and associated functional impairment through care Theme 3: Building capacity to identify, intervene on, and monitor MHPSS problems Theme 4: Promoting optimal human development within existing social systems Theme 5: Macro level goals that seek to build peace between groups after conflict and to address structural problems within societies Theme 6: Protection of vulnerable groups, such as women, children, the elderly, and people with disabilities

30. Conclusions The current draft of the common MHPSS M&E framework can be applied to existing documents and peer-reviewed literature Confusion with regard to defining goal and outcomes Although psychometrically sounds means of verification are published in the peer- reviewed literature, these are not commonly applied to logframes or theories of change

31. IASC MHPSS Reference Group Wietse Tol Ananda Galappatti Saji Thomas Alison Schafer Margriet Blaauw Mark van Ommeren Johns Hopkins Bloomberg School of Public Health M. Claire Greene Annisa Harsha Megan Cherewick Jillian Emerson Katharine Shelley Amanda Samson Srihari Cattamanchi Saumya Singh Acknowledgements With funding from UNICEF

32. Additional Slides

33. Key Concepts: M&E Monitoring: A continuing function that aims primarily to provide the management and main stakeholders of an ongoing intervention with early indicators of progress, or lack thereof, in the achievement of results

34. Key Concepts: M&E Evaluation: A selective exercise that attempts to systematically and objectively assess progress towards and the achievement of a health outcome

35. Key Concepts: M&E Programme MonitoringProgramme Evaluation What is it? Assessment of whether a project is being implemented as planned Assessment of whether: The project achieved the objectives/intended results The project provided value and made a difference Why do it? When is it done? What is assessed?

36. Key Concepts: M&E Programme MonitoringProgramme Evaluation What is it? Assessment of whether a project is being implemented as planned Assessment of whether: The project achieved the objectives/intended results The project provided value and made a difference Why do it? To determine what changes are needing during the project To decide what activities to repeat in future projects When is it done? What is assessed?

37. Key Concepts: M&E Programme MonitoringProgramme Evaluation What is it? Assessment of whether a project is being implemented as planned Assessment of whether: The project achieved the objectives/intended results The project provided value and made a difference Why do it? To determine what changes are needing during the project To decide what activities to repeat in future projects When is it done? During the life of the project Frequently or continuously At the beginning and end of a project (occasionally at the midpoint) What is assessed?

38. Key Concepts: M&E Programme MonitoringProgramme Evaluation What is it? Assessment of whether a project is being implemented as planned Assessment of whether: The project achieved the objectives/intended results The project provided value and made a difference Why do it? To determine what changes are needing during the project To decide what activities to repeat in future projects When is it done? During the life of the project Frequently or continuously At the beginning and end of a project (occasionally at the midpoint) What is assessed? Resource amounts and their distribution Delivery, distribution, and quantity of services Costs and expenditures Actual changes relative to intended changes Value of project activities and achievements