Monitoring and evaluation PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION

Focus of this workshop session MonitoringEvaluation What they are? Why they are important? Who is involved? Planning Methods Psychosocial interventions Indicators

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Write a word or phrase that you associate with the term ’monitoring’ on the cards you have received. When you have finished, display your cards on the board/wall space. If someone has written the same word as you then put your card on top of that card. What is monitoring?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Monitoring is the regular and continuous process of collecting and analyzing data to assess progress and development. What is monitoring?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Two main types of monitoring in RCRC Process oriented Progress and development of response Activities implemented as planned? Use of resources Problems? How should they be dealt with? New opportunities for improving response?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Two main types of monitoring in RCRC Results oriented Direct results of the interventions Relevance of present and planned activities Are objectives still realistic/relevant? Changes in targeted population / environment (impact) Is more information needed?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Is the response still relevant to the needs of the population? What is monitoring? Keeps track of inputs, outputs and outcomes using indicators

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION What are inputs, outputs and outcomes? Input Any resources that is ‘put in’ to the program E.g. funds, materials, personnel, time Outputs: Measurable achievements that have been ‘put out’ or produced as a result of the intervention E.g. numbers of people trained, numbers of meetings held, etc. Outcomes: Changes that have come about as a result of the program E.g. improvement of psychosocial wellbeing, skills and knowledge on PSS improved,

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION INDICATORS Brainstorm: What are indicators?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION What are indicators? Criteria used to measure the data collected Measure changes related to implementation Examples of (outcome) indicators in PS responses: Decrease in stress related symptoms of population A Increase knowledge and skills in providing PFA Children regain desire to play Beneficiaries regain capacity to relate to others

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Examples of Input, output and outcome indicators Input Indicators: AMOUNT $ spent; NUMBERS of personnel working; NUMBERS of hours Output indicators: NUMBERS of people trained; NUMBERS of meetings held; NUMBERS of PS kits distributed Outcome indicators: NUMBER of children playing (quantitative); self- reported INCREASE in desire for social interaction (qualitative); LEVEL of skill improvement in PSS

Keeping track of inputs, outputs and outcomes for activities - An example Logical framework approachIndicators Overall Goal To reduce suffering and risk for development of severe trauma of Population A Adults and children show healthy signs of coping with impact of crisis event Activity 1Training in PFA InputResources to enable trainingAmount of money; personnel; training manuals; materials OutputTraining has taken placeNumber of people trained OutcomeVolunteers can provide PFASkill level in PFA has increased Activity 2:Workshops with children InputResources for workshopsAmount of money; personnel; training manuals; materials OutputWorkshops are heldNumber of workshops held; number of children attended; number of trainers OutcomeChildren are coping betterIncrease in children’s playfulness, self-confidence, trusting of others

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Why is monitoring important? 1.Relevance 2.Responsibility and communication 3.Accountability Simply put: Are we on track? Are we on the RIGHT track?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Who is involved in monitoring? Responsibility Including monitoring and evaluation Data collection Other partners

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION In buzz groups of 2 or 3, discuss amongst yourselves what evaluations are and how they differ from monitoring? Try to write a simple and clear definition of what an evaluation does in a psychosocial response. What are evaluations?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION What are evaluations? “Evaluations explore whether the interventions succeeded in achieving the overall goals and aims of the psychosocial response” Discussion points: 1.Evaluations are meant to be OBJECTIVE – what does it mean and why is it important?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION What are evaluations? Evaluations = Monitoring + more Final evaluation Mid term evaluation M1M2M3M4M5M6M7M8

Evaluations focus on assessing OVERALL AIM/GOAL Logical framework approachIndicators Overall Goal To reduce suffering and risk for development of severe trauma of Population A Adults and children show healthy signs of coping with impact of crisis event Activity 1Training in PFA InputResources to enable trainingAmount of money; personnel; training manuals; materials OutputTraining has taken placeNumber of people trained OutcomeVolunteers can provide PFASkill level in PFA has increased Activity 2:Workshops with children InputResources for workshopsAmount of money; personnel; training manuals; materials OutputWorkshops are heldNumber of workshops held; number of children attended; number of trainers OutcomeChildren are coping betterIncrease in children’s playfulness, self-confidence, trusting of others

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Types of evaluations Real time evaluations Early in response Internal staff Focus: process / operational issues Mid-term evaluations Assess impact Accountability (budgets/admin) Guide needed adaptations Final evaluations Impact and process key successes & challenges Invaluable info for future PS interventions

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Evaluating impact How do you measure the impact of an intervention? What do you need? Data SAME INDICATORS

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION How to define Psychosocial indicators Assess - How is the population affected? What kind of change is desired? (Goals/objectives/activities of response) What is the local definition of psychosocial wellbeing?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Indicators of psychosocial wellbeing

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Psychosocial indicators 1. Write three things about yourself or your life that you think show to others that you are feeling good and doing well. 2. Now think and write about how others could measure whether you felt better from one day to the next. 3. In plenary share some of your indicators of wellbeing. We aim to come up with a good variety of indicators, and also some indicators that may be the same for the whole group.

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Psychosocial indicators People have different things that indicate their psychosocial wellbeing, but if we spent some time we would probably be able to find some common indicators that everyone agree would be both general, and specific, enough to determine the groups’ psychosocial wellbeing. Indicators of psychosocial wellbeing focus on: - How the individuals are doing - How the group is doing. The nature of psychosocial is a focus on the individual’s psyche and on the social interaction and network.

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Psychosocial indicators We need to help the children feel better…. But how will we KNOW if they feel better? I KNOW.. We will ask them how THEY know when they feel better and then we will find ways to measure that…

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Why is evaluation important? 1.Relevance (response relevant to needs of pop.) 2.Efficiency (time-frame, cost) 3.Impact (succeeded or not) 4.Effectiveness (objective of program met) 5.Sustainability (benefits continue beyond the program)

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Who is involved in evaluations apart from staff, volunteers, partners? Responsibility Including monitoring and evaluation Data collection Other partners External consultants

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Sampling Broad & varied Targeted populationBroad and varied

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION When to monitor and evaluate

PSYCHOSOCIAL INTERVENTIONS ASSESSMENT How to conduct assessments? Two main categories of data collection methods QuantitativeQualitative Surveys, Key informant interviews Questionnaires Focus Group Discussions Psychometric tools Observations (measure reactions, behaviour, feelings, using a scaled measure) 1,2,3 Word descriptions =, %, a:b

Logical framework approachIndicators Overall Goal To reduce suffering and risk for development of severe trauma of Population A Adults and children show healthy signs of coping with impact of crisis event Activity 1Training in PFA InputResources to enable trainingAmount of money; personnel; training manuals; materials OutputTraining has taken placeNumber of people trained OutcomeVolunteers can provide PFASkill level in PFA has increased Activity 2:Workshops with children InputResources for workshopsAmount of money; personnel; training manuals; materials OutputWorkshops are heldNumber of workshops held; number of children attended; number of trainers OutcomeChildren are coping betterIncrease in children’s playfulness, self-confidence, trusting of others Are data needed for these indicators quantitative or qualitative? PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION

Collecting the data 1.Well planned and justifiable 2.Coordination 3.Clarifying aims and procedures 4.Participatory and collaborative Ethical principals to data collection

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Collecting the data 5.Comparison groups 6.Conduct and consent 7.Privacy and confidentially 8.Anticipate adverse consequences Ethical principals to data collection

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Planning for monitoring and evaluation Group work: What needs to be planned to ensure good monitoring and evaluation?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Planning for monitoring and evaluation 1.Timing a.Response length (short vs. long term) b.Appropriate time for affected pop. (ex. Do not plan monitoring activities with in-school children during exam time etc) a.Time to plan, implement, analyze, follow up (time and resources should be allocated) 2.Training a.Management on PSS b.Data collectors c.Program team – analysis and reporting

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Planning for monitoring and evaluation 3.Resources a.Staff and volunteers b.Financial c.Time d.Logistics 4.Information sharing a.Internal standards and formats b.External – different audiences (beneficiaries/donors/public)

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Role and importance of M&E in PSS Helps us understand local – and relevant – meaning of psychosocial wellbeing Keeps us on the right track Helps us do our best, now and in the future