Evaluating Social Cohesion: Lessons from UNICEF’s PBEA Program October 2014 © UNICEF/UGDA02504/Hyun.

Slides:



Advertisements
Similar presentations
Intelligence Step 5 - Capacity Analysis Capacity Analysis Without capacity, the most innovative and brilliant interventions will not be implemented, wont.
Advertisements

Thematic evaluation on the contribution of UN Women to increasing women’s leadership and participation in Peace and Security and in Humanitarian Response.
SCHOOL PSYCHOLOGISTS Helping children achieve their best. In school. At home. In life. National Association of School Psychologists.
Overview of the Global Fund: Guiding Principles Grant Cycle / Processes & Role of Public Private Partnerships Johannesburg, South Africa Tatjana Peterson,
TACKLING POVERTY TOGETHER Youth Contributing to Poverty Reduction.
Research Into Use Programme Supporting Innovation - Changing Lives Andy Frost For FANRPAN Workshop May 2007.
Evaluating Collaboration National Extension Family Life Specialists Conference April 28, 2005 Ellen Taylor-Powell, Ph.D. Evaluation Specialist University.
The Power of Early Childhood Developments in Conflict and Post Conflict Societies 6 February 2013 Europa Hotel, Belfast. Northern Ireland.
IIEP’s conflict-sensitive approach to education sector planning Suzanne Grant Lewis, Deputy Director UNESCO International Institute for Educational Planning.
AFRICAN UNION A FRAMEWORK FOR HARMONISED LAND POLICIES IN WEST AFRICA: an LPI – ECOWAS partnership Presentation to the World Bank Conference on Land Land.
Interagency Contingency Planning and Emergency Preparedness: Challenges and Lessons Learnt V. Harutyunyan M.D. Head of Health / Merlin Global Nutrition.
Implementation of Humanitarian Strategy Executive Board Informal August 28, 2014.
Global Poverty Action Fund Community Partnership Window Funding Seminar January 2014 Global Poverty Action Fund Community Partnership Window Funding Seminar.
Annual Report on the Implementation of the UNICEF Gender Action Plan, UNICEF Executive Board June 17, 2015.
Schlosser & Pfirman, Nature Geosciences, 2012 Integrated Approach Pilots: Charting a New Frontier for the Global Environment.
1 By Noreen M. Huni 6 th October, 2008 Dublin Outcomes of the East & Southern Africa Regional Inter-Agency Task Team (RIATT) Children’s Conference in Dar-es-Salaam.
Education for Peacebuilding June Why should Educators be interested in peacebuilding? Over 1 billion children under 18 live in areas affected by.
Integrating CS/PB into the CPD. Results Frameworks and M&E PB results frameworks and M&E must build on conflict analysis and programming for change Types.
Paper Title: “The influence of gender in the relation between Participatory Monitoring and Evaluation, and Citizen Empowerment” Conference Paper by: Kennedy.
Beyond the phases International Mosaic Committee 2006.
Understanding Conflict -- Introduction and Overview of the Global Peacebuilding, Education and Advocacy Programme
Education for Peacebuilding June Why should Educators be interested in peacebuilding? Over 1 billion children under 18 live in areas affected by.
Beyond Primary Education: Challenges of and Approaches to Expanding Learning Opportunities in AfricaAssociation for the Development of Education in Africa.
World Vision on a journey to evaluate Child Well-Being The evolution of our framework from a set of few indicators to a compendium of indicators linked.
Session 3C: Monitoring and Evaluation of C/DRR: Tools and Strategies
Using social norms perspectives for sexual violence prevention with conflict-affected communities Sophie Read-Hamilton & Mendy Marsh.
IPC Global Strategic Programme ( ) IPC Global Partners: IPC REGIONAL Strategic Programme IPC Regional Steering Committee Meeting – March.
REGIONAL ANALYSIS ON DISASTER RISK REDUCTION EDUCATION IN THE ASIA PACIFIC REGION IN THE CONTEXT OF HFA PRIORITY 3 IMPLEMENTATION Bangkok, March 2009 Prepared.
Participatory research to enhance climate change policy and institutions in the Caribbean: ARIA toolkit pilot 27 th meeting of the CANARI Partnership January.
Innovative Schools toolkit Strategic Workshop 3 - Exploring good practice case studies.
Developing Indicators
A conflict-sensitive approach involves:  Gaining a sound understanding of the two- way interaction between activities and context;  Acting to minimize.
Evaluation framework: Promoting health through strengthening community action Lori Baugh Littlejohns & Neale Smith David Thompson Health Region, Red Deer,
UNFPA-UNICEF Joint Programme on Female Genital Mutilation/Cutting: Accelerating Change Management Response and Key Actions.
UNICEF’s work and planned activities for the production of data on children with disabilities Claudia Cappa, Data and Analytics Section, UNICEF, NY.
System Implementation and Monitoring Regional Session Spring, 2014 Resources are available at sim.abel.yorku.ca.
1 Analysing the contributions of fellowships to industrial development November 2010 Johannes Dobinger, UNIDO Evaluation Group.
8 TH -11 TH NOVEMBER, 2010 UN Complex, Nairobi, Kenya MEETING OUTCOMES David Smith, Manager PEI Africa.
UNDAF M&E Systems Purpose Can explain the importance of functioning M&E system for the UNDAF Can support formulation and implementation of UNDAF M&E plans.
Margaret Barihaihi National Programme Coordinator, World Vision - ACCRA Uganda.
United Nations Volunteers Volunteerism for Development in the context of CBA Adeline Aubry CBA Volunteerism & Community Adaptation Specialist United Nations.
Third Sector Evaluation: Challenges and Opportunities Presentation to the Public Legal Education in Canada National Conference on “Making an Impact” 26.
The Next Stage for Results in Africa. Context 2005 Paris Declaration on Aid Effectiveness 2006 Mutual Learning Events Uganda & Burkina Faso 2007 Hanoi.
Engendering Change Mid- Term Learning Review Laura Haylock Kaia Ambrose October 2012.
1 Doing Things Differently in FCS Yongmei Zhou, CCSD Manager Civil Society Policy Annual Meetings October 10 th
1 UNDAF PRIORITY WORKING GROUP 5 HUMAN CAPITAL 4 December 2011.
Consultant Advance Research Team. Outline UNDERSTANDING M&E DATA NEEDS PEOPLE, PARTNERSHIP AND PLANNING 1.Organizational structures with HIV M&E functions.
Developing a Framework In Support of a Community of Practice in ABI Jason Newberry, Research Director Tanya Darisi, Senior Researcher
© Plan COMPETENCIES Transferring competencies to the field level Bell’Aube Houinato Geneva, February 2007.
Livia Bizikova and Laszlo Pinter
From Outcome To Output. Outcome  The intended or achieved short-term and medium-term effects of an intervention’s outputs,  usually requiring the collective.
“Clouds but little rain…” Views from the Frontline A local perspective of progress towards implementation of the Hyogo Framework for Action
Rafael Obregon Chief, C4D Section UNICEF NYHQ Bangladesh, May 2015
INEE Guidance Note on Conflict Sensitive Education Location, Date, 2013.
Innovative Schools toolkit STRATEGIC WORKSHOP 2 Exploring good practice case studies.
Findings From the National Preparations For Rio + 20 for Select Developing Countries A UNDESA/UNDP partnership International Workshop on Strengthening.
Capacity Development Results Framework A strategic and results-oriented approach to learning for capacity development.
Advancing learning through service Tamara Thorpe Trainer | Coach | Consultant Region 2 NAFSA Albuquerque, NM.
Bosun Jang & Rebecca Herrington 2015 AEA Conference 13 November 2015 © UNICEF/NYHQ /Asselin Capturing Voices of Young Beneficiaries in the Midst.
Upcoming Work on the Enabling Environment for Civic Engagement Initiative Jeff Thindwa Participation and Civic Engagement Group Social Development Department,
WG/UNICEF Child functioning module: Preliminary results from Samoa & Supporting documentation Mitchell Loeb National Center for Health Statistics/ Washington.
Small Charities Challenge Fund (SCCF) Guidance Webinar
Criteria for Assessing MHPSS Proposals Submitted through the CAP, CERF and HRF Funding Mechanisms to the Protection Cluster.
Developing culturally responsive and gender transformative evaluation curriculum: Reflections on South to South collaboration Madri Jansen van Rensburg,
Assessing Trust Fund Performance
5th Capitalization Meeting of the EU Land Governance Programme :
“The Anatomy of Grass root Capacity Building for Sustainable Management of Natural capital in the Nile Basin” -A Political Economy Approach Donald Kasongi.
Engendering Change Mid-Term Learning Review
Case study from a market-systems development programme in Ethiopia
REACH and Impact/Outcome PIIRS DATA on Resilience
Presentation transcript:

Evaluating Social Cohesion: Lessons from UNICEF’s PBEA Program October 2014 © UNICEF/UGDA02504/Hyun

The Evolution of M&E Systems for Peacebuilding, Education and Advocacy (PBEA) Program Bosun Jang Education Specialist, PBEA M&E UNICEF

VISION / IMPACT Strengthened resilience, social cohesion and human security in conflict-affected contexts PBEA program design 2 TARGET COUNTRIES South Asia Pakistan East & Southern Africa Burundi, Ethiopia, Somalia, South Sudan, Uganda East Asia & Pacific Myanmar West & Central Africa Chad, Cote d’Ivoire, Democratic Republic of the Congo, Liberia, Sierra Leone Middle East & North Africa State of Palestine, Yemen OUTCOMES Policy Peacebuilding and Education Integrated Increased inclusion of education into peacebuilding and conflict reduction policies, analyses and implementation and vice versa. Institution Building Increased institutional (including UNICEF) capacities to supply conflict sensitive and peace education. Individual Capacity Development Increased capacity of children, parents, teachers and community members to prevent, reduce and cope with conflict and promote peace Access to Conflict Sensitive Education Increased access for children to quality, relevant education that contributes to peace, including education delivered as a peace dividend Knowledge, Evidence and Advocacy Adequate generation and use of evidence and knowledge in policies and programming on linkages between education, conflict and peacebuilding

PBEA M&E operationalization 3 Global Conflict driver Theory of change ActivityOutputOutcome Impact Strengthened resilience, social cohesion and human security in conflict- affected contexts Country Conflict driver Theory of change ActivityOutputOutcome Partner Conflict driver Theory of change ActivityOutputOutcome

PBEA Global Results Framework (GRF) No.Key characteristicAdvantageDisadvantage 1 st Key performance indicators (KPIs) Helped set targets and structure to a global program Structure too rigid for meaningful, contextualized programming 2 nd Replaced KPIs with “menu” of indicators based on country- level indicators Flexibility in logic models and work plans for contextualization Too many unique indicators not amenable to aggregation 3 rd Indicators “bucketed” or coded for country-, regional-, and global- level aggregation and analysis Bottom-up approach that enables flexibility in design as well as aggregation Potential for misinterpretation of indicators in the coding process 4

Examples 5 Conflict driver The acceptance of violence and social disruption as a norm has resulted in a deterioration of social cohesion within the Palestinian territory. Theory of change If government and civil society provides civil and capacity development opportunities for adolescents, then adolescents will participate more effectively in community as a sustainable alternative to violence. ActivityOutputOutcome Support capacity building of adolescents to led initiatives including community- based activities. Adolescents have increased knowledge and skills to lead and participate in constructive initiatives in marginalized areas. Adolescents constructively engage in their societies as agents of social transformation for advancing sustainable alternatives to social violence and discord.

Going beyond performance monitoring Output Adolescents have increased knowledge and skills to lead and participate in constructive initiatives in marginalized areas. Indicator # of adolescent who participate in capacity building training # of initiatives conducted per year and type Result 360 adolescents (180 boys / 180 girls) (Target: 2,000 with 50% girls) 54 initiatives (Target: 108 initiatives) Captured the process of recruiting the 360 boys and girls, training them, and guiding them through the development of their own initiatives Case study

…and beyond process capturing 7 Research Three-pillar research efforts in Myanmar, Liberia, Pakistan, Sierra Leone, and Uganda on policy, teachers, and youth in partnership with Universities of Amsterdam, Sussex, and Ulster In Burundi, long term research partnerships have been established to investigate the mechanisms behind the intergenerational transmission of violence / resilience Impact measuring Beneficiary population-based knowledge, attitude, and perception (KAP) surveys in South Sudan and Somalia in partnership with Echo Mobile Population-based KAP surveys in Burundi and Uganda in partnership with Harvard Humanitarian Initiative (HHI) Developmental Eval. Infusion of evaluable thinking, recognition of complexities, and process documentation in Myanmar

Key lesson learned: Balance Structure vs. flexibility Clarity vs. ambiguity Bean counting vs. process and outcomes capturing Peacebuilding outcomes vs. education outcomes New system vs. existing system 8

Burning question 9 Global Conflict driver Theory of change ActivityOutputOutcome Impact Strengthened resilience, social cohesion and human security in conflict- affected contexts Country Conflict driver Theory of change ActivityOutputOutcome Partner Conflict driver Theory of change ActivityOutputOutcome Is this the right way to do our M&E given the complexity of the program and peacebuilding in general?

Approaches to Assessing Changes in Social Cohesion Julie Younes Design, Monitoring, and Evaluation Specialist Search for Common Ground (SFCG) 10

The Challenge Social cohesion is a fluid concept subject to multiple interpretations No official guidance on definitions from UNICEF Headquarters at program outset Political sensitivity (e.g., UNICEF Pakistan unable to use the term “peacebuilding”) Difficulties integrating outcome statements and indicators related to social cohesion into program logical frameworks –“There are only a few indicators that track changes in the key concepts of social cohesion, resilience or human security – these need improvement.” (PBEA Evaluability Assessment Report November 2013) Global Outcome 3 Individual Capacity Development: Increased capacity of children, parents, teachers and community members to prevent, reduce and cope with conflict and promote peace. Global KPI 3.1 % of targeted children (B/G), teachers and adult community members (M/F) reporting a positive change in their own ability to prevent, reduce and cope with conflict and promote peace. Pakistan KPI 3.1 % of targeted children (B/G), teachers and adult community members (M/F) reporting a positive change in their own ability to prevent, reduce and cope with conflict and promote peace [Social Cohesion].

Key Considerations – Pakistan vs. Somalia  Target group: Adolescents and adults Questions are more complex, delve into social and political issues  Direct beneficiaries only  Part of a regional survey effort, with questions adapted to local context  Not attached to a specific indicator  Use: Inform ongoing + future programming  Qualitative data collection: Case study  Target group: Children Questions are shorter and simpler, focused on personal, family and school interactions  Direct beneficiaries only  Administered by UNICEF PBEA Pakistan only  MOV for Indicator 3.1  Use: Inform ongoing programming  Qualitative data collection: Participatory focus groups Pakistan: Youth Social Cohesion Survey Somalia: Social Cohesion and Resilience KAP Survey

The Approach – Pakistan  Define concept of social cohesion: Five domains identified as most relevant to the Pakistan context  Develop initial survey questions based on existing literature e.g. Cohesion Radar: Measuring Cohesiveness; Measuring Cohesiveness: Social Attraction Scale; Social Cohesion: Measurement Based on the Data from the European Value Study  First round of “piloting” with UNICEF Pakistan staff; second round of more traditional piloting with respondents who fit profile of program participants

The Approach – Somalia  Define concept of social cohesion: Five relevant domains identified, along with one section on resilience  Questions developed based on existing literature and previous large-scale surveys e.g. World Bank working papers Understanding and Measuring Social Capital and Resilience in Education Setting Research; Surveys conducted by the World Bank and IOM in East Timor ( ) and Aceh ( )  Part of an East Africa regional survey effort and thus could use existing versions of the KAP (e.g. South Sudan) as initial guidance  First round of “piloting”: Full-day workshop with UNICEF Somalia staff and local partner organizations to go through survey questions and adapt to local context; traditional piloting also conducted

The Approach – Pakistan Trust If children are fighting in school, what is the first thing that the teacher should do? Belonging and Inclusion I feel that I am part of my community Participation Outside of my home, I like to do activities with a club, team or other group Tolerance I do not like to be around people who have opinions or beliefs that are different than mine Recognition and Legitimacy I can play a part in influencing group decisions that are important to me

The Approach – Somalia Attitudes Towards Violence If children are fighting in school, what is the first thing that the teacher should do? Belonging and Inclusion Do you think that schools should be equipped so that children with disabilities can study in the same school as children with no disabilities? Respect and Trust If you have a land dispute with your neighbor, how much do you trust your community elders to solve the problem? Membership and Participation Are you a member of a community group? Attitudes Towards Social Services If the government is planning an education project and you want to give your opinion on it, do you think they would listen? Resilience The last time there was a drought or a flood, did you have to sell any property?

The Approach – Pakistan  Results to inform programming: Baseline for KPI 3.1: Overall, 27% of participants reported a positive Social Cohesion Score Qualitative data collection: Participatory focus groups to take a deep dive into survey results (November 2014)

The Approach – Somalia  Results: Baseline data collected; currently in the data entry, cleaning and analysis phase, which is being conducted at the regional level Results from the survey will feed into a regional dashboard to allow for analysis across program countries  Qualitative data collection: Case study to examine issues around social cohesion among targeted children and youth (specifically those younger than 14 who cannot take the KAP survey)

Lessons Learned ✓ Break down the concept into measureable pieces most relevant to program context ✓ Existing literature or surveys as a starting point ✓ “Double” piloting ✓ Reaffirmed utility of using of mixed methods for a single indicator

Thank you!

Panel discussion questions 21

Panel discussion questions Experiences or suggestions for designing M&E systems for complex, global programs? Approaches for defining complex concepts like social cohesion or peacebuilding? Suggestions for qualitative data collection tools to complement Pakistan Youth Social Cohesion Survey? 22

Useful links UNICEF Learning for Peace SFCG DM&E for Peace: Education for Peacebuilding M&E Community of PracticeSFCG DM&E for Peace: Education for Peacebuilding M&E Community of Practice 23