1. Engaging stakeholders in PM&E Processes: Lessons from the Field Presentation to AGDEN by Susan Kaaria Ford Foundation

2. Outline Background Why is PM&E Important Lessons Learned from my own experience Results from this work Conclusions

3. My background My history with PM&E Working with PR program at CIAT – developing participatory approaches for involving communities in the R&D processes –PTD, Evaluating technologies, identifying farmer priorities for technology –Need to empower these communities to M&E their initiatives, so that they can engage with development more effectively –Developing PM&E methods to build capacity of NARS, NARES, NGOs and communities in establishing and applying these systems for learning and assessing change Today I will share with you some of the lessons from my experience implementing a PM&E project in Eastern and Southern Africa Assessment and evaluation at FF –My colleagues told me that we cannot assess impacts of social change processes. –Concern that this would have additional budgetary implications –New President Luis Ubinas is very keen to see that assessment and evaluation are integrated in all our work. There is a new office for IA, and regions and programs are required to have an IA plan. –An initiative on developing capacity for assessment and evaluation is going to be crucial for the new thinking with the FF.

4. Key Schools of thought that have influenced the evolution of this work 1.Michael Quinn Patton -- “Utilization focused evaluation” –Has as its central tenet the need for any assessment process to be useful. Its concern for ensuring that learning ensues from an assessment process among those involved with the program or process being evaluated makes it relevant for social change processes. 2.David Fetterman “Empowerment Evaluation” –Participatory and empowerment evaluation stresses the need for people to assess the merits of their own or externally-driven initiatives that affect them, thus enabling improvement and strengthening people’s agency. 3.Organizational learning relates to a set of perspectives and procedures that enable learning to be embedded in a program or organization. 4.The Participation, Power and Social Change Team at IDS Sussex: Irene Guijt, John Gaventa, Andrea Cornwall, Joan Abbot, Marisol Estrella, IDRC –Action research to change practices and social structures. The cycle of action and critical reflection is participatory, value- oriented, and democratic in its intentions.

5. The context From 2003, CIAT co-implemented a project with the Kenya Agricultural Research Institute (KARI) on improving PM&E KARI has 16 sub regional, regional and national research centers 5 (1 national and 4 regional) centers were involved in the PM&E project 10 research programmes were actively participating. This number rose to 17 by the end of the project Levels of PM&E Community level Project level

6. Why the interest in strengthening PM&E processes? Moving away from…..  Stakeholders, who are directly involved in or affected by the project, have little or no input in the process. “Outsiders” are contracted to carry out the evaluation  A tool for policing / supervisory tool that is done by outsiders mainly donors and external experts to check on the accountability  Ex. KARI – Monitoring visits by HQ staff. So the staff only focus on successes Moving towards…..  Measuring change from the perspective of those impacted by the project. Farmers and Researchers are active participants- not just source of information  Building stakeholders’ capacity for self reflection, critical learning and problem solving  It is a system for tracking and documenting processes and changes by and for various stakeholders  A process of internal learning and self-reflection  Strengthen accountability -- local capacity to monitor and evaluate how well service providers and products are meeting their needs

7. Lessons Learned from my own experience 1. Capacity building is crucial A majority of Civil society, government, etc personnel have very little knowledge on how to establish a PM&E. Ex. KARI staff –M&E was a tour by their superiors in HQ. It was a policing event –Lack of in-built PM&E during project development and well defined frameworks –Technical skills on establishing and supporting PM&E systems were limited –Community members are not empowered to participate actively in these processes –Lag between baseline data collection, analysis and feedback does not allow taking corrective measures/actions –PM&E as the mandate for the social scientist What we did Changing the mindset – understanding the role of participation, involving stakeholders Developed a whole range of capacity development material for all levels of stakeholders – NGO partners, NARS partners, Communities –Extensive use of graphics to conceptualize terms –Role plays: River code, examples from daily lives –Created an-house capacity in PM&E Mentoring process, practical application and with continuous capacity building

8. 2. Creating a shared understanding of the theory of change amongst the stakeholders –Many initiatives are rarely discussed with all stakeholders –Projects are developed in centralized offices and then implemented without consultations Discussions on the theory of change –Vision of the community for their own development, expected outcomes and results, and actions required to achieve these results, indicators to track the project –Many projects had not included key activities –Clear differences between expected outcomes by communities and researchers: Researchers, technical oriented outcomes (Improved soil fertility) Communities had more livelihood and empowerment oriented outcomes (Empowerment, Ability to demand and access to other services) –Other projects had to change focus completely because of these discussions –Created a shared understanding of what the project wanted to achieve Lessons Learned from my own experience

9. 3. Engendering the process A majority of initiatives have a focus on women and the poor, but it was not always apparent how this goal would be achieved Many poor rural women and youth are not involved in the village meetings, so how can one get their views Action: Vision of the community, expected outcomes and results, activities and indicators had to include these groups –Increased capacity of poor men and women… –“Observable increase in the capacities of poor men and women to advocate for their land rights” Ensuring that both men and women could contribute – separate meetings Creating teams that included both men and women Using simple tools for training so all can understand Using questions that helped to highlight these aspects –How will we know that women are benefiting from this project Having specific gender awareness training sessions – using tools and methods to highlight the differences Lessons Learned from my own experience

10. 4. To track complex processes such as social change requires creativity on the part of the evaluator –Large survey instruments are not always appropriate to track social change processes – empowerment, power, voice and accountability, realization of rights Action: Creativity and a mix of methods and tools are required Use of qualitative indicators – which means that one must tell the story. Indicator for Food security –Shiny faces; Presence of dish racks with clean utensils; Fruits ripening on the trees; Reduction in conflicts in the home, People’s pets are healthy Methods: Outcome mapping, most-significant change, Impact diagramming, stories, drama, Oral testimonies, Songs and drama, Pictures Impact Diagramming –When analyzed using conventional M&E methods an Impact study found that Beans production had increased income and food. –Further assessment using impact diagramming showed other negative impacts: More work for women in weeding More people drinking local beer Extra marital affairs increased More fighting in the house AIDS prevalence was increasing Lessons Learned from my own experience

11. 5. Use of PM&E findings is not automatic Lag between baseline data collection, analysis and feedback does not allow taking corrective measures/actions Lots of data is collected but never used Data collection tools are very complex so the data is unavailable to everyone Action: Regular process of reflection must be inbuilt into the process: –What have we achieved this season/ this year etc? –What worked well? –What did not work well? –What do we need to change? Using the results to inform what we are doing –What have we learned? –What approaches worked or did not work? –How can we integrate these lessons into future strategies? –What is the best way to inform policy makers? Lessons Learned from my own experience

12. Some Results Changes at individual researcher level Improved technical skills in PM&E- from “inadequate” to “good” Increased confidence in working with communities Better engagement and dialogue with communities Commitment to monitoring and evaluation of own programs Improved and targeted data collection within programs Changes in institutional behaviour PM&E seen as everyone's responsibilities Commitment by management to PM&E Not a policing tool but a objective assessment of change for self improvement Joint planning and reflection Culture of data collection, analysis and use

13. Some results Changes in relationships with other stakeholders Involvement of stakeholders especially farmers and the extension in planning Involvement of farmers not just as sources of information but also in developing objectives and outcomes of the programs, indicators and in data collection and analysis Improved facilitation skills leading to better engagement with other stakeholders Improved downward accountability Changes at community level Better articulation of their needs and priorities Improved decision making and collective responsibility for achieving outcomes Increased transparency and accountability within farmer groups More awareness and ownership of initiatives by communities

14. Summary of Lessons Need for change of attitude –monitoring and evaluation- accountability –monitoring and evaluation- for learning and change Capacity building is crucial Processes are as important (More important?) as outputs PM&E results do not always lead to change Supporting institutional mechanisms for buy-in from management High initial investments in time, financial resources and facilitation

15. Conclusion Process Use Process use refers to and is indicated by individual changes in thinking and behavior, and program or organizational changes in procedures and culture, that occur among those involved in evaluation as a result of the learning that occurs during the evaluation process. Evidence of process use is represented by the following kind of statement after an evaluation: "The impact on our program came not just from the findings but from going through the thinking process that the evaluation required.“ Michael Q. Patton AfrEA 2007